So in the previous lecture, I concluded by showing you a graph of a normal advertising campaign. So this is real data from a real campaign that I collected. And what we see on the x axis is the amount of websites that exposes specific consumer to ads. And on the y axis on the left, we'll see the click-through rate. What is the the probability of a consumer clicking on those ads if they see more and more of ads on different channels? Now is this conclusion, is this model correct? Let's try to think about it, right? Does it make sense that the more we expose consumers to ads, the more they click on them? Are there any alternative explanations, or can we think of other things that might happen in this data? You know what? Let's try to look at a different experiment we tried and see what the data tells us. So because we measure click-through rates, and because those networks are actually compensated by click-through rates, that is, the advertisers pays the websites if consumers clicked more on their ads. Maybe the websites will show ads to consumers who will actually click the ads anyway. That is, the websites know how to recognize consumers that click a lot of ads. And they show them the ads even though the ads have no impact. And that might be what may be going on. So let's take a look at this graph. And this graph is very, very interesting. In this case, what we see is, we show the click-through rate, or the probability of clicking an ad after a consumer has already visited the advertiser website. That is, a consumer has said, I'm going to buy the product with a very high probability. And only after that, the website starts showing the consumer those ads, many, many, many, many times. And what we can see is that If you show an individual ad after that visit, actually there is some increase in click-through rates. Actually consumers say this ad reminded me of the product I didn't buy, and leave in my shopping cart. Or maybe I would like to look at that brand again. And actually, there's a bump there, moving from 0 to 1, from seeing no ads to actually seeing 1 ad. But when we increase the number of ads shown more and more, that is when we show more and more ads on different websites, we go back to the same click-through rate that we had just before we showed the additional ad. That is, those websites actually know how to recognize consumers who will click the ads anyway but have no impact on the click-through rate. The click-through rate doesn't go up. It just remains the same and I show more and more ads. So just to conclude, we've seen examples of how applying the same descriptive data in the same predictive analytics to different goals and different models actually finds a different optimal action. And we've shown how to find this action depending on our goal. Many times we think or we sometimes make the mistake of saying this graph describes what consumers actually do. If I decrease the price, the consumers will buy more. And if I increase the price, the consumers will buy less, which is kind of a general truth. But what we don't know is what happens outside of this graph. And many times, the interpretation as a causal model is slightly problematic, as we've seen in the previous slide. If we think that ads are shown randomly to consumers, then of course, when we show more ads, the graph implies that consumers click more on the ads. But if the websites can choose who to show the ads to, maybe they will show the ads to consumers that will click the ads anyway just to maximize their profit. And then our conclusion is incorrect. So to summarize, what we need to do is we need to be very, very careful about drawing the conclusions from the descriptive data into an action that might yield the wrong conclusion. In order to solve that, we sometimes need to run an experiment or we need to test different models to see which one actually describes the reality the best. So if you would like to learn more about this topics, what I've discussed in this set of lectures is covered in different fields in economics, in marketing, and also in other areas. Specifically, consumer theory in economics covers basic pricing theories and the optimization exercises we've done in this set of lectures. In addition, competition and more sophisticated models require knowing some game theory. And within economics, the field is called industrial organization, or IO, that looks at how companies compete and also interact with consumers. In the next set of lectures, you will see a range of applications discussing different case studies and applying the different principles you've learned in this lectures in the customer analytics course.