Drop out. Does this seemingly crazy thing of randomly knocking out units in your network? Why does it work? So as a regulizer, let's give some better intuition. In the previous video, I gave this intuition that drop out randomly knocks out units in your network. So it's as if on every iteration you're working with a smaller neural network. And so using a smaller neural network seems like it should have a regularizing effect. Here's the second intuition which is, you know, let's look at it from the perspective of a single unit. Right, let's say this one. Now for this unit to do his job has four inputs and it needs to generate some meaningful output. Now with drop out, the inputs can get randomly eliminated. You know, sometimes those two units will get eliminated. Sometimes a different unit will get eliminated. So what this means is that this unit which I'm circling purple. It can't rely on anyone feature because anyone feature could go away at random or anyone of its own inputs could go away at random. So in particular, I will be reluctant to put all of its bets on say just this input, right. The ways were reluctant to put too much weight on anyone input because it could go away. So this unit will be more motivated to spread out this ways and give you a little bit of weight to each of the four inputs to this unit. And by spreading out the weights this will tend to have an effect of shrinking the squared norm of the weights, and so similar to what we saw with L2 regularization. The effect of implementing dropout is that its strength the ways and similar to L2 regularization, it helps to prevent overfitting, but it turns out that dropout can formally be shown to be an adaptive form of L2 regularization, but the L2 penalty on different ways are different depending on the size of the activation is being multiplied into that way. But to summarize it is possible to show that dropout has a similar effect to. L2 regularization. Only the L2 regularization applied to different ways can be a little bit different and even more adaptive to the scale of different inputs. One more detail for when you're implementing dropout, here's a network where you have three input features. This is seven hidden units here. 7, 3, 2, 1, so one of the practice we have to choose was the keep prop which is a chance of keeping a unit in each layer. So it is also feasible to vary keep-propped by layer. So for the first layer, your matrix W1 will be 7 by 3. Your second weight matrix will be 7 by 7. W3 will be 3 by 7 and so on. And so W2 is actually the biggest weight matrix, right? Because they're actually the largest set of parameters. B and W2, which is 7 by 7. So to prevent, to reduce overfitting of that matrix, maybe for this layer, I guess this is layer 2, you might have a key prop that's relatively low, say 0.5, whereas for different layers where you might worry less about over 15, you could have a higher key problem. Maybe just 0.7, maybe this is 0.7. And then for layers we don't worry about overfitting at all. You can have a key prop of 1.0. Right? So, you know, for clarity, these are numbers I'm drawing in the purple boxes. These could be different key props for different layers. Notice that the key problem 1.0 means that you're keeping every unit. And so you're really not using drop out for that layer. But for layers where you're more worried about overfitting really the layers with a lot of parameters you could say keep prop to be smaller to apply a more powerful form of dropout. It's kind of like cranking up the regularization. Parameter lambda of L2 regularization where you try to regularize some layers more than others. And technically you can also apply drop out to the input layer where you can have some chance of just acting out one or more of the input features, although in practice, usually don't do that often. And so key problem of 1.0 is quite common for the input there. You might also use a very high value, maybe 0.9 but is much less likely that you want to eliminate half of the input features so usually keep prop. If you apply that all will be a number close to 1. If you even apply dropout at all to the input layer. So just to summarize if you're more worried about some layers of fitting than others, you can set a lower key prop for some layers than others. The downside is this gives you even more hyper parameters to search for using cross validation. One other alternative might be to have some layers where you apply dropout and some layers where you don't apply drop out and then just have one hyper parameter which is a key prop for the layers for which you do apply drop out and before we wrap up just a couple implantation all tips. Many of the first successful implementations of dropouts were to computer vision, so in computer vision, the input sizes so big in putting all these pixels that you almost never have enough data. And so drop out is very frequently used by the computer vision and there are some common vision research is that pretty much always use it almost as a default. But really, the thing to remember is that drop out is a regularization technique, it helps prevent overfitting. And so unless my avram is overfitting, I wouldn't actually bother to use drop out. So as you somewhat less often in other application areas, there's just a computer vision, you usually just don't have enough data so you almost always overfitting, which is why they tend to be some computer vision researchers swear by drop out by the intuition. I was, doesn't always generalize, I think to other disciplines. One big downside of drop out is that the cost function J is no longer well defined on every iteration. You're randomly, calling off a bunch of notes. And so if you are double checking the performance of great inter sent is actually harder to double check that, right? You have a well defined cost function J. That is going downhill on every elevation because the cost function J. That you're optimizing is actually less. Less well defined or it's certainly hard to calculate. So you lose this debugging tool to have a plot a draft like this. So what I usually do is turn off drop out or if you will set keep-propped = 1 and run my code and make sure that it is monitored quickly decreasing J. And then turn on drop out and hope that, I didn't introduce, welcome to my code during drop out because you need other ways, I guess, but not plotting these figures to make sure that your code is working, the greatest is working even with drop out. So with that there's still a few more regularization techniques that were feel knowing. Let's talk about a few more such techniques in the next video.
