In machine learning, the vanishing gradient problem is encountered when training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, during each iteration of training each of the neural networks weights receives an update proportional to the partial derivative of the error function with respect to the current weight. The problem is that in some cases, the gradient will be vanishingly small, effectively preventing the weight from changing its value. In the worst case, this may completely stop the neural network from further training. As one example of the problem cause, traditional activation functions such as the hyperbolic tangent function have gradients in the range , and backpropagation computes gradients by the chain rule. This has the effect of multiplying of these small numbers to compute gradients of the early layers in an -layer network, meaning that the gradient (error signal) decreases exponentially with while the early layers train very slowly.
Back-propagation allowed researchers to train supervised deep artificial neural networks from scratch, initially with little success. Hochreiter's diplom thesis of 1991 formally identified the reason for this failure in the "vanishing gradient problem", which not only affects many-layered feedforward networks, but also recurrent networks. The latter are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time step of an input sequence processed by the network. (The combination of unfolding and backpropagation is termed backpropagation through time.)
When activation functions are used whose derivatives can take on larger values, one risks encountering the related exploding gradient problem.
Prototypical models
This section is based on the paper On the difficulty of training Recurrent Neural Networks by Pascanu, Mikolov, and Bengio.
Recurrent network model
A generic recurrent network has hidden states inputs , and outputs . Let it be parametrized by , so that the system evolves asOften, the output is a function of , as some . The vanishing gradient problem already presents itself clearly when , so we simplify our notation to the special case with: Now, take its differential:Training the network requires us to define a loss function to be minimized. Let it be , then minimizing it by gradient descent giveswhere is the learning rate.
The vanishing/exploding gradient problem appears because there are repeated multiplications, of the form
Example: recurrent network with sigmoid activation
For a concrete example, consider a typical recurrent network defined by
where is the network parameter, is the sigmoid activation function, applied to each vector coordinate separately, and is the bias vector.
Then, , and so Since , the operator norm of the above multiplication is bounded above by . So if the spectral radius of is , then at large , the above multiplication has operator norm bounded above by . This is the prototypical vanishing gradient problem.
The effect of a vanishing gradient is that the network cannot learn long-range effects. Recall Equation ():The components of are just components of and , so if are bounded, then is also bounded by some , and so the terms in decay as . This means that, effectively, is affected only by the first terms in the sum.
If , the above analysis does not quite work. For the prototypical exploding gradient problem, the next model is clearer.
Dynamical systems model
Following (Doya, 1993), consider this one-neuron recurrent network with sigmoid activation:At the small limit, the dynamics of the network becomesConsider first the autonomous case, with . Set , and vary in . As decreases, the system has 1 stable point, then has 2 stable points and 1 unstable point, and finally has 1 stable point again. Explicitly, the stable points are .
Now consider and , where is large enough that the system has settled into one of the stable points.
If puts the system very close to an unstable point, then a tiny variation in or would make move from one stable point to the other. This makes and both very large, a case of the exploding gradient.
If puts the system far from an unstable point, then a small variation in would have no effect on , making , a case of the vanishing gradient.