Vanishing gradient problem - Misplaced Pages

Machine learning model training problem

Machine learning and data mining
Part of a series on
Paradigms Supervised learning Unsupervised learning Semi-supervised learning Self-supervised learning Reinforcement learning Meta-learning Online learning Batch learning Curriculum learning Rule-based learning Neuro-symbolic AI Neuromorphic engineering Quantum machine learning
Problems Classification Generative modeling Regression Clustering Dimensionality reduction Density estimation Anomaly detection Data cleaning AutoML Association rules Semantic analysis Structured prediction Feature engineering Feature learning Learning to rank Grammar induction Ontology learning Multimodal learning
Supervised learning (classification • regression) Apprenticeship learning Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Fuzzy Expectation–maximization (EM) DBSCAN OPTICS Mean shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE SDL
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection RANSAC k-NN Local outlier factor Isolation forest
Artificial neural network Autoencoder Deep learning Feedforward neural network Recurrent neural network LSTM GRU ESN reservoir computing Boltzmann machine Restricted GAN Diffusion model SOM Convolutional neural network U-Net LeNet AlexNet DeepDream Neural radiance field Transformer Vision Mamba Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
Reinforcement learning Q-learning SARSA Temporal difference (TD) Multi-agent Self-play
Learning with humans Active learning Crowdsourcing Human-in-the-loop RLHF
Model diagnostics Coefficient of determination Confusion matrix Learning curve ROC curve
Mathematical foundations Kernel machines Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Journals and conferences ECML PKDD NeurIPS ICML ICLR IJCAI ML JMLR
Related articles Glossary of artificial intelligence List of datasets for machine-learning research List of datasets in computer vision and image processing Outline of machine learning
v t e

In machine learning, the vanishing gradient problem is encountered when training neural networks with gradient-based learning methods and backpropagation. In such methods, during each training iteration, each neural network weight receives an update proportional to the partial derivative of the loss function with respect to the current weight. The problem is that as the network depth or sequence length increases, the gradient magnitude typically is expected to decrease (or grow uncontrollably), slowing the training process. In the worst case, this may completely stop the neural network from further learning. As one example of this problem, traditional activation functions such as the hyperbolic tangent function have gradients in the range [-1,1], and backpropagation computes gradients using the chain rule. This has the effect of multiplying n of these small numbers to compute gradients of the early layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n while the early layers train very slowly.

Backpropagation allowed researchers to train supervised deep artificial neural networks from scratch, initially with little success. Hochreiter's diplom thesis of 1991 formally identified the reason for this failure in the "vanishing gradient problem", which not only affects many-layered feedforward networks, but also recurrent networks. The latter are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time-step of an input sequence processed by the network (the combination of unfolding and backpropagation is termed backpropagation through time).

When activation functions are used whose derivatives can take on larger values, one risk is encountering the related exploding gradient problem.

Prototypical models

This section is based on the paper On the difficulty of training Recurrent Neural Networks by Pascanu, Mikolov, and Bengio.

Recurrent network model

A generic recurrent network has hidden states $h_{1},h_{2},...$ inputs $u_{1},u_{2},...$ , and outputs $x_{1},x_{2},...$ . Let it be parametrized by $\theta$ , so that the system evolves as $(h_{t},x_{t})=F(h_{t-1},u_{t},\theta )$ Often, the output $x_{t}$ is a function of $h_{t}$ , as some $x_{t}=G(h_{t})$ . The vanishing gradient problem already presents itself clearly when $x_{t}=h_{t}$ , so we simplify our notation to the special case with: $x_{t}=F(x_{t-1},u_{t},\theta )$ Now, take its differential: ${\begin{aligned}dx_{t}&=\nabla _{\theta }F(x_{t-1},u_{t},\theta )d\theta +\nabla _{x}F(x_{t-1},u_{t},\theta )dx_{t-1}\\&=\nabla _{\theta }F(x_{t-1},u_{t},\theta )d\theta +\nabla _{x}F(x_{t-1},u_{t},\theta )(\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )d\theta +\nabla _{x}F(x_{t-2},u_{t-1},\theta )dx_{t-2})\\&=\cdots \\&=\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)d\theta \end{aligned}}$ Training the network requires us to define a loss function to be minimized. Let it be $L(x_{T},u_{1},...,u_{T})$ , then minimizing it by gradient descent gives

dL=\nabla _{x}L(x_{T},u_{1},...,u_{T})\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)d\theta

(loss differential)

$\Delta \theta =-\eta \cdot \left^{T}$ where $\eta$ is the learning rate.

The vanishing/exploding gradient problem appears because there are repeated multiplications, of the form $\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{x}F(x_{t-2},u_{t-1},\theta )\nabla _{x}F(x_{t-3},u_{t-2},\theta )\cdots$

Example: recurrent network with sigmoid activation

For a concrete example, consider a typical recurrent network defined by

$x_{t}=F(x_{t-1},u_{t},\theta )=W_{rec}\sigma (x_{t-1})+W_{in}u_{t}+b$ where $\theta =(W_{rec},W_{in})$ is the network parameter, $\sigma$ is the sigmoid activation function, applied to each vector coordinate separately, and $b$ is the bias vector.

Then, $\nabla _{x}F(x_{t-1},u_{t},\theta )=W_{rec}\mathop {diag} (\sigma '(x_{t-1}))$ , and so ${\begin{aligned}\nabla _{x}F(x_{t-1},u_{t},\theta )&\nabla _{x}F(x_{t-2},u_{t-1},\theta )\cdots \nabla _{x}F(x_{t-k},u_{t-k+1},\theta )\\=W_{rec}\mathop {diag} (\sigma '(x_{t-1}))&W_{rec}\mathop {diag} (\sigma '(x_{t-2}))\cdots W_{rec}\mathop {diag} (\sigma '(x_{t-k}))\end{aligned}}$ Since $|\sigma '|\leq 1$ , the operator norm of the above multiplication is bounded above by $\|W_{rec}\|^{k}$ . So if the spectral radius of $W_{rec}$ is $\gamma <1$ , then at large $k$ , the above multiplication has operator norm bounded above by $\gamma ^{k}\to 0$ . This is the prototypical vanishing gradient problem.

The effect of a vanishing gradient is that the network cannot learn long-range effects. Recall Equation (loss differential): $\nabla _{\theta }L=\nabla _{x}L(x_{T},u_{1},...,u_{T})\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)$ The components of $\nabla _{\theta }F(x,u,\theta )$ are just components of $\sigma (x)$ and $u$ , so if $u_{t},u_{t-1},...$ are bounded, then $\|\nabla _{\theta }F(x_{t-k-1},u_{t-k},\theta )\|$ is also bounded by some $M>0$ , and so the terms in $\nabla _{\theta }L$ decay as $M\gamma ^{k}$ . This means that, effectively, $\nabla _{\theta }L$ is affected only by the first $O(\gamma ^{-1})$ terms in the sum.

If $\gamma \geq 1$ , the above analysis does not quite work. For the prototypical exploding gradient problem, the next model is clearer.

Dynamical systems model

Following (Doya, 1993), consider this one-neuron recurrent network with sigmoid activation: $x_{t+1}=(1-\epsilon )x_{t}+\epsilon \sigma (wx_{t}+b)+\epsilon w'u_{t}$ At the small $\epsilon$ limit, the dynamics of the network becomes ${\frac {dx}{dt}}=-x(t)+\sigma (wx(t)+b)+w'u(t)$ Consider first the autonomous case, with $u=0$ . Set $w=5.0$ , and vary $b$ in $[- 3, - 2]$ . As $b$ decreases, the system has 1 stable point, then has 2 stable points and 1 unstable point, and finally has 1 stable point again. Explicitly, the stable points are $(x,b)=\left(x,\ln \left({\frac {x}{1-x}}\right)-5x\right)$ .

Now consider ${\frac {\Delta x(T)}{\Delta x(0)}}$ and ${\frac {\Delta x(T)}{\Delta b}}$ , where $T$ is large enough that the system has settled into one of the stable points.

If $(x(0),b)$ puts the system very close to an unstable point, then a tiny variation in $x(0)$ or $b$ would make $x(T)$ move from one stable point to the other. This makes ${\frac {\Delta x(T)}{\Delta x(0)}}$ and ${\frac {\Delta x(T)}{\Delta b}}$ both very large, a case of the exploding gradient.

If $(x(0),b)$ puts the system far from an unstable point, then a small variation in $x(0)$ would have no effect on $x(T)$ , making ${\frac {\Delta x(T)}{\Delta x(0)}}=0$ , a case of the vanishing gradient.

Note that in this case, ${\frac {\Delta x(T)}{\Delta b}}\approx {\frac {\partial x(T)}{\partial b}}=\left({\frac {1}{x(T)(1-x(T))}}-5\right)^{-1}$ neither decays to zero nor blows up to infinity. Indeed, it's the only well-behaved gradient, which explains why early researches focused on learning or designing recurrent networks systems that could perform long-ranged computations (such as outputting the first input it sees at the very end of an episode) by shaping its stable attractors.

For the general case, the intuition still holds ( Figures 3, 4, and 5).

Geometric model

Continue using the above one-neuron network, fixing $w=5,x(0)=0.5,u(t)=0$ , and consider a loss function defined by $L(x(T))=(0.855-x(T))^{2}$ . This produces a rather pathological loss landscape: as $b$ approach $-2.5$ from above, the loss approaches zero, but as soon as $b$ crosses $-2.5$ , the attractor basin changes, and loss jumps to 0.50.

Consequently, attempting to train $b$ by gradient descent would "hit a wall in the loss landscape", and cause exploding gradient. A slightly more complex situation is plotted in, Figures 6.

Solutions

This section has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages)

This section needs additional citations to secondary or tertiary sources. Help add sources such as review articles, monographs, or textbooks. Please also establish the relevance for any primary research articles cited. Unsourced or poorly sourced material may be challenged and removed. (December 2017) (Learn how and when to remove this message)

Some of this section 's listed sources may not be reliable. Please help improve this article by looking for better, more reliable sources. Unreliable citations may be challenged and removed. (December 2017) (Learn how and when to remove this message)

(Learn how and when to remove this message)

To overcome this problem, several methods were proposed.

RNN

For recurrent neural networks, the long short-term memory (LSTM) network was designed to solve the problem (Hochreiter & Schmidhuber, 1997).

For the exploding gradient problem, (Pascanu et al, 2012) recommended gradient clipping, meaning dividing the gradient vector $g$ by $\|g\|/g_{max}$ if $\|g\|>g_{max}$ . This restricts the gradient vectors within a ball of radius $g_{max}$ .

Batch normalization

Batch normalization is a standard method for solving both the exploding and the vanishing gradient problems.

Multi-level hierarchy

In multi-level hierarchy of networks (Schmidhuber, 1992), pre-trained one level at a time through unsupervised learning, fine-tuned through backpropagation. Here each level learns a compressed representation of the observations that is fed to the next level.

Deep belief network

Similar ideas have been used in feed-forward neural networks for unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised backpropagation to classify labeled data. The deep belief network model by Hinton et al. (2006) involves learning the distribution of a high-level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations. Hinton reports that his models are effective feature extractors over high-dimensional, structured data.

Faster hardware

Hardware advances have meant that from 1991 to 2015, computer power (especially as delivered by GPUs) has increased around a million-fold, making standard backpropagation feasible for networks several layers deeper than when the vanishing gradient problem was recognized. Schmidhuber notes that this "is basically what is winning many of the image recognition competitions now", but that it "does not really overcome the problem in a fundamental way" since the original models tackling the vanishing gradient problem by Hinton and others were trained in a Xeon processor, not GPUs.

Residual connection

Residual connections, or skip connections, refers to the architectural motif of $x\mapsto f(x)+x$ , where $f$ is an arbitrary neural network module. This gives the gradient of $\nabla f+I$ , where the identity matrix do not suffer from the vanishing or exploding gradient. During backpropagation, part of the gradient flows through the residual connections.

Concretely, let the neural network (without residual connections) be $f_{n}\circ f_{n-1}\circ \cdots \circ f_{1}$ , then with residual connections, the gradient of output with respect to the activations at layer $l$ is $I+\nabla f_{l+1}+\nabla f_{l+2}\nabla f_{l+1}+\cdots$ . The gradient thus does not vanish in arbitrarily deep networks.

Feedforward networks with residual connections can be regarded as an ensemble of relatively shallow nets. In this perspective, they resolve the vanishing gradient problem by being equivalent to ensembles of many shallow networks, for which there is no vanishing gradient problem.

Other activation functions

Rectifiers such as ReLU suffer less from the vanishing gradient problem, because they only saturate in one direction.

Weight initialization

Weight initialization is another approach that has been proposed to reduce the vanishing gradient problem in deep networks.

Kumar suggested that the distribution of initial weights should vary according to activation function used and proposed to initialize the weights in networks with the logistic activation function using a Gaussian distribution with a zero mean and a standard deviation of 3.6/sqrt(N), where N is the number of neurons in a layer.

Recently, Yilmaz and Poli performed a theoretical analysis on how gradients are affected by the mean of the initial weights in deep neural networks using the logistic activation function and found that gradients do not vanish if the mean of the initial weights is set according to the formula: max(−1,-8/N). This simple strategy allows networks with 10 or 15 hidden layers to be trained very efficiently and effectively using the standard backpropagation.

Other

Behnke relied only on the sign of the gradient (Rprop) when training his Neural Abstraction Pyramid to solve problems like image reconstruction and face localization.

Neural networks can also be optimized by using a universal search algorithm on the space of neural network's weights, e.g., random guess or more systematically genetic algorithm. This approach is not based on gradient and avoids the vanishing gradient problem.

Notes

A more general loss function could depend on the entire sequence of outputs, as $L(x_{1},...,x_{T},u_{1},...,u_{T})=\sum _{t=1}^{T}{\mathcal {E}}(x_{t},u_{1},...,u_{t})$ for which the problem is the same, just with more complex notations.
Any activation function works, as long as it is differentiable with bounded derivative.
Consider $W_{rec}={\begin{bmatrix}0&2\\\epsilon &0\end{bmatrix}}$ and $D={\begin{bmatrix}c&0\\0&c\end{bmatrix}}$ , with $\epsilon >{\frac {1}{2}}$ and $c\in (0,1)$ . Then $W_{rec}$ has spectral radius ${\sqrt {2\epsilon }}>1$ , and $(W_{rec}D)^{2N}=(2\epsilon \cdot c^{2})^{N}I_{2\times 2}$ , which might go to infinity or zero depending on choice of $c$ .
This is because at $b=-2.5$ , the two stable attractors are $x=0.145,0.855$ , and the unstable attractor is $x=0.5$ .