About this experiment
Training a neural network means walking downhill on a loss surface. Gradient descent takes a step against the gradient, scaled by the learning rate. That simple rule hides a lot of behaviour: too small a step crawls, too large a step oscillates or blows up, narrow valleys make it zigzag, and it can get stuck in a local minimum. Pick a surface, click a starting point and compare plain gradient descent with momentum and Adam, step by step, on the map and on the loss curve.
Learning goals
- Write the gradient descent update and explain the role of the learning rate
- Derive the stability limit η < 2/λ_max for a quadratic loss and test it
- Explain why ill-conditioned valleys make gradient descent zigzag and how momentum helps
- Describe how Adam rescales each coordinate using running averages of the gradient
- Recognise convergence to a local rather than a global minimum
How to use it
- 1Choose Round bowl with gradient descent. Try learning rates of 0.1, 1 and 2.1. Describe the three paths.
- 2Choose Narrow valley. Find the largest learning rate that still converges and compare it with the stable limit shown under the slider.
- 3In the narrow valley set η to 0.05. Count the steps gradient descent needs to reach a loss of 1e-6, then switch to Momentum with β = 0.9 and count again.
- 4Choose Banana valley (the Rosenbrock function). Compare gradient descent, Momentum and Adam after 300 steps.
- 5Choose Two minima. Gradient descent from the default start stops in the shallow basin on the right. Click starting points on both sides of the ridge: which ones reach the deeper basin?
- 6Go back to the default start and switch to Momentum with β = 0.9 and η = 0.1. Why does the path now roll over the ridge?
The model
Assumptions
- Gradients are exact. In real training they are estimated from mini-batches, which adds noise (stochastic gradient descent).
- The loss surfaces are fixed two-variable functions: round bowl ½(x² + y²), narrow valley ½(0.5x² + 5y²), Rosenbrock (1 − x)² + 10(y − x²)², and a tilted double well.
- Every run stops after 300 steps with a fixed learning rate. There are no learning rate schedules.
- Runs are deterministic: the same start and settings always give the same path.
Try this in class
- Q1In the narrow valley the stable limit is set by the steep direction, but progress is limited by the shallow one. Explain using the factors 1 − ηλ.
- Q2Why can momentum converge faster even though each step uses the same gradient?
- Q3Adam takes steps of roughly size η regardless of how steep the surface is. When is that an advantage, and when a problem?
- Q4Neural network losses have millions of dimensions. Do you expect local minima or saddle points to be the bigger obstacle? Why?
- Q5Suggest a learning rate schedule that would combine fast early progress with accurate convergence.
Frequently asked questions
Why does a learning rate that is too large make the loss grow?
Each step overshoots the bottom of the valley and lands higher on the opposite wall. On a quadratic the overshoot multiplies the distance by |1 − ηλ| every step, which is more than one when η is larger than 2/λ.
What does momentum do in a narrow valley?
The zigzag gradients across the valley point in opposite directions and cancel in the running velocity, while the small gradients along the valley all point the same way and add up. The path straightens out and speeds up along the valley floor.
Is Adam always better than gradient descent?
No. Adam is robust and needs little tuning, which is why it is popular for training networks, but on simple well-conditioned problems plain gradient descent or momentum can converge faster or to better solutions.