Picture this: someone blindfolds you halfway up a hillside and asks you to find the lowest point. You cannot see a thing. All you can do is feel which way the ground tilts under your shoes, take one step, and feel again.

That exact walk is what every machine learning model performs, over and over, while it is being trained. Its hillside is just built out of numbers instead of rocks and grass. By the end of this post you will understand that walk well enough to read it in code, not just picture it.

A blindfolded figure using a walking stick descends a dark hillside at night, following a trail of footprints toward a glowing marker at the valley floor

What a model is actually doing when it "learns"

Say you are building a model to guess house prices. You feed it a house, and it spits out a number: maybe $300,000. The real price was $340,000. Your model was wrong by $40,000.

That wrongness has a name: a loss function (sometimes called a cost function). It is one number that says how bad the current guess is. Zero means a perfect guess.

A model's guess is controlled by adjustable numbers inside it, called parameters or weights. Learning means searching for parameter values that make the loss as small as possible, across every example the model has seen.

Plot the loss as a graph, with height meaning "how wrong," and you get an imaginary hillside. The lowest point is the best the model can do. The slope under your feet, which direction makes the loss grow fastest and by how much, is called the gradient.

Put it together: gradient descent means feel the slope, step downhill, feel again, repeat, until you cannot go down any further. In one line of pseudocode:

parameters = parameters - step_size * gradient(loss, parameters)

Why step size changes everything

Tiny steps are safe but slow. Giant steps can overshoot the bottom entirely and land you worse off than where you started, then overshoot again on the way back.

Line chart comparing three learning rates in gradient descent: a small step size crawling slowly toward zero, a well chosen step size settling quickly, and a large step size bouncing to larger and larger values with each step

The chart above is a real, computed run starting five units from the bottom. A step size of 0.05 is still almost two units away after ten steps. A step size of 0.30 settles near zero in about five steps. A step size of 1.05 never settles, it swings further out with every step.

Here is a tiny, runnable version of the "just right" line from that chart, using plain f(x) = x^2 as the loss:

def loss(x): return x ** 2 def gradient(x): return 2 * x x = 5.0 step_size = 0.3 for step in range(1, 7): x = x - step_size * gradient(x) print(f"step {step}: x = {x:.4f}, loss = {loss(x):.4f}")

Real output:

step 1: x = 2.0000, loss = 4.0000 step 2: x = 0.8000, loss = 0.6400 step 3: x = 0.3200, loss = 0.1024 step 4: x = 0.1280, loss = 0.0164 step 5: x = 0.0512, loss = 0.0026 step 6: x = 0.0205, loss = 0.0004

Every optimizer you have ever passed to a training loop, SGD, Adam, RMSProp, is some descendant of this same loop, with extra bookkeeping added to make the steps faster or steadier.

I built my students a window into this hill

I teach advanced mathematics and statistics, and gradient descent is one of those ideas that looks simple on a whiteboard and still leaves students stuck. So I animated it in Manim, the same engine behind 3Blue1Brown's videos, building a full visual proof of exactly how fast gradient descent is guaranteed to converge, worked through on a real example.

PLACEHOLDER: a frame from the Manim animation showing the walker descending the valley, captured by Sohaib from his own render

Where the 1/k guarantee comes from

In that video, the walker starts 4 units from the bottom. The hill is smooth: its slope never changes faster than a fixed rate L, here L = 1, so step size 1/L = 1 is mathematically safe.

xnext​=xcurrent​−(step size)×(slope at xcurrent​)

Walking it out: 4, 3, 2, 1, 0, the bottom reached in exactly four steps. The error after each step: 2.5, 1.5, 0.5, 0.

Because the hill is smooth and convex (shaped like a single bowl with one lowest point), there is a proven ceiling on the error after k steps:

error after k steps≤k8​

Chart showing the guaranteed error bound of eight divided by k as a smooth gold curve, with the four actual error values from the worked example, 2.5, 1.5, 0.5 and 0, plotted as dots safely underneath it at each step

That bound decays like "one over k": roughly ten times more steps for ten times more accuracy. This method is not new either. Augustin-Louis Cauchy described it in a short note in 1847, while computing a star's orbit using algebra instead of harder equations by hand.

Where this breaks down

Real loss surfaces are rarely one clean bowl. A model can land in a shallow dip, a local minimum, and stop there even though a deeper valley exists elsewhere.

Picking L exactly, the way we did above, only works because the example was simple. Real step sizes are usually tuned through experiment instead.

And a real model's hill has one direction per parameter, sometimes millions of them, which is exactly why plain gradient descent gets replaced by its smarter cousins at scale.

One line to take with you

Every model that has ever learned anything has done nothing more than take small, careful steps downhill until it could not go any lower, the same walk Cauchy described in 1847, just running in a training loop instead of an astronomer's notebook.

Words used in this post

  • Gradient descent: a method for finding the lowest point of a hill of numbers by repeatedly stepping downhill.
  • Loss function (cost function): a single number that measures how wrong a model's current guess is.
  • Parameter (weight): one of the adjustable numbers inside a model that changes its guesses.
  • Learning rate: how big a step the model takes downhill at each try.
  • Local minimum: a dip in the hill that looks like the bottom but is not the lowest point overall.
  • Convergence: the point where the steps stop improving the guess by any meaningful amount.
  • Convex: shaped like a single smooth bowl, with exactly one lowest point.

Explaining an idea like this to your team or in a paper? If a concept like gradient descent, backpropagation, or any algorithm with moving parts keeps losing people on a static slide or a whiteboard, I build clear Manim animations like the one in this post, every symbol explained, every step visible. The video above is an example of exactly that kind of work. Send me the idea, who will watch it, and where it will be shown, through my Work With Me page.

About the author

I'm Sohaib Hasan. I have 9+ years of teaching experience in advanced mathematics and statistics, and I'm currently a section leader for Stanford's Probability for AI course, selected out of more than 2,700 participants from over 70 countries to teach students from around the world. After each class, I explain the same ideas again through Manim animations, and I'm glad that Stanford professors and fellow section leaders have appreciated them.

I share these animations on my YouTube channel, PlotLab Algorithms, where I teach data science, machine learning, cryptography, math, DSA and statistics visually, in the style made popular by 3Blue1Brown. Shorter clips live on Facebook (85K+ followers) and Instagram (15K+ followers).

Want an animation like this for your own work? If you have a project, a research article, a thesis defense, a YouTube video, or any idea that is hard to explain with slides alone, I can turn it into a clear, professional animation. Reach out through my Work With Me page and tell me what you are working on. If you would rather learn to build animations yourself, start with my Manim CE Beginner's Handbook. And I'd like to hear what you think of this post, drop a comment.