Gradient Descent: The Update Rule That Trains Every Model

Every model — a neural network, a regression line, anything that learns from data — is really just a set of internal numbers called parameters, plus a single score called the loss that measures how wrong its current predictions are. Every parameter has a slope: nudge it up or down, and the loss changes. Stack all those slopes into one vector and you get the Gradient. It always points uphill — toward the steepest way to make the loss worse — whether you have 2 parameters or 175 billion.

But knowing the uphill direction isn't the same as knowing what to do with it. The gradient only points up. So what good is that to a model trying to reach the bottom?

Gradient Descent takes "here is uphill" and turns it into "here is your next step downhill." That's the whole idea. Every neural network, every regression model, every fine-tuning run learns using exactly this.

The Update Rule

Here is the entire algorithm, in one line:

θ(t+1) = θ(t) - η · ∇L(θ(t))

Let's break down each piece:

  • θ(t): the current parameters — where you are standing right now.
  • ∇L(θ(t)): the gradient of the loss at that point — the uphill direction.
  • η (eta): the learning rate — how big a step you take.
  • θ(t+1): where you land after the step.

The gradient points uphill — so why not just follow it? Because uphill makes the loss worse, not better. We subtract the gradient because downhill — the exact opposite direction — is where the loss actually shrinks. That one minus sign is the whole idea of "descent."

Verifying It on the Simplest Possible Loss

Let L(θ) = θ². Its gradient is 2θ, and its minimum sits at θ = 0. That's the theory — does it actually happen? If the update rule works, applying it again and again should walk θ toward zero, no matter where you start.

def grad_1d(theta):
    return 2 * theta   # analytic gradient — derived by hand

def gradient_descent_1d(theta_init, lr, n_steps):
    theta = theta_init
    history = [theta]
    for _ in range(n_steps):
        grad = grad_1d(theta)
        theta = theta - lr * grad   # THE update rule
        history.append(theta)
    return history

Starting at θ = 3 with a learning rate of 0.1, the sequence heads straight toward 0, shrinking a little more with every step. No library, no autograd (the tool that would normally compute this gradient for you automatically) — just the update rule, applied on repeat.

Gradient Descent walking θ from 3 toward 0 on the θ² bowl, step by step

The Learning Rate Is Doing More Work Than It Looks Like

η looks like a minor detail — just a small Greek letter at the end of the formula. If the direction is already correct, why should step size matter at all? Because even the right direction can fail you if you jump too far, or not far enough.

  • Too small — the steps crawl. You're still moving the right way, but it could take thousands of extra steps to get somewhere close.
  • Too large — the steps overshoot the minimum, then overshoot back the other way. Instead of settling down, θ bounces back and forth, sometimes growing without end.
  • Just right — steady, shrinking steps that land near the minimum and stay there.

Same update rule, same gradient, three completely different outcomes. The formula is fixed. η is the dial.

Four panels comparing too-small, just-right, too-large, and diverging learning rates

Where Vanilla Gradient Descent Breaks

The θ² bowl is about as friendly as a surface gets — smooth and perfectly symmetric. If the update rule works cleanly here, does that mean it works everywhere? Real loss surfaces are rarely this kind.

Run the same update rule on the Rosenbrock function — a long, curved valley built specifically to stress-test optimizers — and vanilla Gradient Descent visibly struggles. It creeps along the flat floor of the valley, taking tiny steps in roughly the right direction, but needing far more steps than the bowl ever did.

The update rule itself never changes. What changes is how much the shape of the loss surface punishes a fixed step size and a fixed direction. That gap is exactly what momentum, Adam, and every other named optimizer — all more advanced versions of this same core idea — were built to close.

Vanilla Gradient Descent crawling along the Rosenbrock valley floor at three different learning rates

Why This Matters at Any Scale

Surely a model with billions of parameters needs something fundamentally different? It doesn't. θ(t+1) = θ(t) - η∇L(θ(t)) stays exactly the same whether θ has 2 parameters or 175 billion. Every parameter gets its own slope (its partial derivative, to use the formal name), all of them stack into the gradient vector, and the same subtraction happens once per parameter, every single training step.

So did Adam and RMSProp — more advanced optimizers built on top of this same idea — make vanilla Gradient Descent obsolete? Not quite. They just change how η gets scaled and how the direction gets adjusted using past gradients. None of them replace the update rule — they all still subtract a scaled gradient from the current parameters.

Get the Code

The companion notebook derives the update rule, implements vanilla Gradient Descent from scratch with no autograd, and visualizes both the convex bowl and the Rosenbrock failure case: GradientDescent.ipynb.

Summary

Gradient Descent is one line: θ(t+1) = θ(t) - η∇L(θ(t)). The Gradient points uphill, so we subtract it to go downhill. The learning rate η decides whether that step crawls, overshoots, or converges cleanly. On a symmetric bowl like θ², the rule walks straight to the minimum; on a curved, real-world surface like the Rosenbrock function, the same rule holds but crawls, because a fixed step size and fixed direction can't adapt to the terrain. None of this changes with scale — whether θ has 2 parameters or 175 billion, the same subtraction happens once per parameter, every step. Adam, RMSProp, and every other named optimizer only change how η is scaled and how the direction is adjusted; underneath, they still subtract a scaled gradient, exactly like SVD and LoRA still rely on this same update rule when only a few parameters are being trained.

Summary panel: the update rule, convergence on the convex bowl, and the Rosenbrock crawl side by side

Comments

Post a Comment

Popular posts from this blog

The Silent Bug That Makes Your Model Wrong, Not Broken

Image
The Silent Bug That Makes Your Model Wrong, Not Broken Two customers. One is 25, the other is 30 — five years apart. Another pair: 25 and 70 — forty-five years apart. Ask a K-Nearest Neighbors model which pair is more similar, using age and income, and it tells you the 45-year age gap pair is the closer match. No error. No crash. Just a wrong answer, delivered with total confidence. Why It Happens Income ranges from roughly zero to a hundred thousand. Age ranges from zero to about a hundred. When a model measures "distance" between two people, it just does math on the raw numbers — and a $1,000 income difference completely swamps a 45-year age difference, because the numbers are so much bigger to begin with. The model isn't wrong about the arithmetic. It's just never been told these two numbers live on completely different scales. The Fix, in Real Numbers Scale those same two customers pro...
Read more

Why LoRA Works: The SVD Math Behind Efficient Fine-Tuning

Image
We keep talking about Gradient Descent when it comes to training a machine learning model. We calculate the gradient, adjust weights to minimize the loss. That sounds simple enough. But what does it mean to train a large language model with billions of parameters? Gradient descent still works exactly as it should — you compute gradients with respect to every single parameter and store optimizer states for all of them. What's the problem then? It works perfectly, but requires a lot of compute power and tens of gigabytes just to hold the training state. It requires GPUs, possibly a connection to a data center. What if you are a student or a researcher with just a laptop? Do you give up on fine-tuning a model with billions of parameters? What if you could train a model where, instead of modifying billions of weights, you only tune a few parameters — and still get nearly the same result? The answer is Low Rank Approximation . And to understand it, ...
Read more

How Spam Filters Work: Coding Naive Bayes from Scratch

Image
Building a Spam Detector from Scratch: The Naive Bayes Classifier Three billion emails are sent every hour. And nearly half of them are spam. "Congratulations! You won free money. Click here now." If we look at this email, we immediately realize it is spam. But how does the machine realize it? It's an engineering problem . An email has thousands of words, and each word changes the probability of it being spam or not spam. Checking how these words relate to each other is computationally impossible. We need to make a naive assumption to solve it. This assumption leads us to one of the most effective classifiers: The Naive Bayes Classifier . The "Naive" Assumption The Naive Bayes classifier assumes that all features are conditionally independent given the class . What does this mean in simple terms? It assumes that the presence of one word does not affect the probability of another wo...
Read more