Topic 08 — Gradient Descent & Its Types

1 Lab Introduction

Gradient Descent (GD) is the optimization algorithm neural networks use to minimize a loss function J(θ) by repeatedly moving the parameters (weights & biases) in the direction opposite to the gradient — i.e. “downhill” on the loss surface.

In this lab we train the same network three times on a small purchase-prediction dataset, changing only one thing — the batch_size — and watch how that single choice trades training time against how fast the loss falls. Every GD variant plugs into the same backpropagation skeleton; what changes is only how many points are used before an update is applied.

2 Conceptual Overview

2.1 What is Gradient Descent?

Training repeatedly updates each weight by stepping against the gradient of the loss:

2.2 Where Updates Happen — The Backpropagation Template

Every GD variant plugs into the same backpropagation skeleton — what changes is only how many points are used before an update is applied.

for epoch in range(epochs):
    # Loop over points depends entirely on the chosen GD Variant
    for point in dataset:  # select 1 single row
        prediction = forward_propagate(point)
        loss = calculate_loss(point, prediction)
        gradients = backward_propagate(loss)
        update_weights_and_biases(gradients)
    average_loss = calculate_epoch_average()

2.3 The Variants at a Glance

The variants differ only in how many samples feed each update, which fixes the number of updates per epoch:

More updates per epoch → the loss falls faster per epoch, but more time is spent per epoch. That tension is the whole lab.

3 Types of Gradient Descent

Select a variant to see what data it uses, how the update is performed, and its key characteristics.

import numpy as np

epochs = 10
weights = np.random.randn(input_dim, output_dim)
bias = np.zeros((1, output_dim))
learning_rate = 0.01

for i in range(epochs):
    # Process the ENTIRE dataset simultaneously (e.g. 50 points)
    y_hat = np.dot(X_train, weights) + bias

    # Cumulative loss across ALL samples combined
    loss = calculate_cumulative_loss(y_train, y_hat)

    # Gradient of the GLOBAL loss
    d_weights = compute_global_gradient(X_train, y_train, y_hat)

    # Parameter update happens precisely ONCE per epoch
    weights = weights - (learning_rate * d_weights)
    print(f"Epoch {i+1} complete. One update applied.")

model = build_model()
start = time.time()
fit(model, epochs=10, batch_size=n_train)     # whole training set in one batch
print('Total time:', round(time.time() - start, 2), 's')

epochs = 10
rows = X_train.shape[0]   # e.g. 50 rows

for epoch in range(epochs):
    # Shuffle dataset at the start of the epoch to reduce systemic bias
    shuffled_X, shuffled_y = shuffle_data(X_train, y_train)

    for i in range(rows):
        # Extract a single random point
        x_point = shuffled_X[i:i+1, :]
        y_point = shuffled_y[i:i+1]

        # Single prediction, single loss, instant update
        y_hat_point = np.dot(x_point, weights) + bias
        loss_point = calculate_loss(y_point, y_hat_point)
        d_weights_point = compute_point_gradient(x_point, y_point, y_hat_point)

        # Update triggers 'rows' times per epoch (e.g. 50 times)
        weights = weights - (learning_rate * d_weights_point)

    print(f"Epoch {epoch+1} complete. Total updates this epoch: {rows}")

model = build_model()
start = time.time()
fit(model, epochs=10, batch_size=1)
print('Total time:', round(time.time() - start, 2), 's')

fit(build_model(), epochs=1, batch_size=32)   # warm-up only; ignore this output

3.1 Summary Cheat-Sheet

4 Loss-Curve Geometry & the Golden Nuggets

4.1 The Optimization Path

Zoomed in on a set of contour rings around the minimum, the two algorithms trace very different paths to the target.

4.2 Escaping Local Minima

On a non-convex, rugged loss landscape with several shallow local valleys plus one deep global canyon, the two algorithms behave very differently when they hit a shallow trap.

4.3 The Double Definition of “Fast”

There are two very different meanings of “fast” in this context, and confusing them is a common mistake.

4.4 Vectorization vs. Memory: The 10-Crore-Row Wall

Vectorization with NumPy dot products is very fast because it avoids slow Python-level loops — but it comes with a hardware cost.

• A full-batch matrix dot product requires loading the entire dataset into RAM at once.
• If the training set is massive (e.g. 100 million / 10 crore rows), this immediately causes an Out-Of-Memory (OOM) crash.

5 Theory → Code Connection

Select a concept on the left. The matching lines of the fit() function are highlighted on the right.

def fit(model, epochs, batch_size, lr=0.1):
    optimizer = torch.optim.SGD(model.parameters(), lr=lr)
    criterion = nn.BCELoss()

    for epoch in range(epochs):
        perm = torch.randperm(n_train)                 # reshuffle the rows each epoch
        epoch_loss, n_batches = 0.0, 0
        for i in range(0, n_train, batch_size):        # step through in batch_size chunks
            idx = perm[i:i + batch_size]               # indices of this batch
            xb, yb = X_train_t[idx], y_train_t[idx]    # slice out one batch by hand

            optimizer.zero_grad()             # 1. clear old gradients
            loss = criterion(model(xb), yb)   # 2+3. forward pass + loss L
            loss.backward()                   # 4. backprop -> computes dL/dw
            optimizer.step()                  # 5. update: W_n = W_0 - lr * dL/dw

            epoch_loss += loss.item()         # accumulate to report the epoch average
            n_batches += 1

        print(f'Epoch {epoch + 1:2d}/{epochs} - loss: {epoch_loss / n_batches:.4f}')

6 Code Walkthrough

The implementation is broken here into logical sections. Each block shows the code, what it does, and how it connects to gradient descent.

6.1 Imports and Reproducibility

import numpy as np
import pandas as pd
import time
import torch
import torch.nn as nn
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

torch.manual_seed(42)
np.random.seed(42)

What happens: the tensor library (torch), the layer module (torch.nn), a timer (time) and two scikit-learn utilities are imported. Both random seeds are fixed.

Connection to gradient descent: the seeds make weight initialisation and the per-epoch shuffle reproducible, so that when we later change only batch_size the comparison between the regimes is fair. time is what lets us measure the “speed per epoch” half of the double definition of fast (Section 4.3).

6.2 Data — Purchase Prediction

Predict Purchased (0/1) from Age and EstimatedSalary.

url = 'https://raw.githubusercontent.com/shivang98/Social-Network-ads-Boost/master/Social_Network_Ads.csv'
df = pd.read_csv(url)[['Age', 'EstimatedSalary', 'Purchased']]
df.head()

   Age  EstimatedSalary  Purchased
0   19            19000          0
1   35            20000          0
2   26            43000          0
3   27            57000          0
4   19            76000          0

What happens: a small binary-classification table with two numeric features is loaded from a URL.

Connection to gradient descent: this table is the “dataset” of the backpropagation template (Section 2.2) — the collection of points over which every epoch iterates.

6.3 Split, then Scale

X = df[['Age', 'EstimatedSalary']].values
y = df['Purchased'].values

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=2)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)   # fit on train
X_val   = scaler.transform(X_val)         # transform val only

What happens: 20 % of the rows are held out for validation; the scaler learns its mean and standard deviation from the training rows only and is then applied to both sets.

Connection to gradient descent: Age (tens) and EstimatedSalary (tens of thousands) live on wildly different scales. Standardising them puts both features on comparable footing before the same learning rate η is applied to every weight.

6.4 Tensors and the Row Count N

X_train_t = torch.tensor(X_train, dtype=torch.float32)
y_train_t = torch.tensor(y_train, dtype=torch.float32).reshape(-1, 1)
X_val_t   = torch.tensor(X_val,   dtype=torch.float32)
y_val_t   = torch.tensor(y_val,   dtype=torch.float32).reshape(-1, 1)

n_train = X_train_t.shape[0]   # number of training rows
print('training rows:', n_train)

training rows: 320

What happens: the arrays become float32 tensors, labels are reshaped into a column, and n_train records the number of training rows: 320.

Connection to gradient descent: n_train is the N of the theory. It is the number of updates SGD performs per epoch, the batch size that turns fit() into Batch GD, and the numerator of N/B for mini-batch.

6.5 The Model

A small network: 2 → 10 → 10 → 1 with a sigmoid output. build_model() returns fresh weights so each of the three runs starts from the same point (a fair comparison).

def build_model():
    return nn.Sequential(
        nn.Linear(2, 10), nn.ReLU(),
        nn.Linear(10, 10), nn.ReLU(),
        nn.Linear(10, 1), nn.Sigmoid()
    )

What happens: three nn.Linear layers stacked with ReLU activations and a final sigmoid squashing the output into (0, 1) for binary classification.

Connection to gradient descent: the weights and biases inside those nn.Linear layers are the parameters θ that gradient descent updates. Rebuilding the model before each run is what keeps the Batch-vs-SGD comparison honest: both regimes start from the same point on the loss surface.

6.6 The Training Function

Keras’ model.fit() hides the training loop. Here we write that loop ourselves in a small fit() — so you can see exactly what happens on every batch. The number of times the inner loop runs per epoch (= updates per epoch) is set purely by batch_size.

def fit(model, epochs, batch_size, lr=0.1):
    optimizer = torch.optim.SGD(model.parameters(), lr=lr)
    criterion = nn.BCELoss()

    for epoch in range(epochs):
        perm = torch.randperm(n_train)                 # reshuffle the rows each epoch
        epoch_loss, n_batches = 0.0, 0
        for i in range(0, n_train, batch_size):        # step through in batch_size chunks
            idx = perm[i:i + batch_size]               # indices of this batch
            xb, yb = X_train_t[idx], y_train_t[idx]    # slice out one batch by hand

            optimizer.zero_grad()             # 1. clear old gradients
            loss = criterion(model(xb), yb)   # 2+3. forward pass + loss L
            loss.backward()                   # 4. backprop -> computes dL/dw
            optimizer.step()                  # 5. update: W_n = W_0 - lr * dL/dw

            epoch_loss += loss.item()         # accumulate to report the epoch average
            n_batches += 1

        print(f'Epoch {epoch + 1:2d}/{epochs} - loss: {epoch_loss / n_batches:.4f}')

What happens: the outer loop is one epoch; the rows are reshuffled with torch.randperm; the inner loop walks the shuffled indices in batch_size chunks, and each chunk performs the five numbered steps: clear gradients, forward pass, loss, backward pass, update. The batch losses are averaged and printed once per epoch.

Connection to gradient descent: this single function is all three variants at once. Because the inner loop is range(0, n_train, batch_size), it executes n_train / batch_size times — exactly the “updates per epoch” column of the theory table. Passing n_train gives 1 update (Batch GD), passing 1 gives N updates (SGD), passing 32 gives N/32 (Mini-batch).

6.7 Warm-up Run

A one-time warm-up run so PyTorch’s start-up cost doesn’t unfairly land on the first timed run below — ignore its printed loss.

fit(build_model(), epochs=1, batch_size=32)   # warm-up only; ignore this output

Epoch  1/1 - loss: 0.6831

Connection to gradient descent: this is methodological hygiene for the timing measurement, not a result. It is, incidentally, the only mini-batch call in Part 1.

6.8 Run 1 — Batch GD

batch_size = all rows → 1 update per epoch.

model = build_model()
start = time.time()
fit(model, epochs=10, batch_size=n_train)     # whole training set in one batch
print('Total time:', round(time.time() - start, 2), 's')

Epoch  1/10 - loss: 0.6661
Epoch  2/10 - loss: 0.6615
Epoch  3/10 - loss: 0.6571
Epoch  4/10 - loss: 0.6527
Epoch  5/10 - loss: 0.6483
Epoch  6/10 - loss: 0.6440
Epoch  7/10 - loss: 0.6397
Epoch  8/10 - loss: 0.6355
Epoch  9/10 - loss: 0.6312
Epoch 10/10 - loss: 0.6269
Total time: 0.0 s

What happens: ten epochs, ten weight updates in total. The loss drifts from 0.6661 to 0.6269 — smooth, monotonic, and small. Wall-clock time rounds to 0.0 s.

Connection to gradient descent: this is the theory’s “1 Epoch = 1 Weight Update” rule made visible. The path is smooth and stable (Section 4.1) but the model has taken only ten steps down the loss surface, so it has barely moved.

6.9 Run 2 — Stochastic GD

batch_size = 1 → one update per sample.

model = build_model()
start = time.time()
fit(model, epochs=10, batch_size=1)
print('Total time:', round(time.time() - start, 2), 's')

Epoch  1/10 - loss: 0.3740
Epoch  2/10 - loss: 0.2825
Epoch  3/10 - loss: 0.2760
Epoch  4/10 - loss: 0.2648
Epoch  5/10 - loss: 0.2551
Epoch  6/10 - loss: 0.2589
Epoch  7/10 - loss: 0.2590
Epoch  8/10 - loss: 0.2543
Epoch  9/10 - loss: 0.2454
Epoch 10/10 - loss: 0.2473
Total time: 0.59 s

What happens: ten epochs, but 320 updates in each — 3 200 updates in total. The loss is already at 0.3740 after the first epoch and settles around 0.25. Wall-clock time is 0.59 s.

Connection to gradient descent: two theoretical claims are visible in one output. First, “needs far fewer epochs” — one epoch of SGD achieves what Batch GD could not in ten. Second, the noise: the loss rises at epochs 6, 7 and 10 (0.2551 → 0.2589, 0.2543 → 0.2454 → 0.2473). That is the jagged zig-zag of Fig. 2 showing up in the printed numbers.

6.10 Runs 3 & 4 — the Same Comparison over 100 Epochs

Both regimes are re-run with a freshly built model, changing only epochs=10 to epochs=100. Everything else — network, learning rate, data — is identical.

Run 3 — Batch GD, 100 epochs

model = build_model()
start = time.time()
fit(model, epochs=100, batch_size=n_train)    # still 1 update per epoch
print('Total time:', round(time.time() - start, 2), 's')

Epoch  1/100 - loss: 0.7077
Epoch  2/100 - loss: 0.6983
Epoch  3/100 - loss: 0.6897
   ...
Epoch 50/100 - loss: 0.4412
   ...
Epoch 98/100 - loss: 0.3210
Epoch 99/100 - loss: 0.3199
Epoch 100/100 - loss: 0.3188
Total time: 0.02 s

Run 4 — Stochastic GD, 100 epochs

model = build_model()
start = time.time()
fit(model, epochs=100, batch_size=1)          # still one update per sample
print('Total time:', round(time.time() - start, 2), 's')

Epoch  1/100 - loss: 0.4020
Epoch  2/100 - loss: 0.2719
Epoch  3/100 - loss: 0.2682
   ...
Epoch 50/100 - loss: 0.2301
   ...
Epoch 98/100 - loss: 0.2263
Epoch 99/100 - loss: 0.2260
Epoch 100/100 - loss: 0.2325
Total time: 5.46 s

What happens: Batch GD now performs 100 updates and reports a readable 0.02 s; Stochastic GD performs 100 × 320 = 32,000 updates and takes 5.46 s.

Connection to gradient descent: the longer run sharpens both halves of the tradeoff.

• Convergence. After a full 100 epochs Batch GD reaches 0.3188 — a loss Stochastic GD had already beaten after its second epoch (0.2719). “Needs many epochs to converge” is no longer a prediction; it is measured.
• Cost. Both regimes push the same amount of data through the network (100 × 320 = 32,000 row-passes each), yet SGD takes roughly 240× longer. The extra time is not extra arithmetic — it is the price of doing that arithmetic in 32,000 tiny Python-level steps instead of 100 vectorized matrix operations. This is the “Speed PER epoch” row of the cheat-sheet, in seconds.
• Noise does not disappear. Stochastic GD flattens out around 0.22–0.23 and keeps bouncing to the very end (0.2260 → 0.2325). Batch GD’s curve stays perfectly smooth throughout.

7 Training-Loop Visualisation

The five numbered steps inside fit(), in order. Click a step to see the exact line of code and what it does.

8 Experiment / Playground

fit() exposes exactly three experimental knobs: batch_size, epochs and lr. Move them below to see how the loop structure changes. The counts are computed from the loop itself — for i in range(0, n_train, batch_size) with n_train = 320.

Bonus tasks — early finishers

• Re-run all three with lr=0.01 — does Batch GD fall even further behind?
• Add a batch_size=8 run — where do its time and loss land relative to the others?
• Print loss.item() for the very first and last epoch of each run and tabulate the drop by hand.

9 Key Observations

Each pair of runs used the same epoch count — 10 in Sections 6.8–6.9 and 100 in Section 6.10 — which lets us answer two separate questions. Use the toggle below the legend to switch between the two experiments.

9.1 The Recorded Loss Curves

9.2 The Tradeoff

• Question A — Which is faster? (wall-clock time): Batch GD does 1 update per epoch → finishes quickest (0.0 s). Stochastic does one update per sample (hundreds per epoch) → clearly the slowest (0.59 s).
• Question B — For the same number of epochs, which converges faster?: the opposite order. Stochastic drops the loss sharply within the first epoch or two (0.6831 → 0.3740 → 0.2825); Batch GD, with a single update per epoch, barely moves in 10 epochs (0.6661 → 0.6269).

9.3 Observations Tied Back to the Theory

10 Student Checkpoints

Short questions on the material of this lab only. Answers are checked immediately.

10.1 Written Questions

1. Stochastic GD was slowest in seconds yet dropped the loss fastest, and Batch GD the reverse. Explain both using updates per epoch.
2. Mini-batch GD (e.g. batch_size=32) was not run above. From its updates per epoch, where would its total time and loss-drop speed fall relative to the two extremes — and why is it the usual choice in practice?
3. A classmate says “SGD is always best because its loss fell fastest.” Give one reason that’s misleading.

11 Lab Takeaways