The Math You Actually Need

Assumes you know from ML-101

You have completed ML-101 lesson 3 — Linear Regression, where you met the normal equation; ML-101 lesson 11 — Gradient Descent & Optimization, where you saw that $\nabla f(x)$ points uphill; and ML-101 lesson 14 — Neural Networks & Backprop, where the chain rule $\partial L / \partial w = \partial L / \partial a \cdot \partial a / \partial w$ was applied without comment. This lesson does not re-derive those facts from the foundations of real analysis. Instead it re-states them in the language that the rest of ML-201 will use: linear maps between finite-dimensional spaces, Jacobians as the matrix form of the chain rule, and gradients as the Riesz representer of the differential. The aim is not to teach new symbols; it is to fix the meaning of the symbols you already use, so that every later derivation in ML-201 can speak in the same vocabulary.

Learning Objectives

1. Given a differentiable map $f: \mathbb{R}^m \to \mathbb{R}^n$, write the Jacobian $J_f(x)$ as the $n \times m$ matrix of partial derivatives, and state the chain rule for the composition $g \circ f$ as a matrix product.
2. Prove that $\nabla f(x)$ is the direction of steepest ascent of $f$ at $x$, and translate that proof into the algorithm "negative gradient descent".
3. For a joint distribution $p(x, y)$, write Bayes' theorem as a reweighting of the prior $p(y)$ by the likelihood ratio $p(x \mid y) / p(x)$, and identify the place in the calculation where the marginal $p(x) = \sum_y p(x \mid y) p(y)$ is intractable.
4. Name the assumption whose violation breaks the chain rule (non-differentiability), breaks the gradient-as-steepest-ascent statement (non-Euclidean norm), and breaks Bayes' theorem (zero measure on the conditioning event).
5. Translate any expression in this lesson into runnable NumPy code, and predict the shape of every intermediate array.

Linear maps and the matrix as function

A matrix $A \in \mathbb{R}^{n \times m}$ is not a grid of numbers. It is a function $A: \mathbb{R}^m \to \mathbb{R}^n$, defined by the rule $A(x) = Ax$, with the properties that $A$ is linear ($A(\alpha u + \beta v) = \alpha A u + \beta A v$) and continuous (which on a finite-dimensional space follows from linearity plus a bounded operator norm). The grid-of-numbers picture is the representation of $A$ in the standard basis; the picture that supports every later derivation is "linear function from one vector space to another".

The matrix-vector product $y = Ax$ has a concrete shape rule: the $i$-th row of $A$ is dotted with $x$ to produce the $i$-th entry of $y$. The $i$-th column of $A$ is the image of the $i$-th standard basis vector $e_i$. The shape rule is the same when you swap $A$ for $A^\top$: now rows become columns, and $A^\top: \mathbb{R}^n \to \mathbb{R}^m$. This is the source of the backprop shape convention: in a two-layer network $y = W_2 \, \sigma(W_1 x + b_1) + b_2$, the weight $W_2$ maps hidden activations to outputs, and the gradient of the loss with respect to $W_1$ must be the product of matrices whose shapes match the rule — not the product of any two adjacent matrices.

The matrix-as-function picture is what makes a non-square matrix coherent. $A \in \mathbb{R}^{n \times m}$ maps a 12-dimensional input to a 4-dimensional output in one application, with no iteration and no implicit "filling in". When you see $A$ with $n \neq m$, ask which space is the domain and which is the codomain — the question is unambiguous once $A$ is read as a function.

Three consequences matter for the rest of the course:

1. Rank is a function statement. $\mathrm{rank}(A) = \dim \mathrm{im}(A)$, which is at most $\min(n, m)$ and equals that bound iff $A$ is surjective onto its image. Rank deficiency is not "small numbers in the determinant"; it is the non-existence of a pre-image for some output.
2. Singular values measure stretch. $\|A\|_2 = \sigma_{\max}(A)$ is the largest factor by which $A$ stretches any unit vector, and the singular vectors are the directions in which that maximum is attained. The condition number $\kappa(A) = \sigma_{\max}(A)/\sigma_{\min}(A)$ is the ratio of the largest to the smallest stretch. It controls the speed of gradient descent: under the spectral bound on $\nabla^2 f$, convergence in $\kappa$ times the optimal steps is required for $O(\kappa \log(1/\varepsilon))$ iteration complexity.
3. Transpose is the adjoint. $\langle Ax, y \rangle = \langle x, A^\top y\rangle$, with respect to the standard inner product. This identity is the bridge between the matrix-as-function picture and the gradient-as-linear-functional picture in the next section.

Jacobians and the chain rule, in matrix form

For $f: \mathbb{R}^m \to \mathbb{R}^n$ differentiable at $x$, the Jacobian is the matrix

The $i$-th row of $J_f(x)$ is the gradient of the $i$-th scalar component $f_i : \mathbb{R}^m \to \mathbb{R}$. The Jacobian is the linear approximation to $f$ at $x$:

This is a Taylor expansion up to first order, with the Jacobian playing the role the derivative plays for scalar functions. The full statement of differentiability at $x$ is exactly this limit, with $o(\|h\|)$ meaning a term whose magnitude is negligible relative to $\|h\|$.

The chain rule in matrix form is the statement that the Jacobian of a composition is the product of Jacobians. If $g: \mathbb{R}^n \to \mathbb{R}^p$ and $f: \mathbb{R}^m \to \mathbb{R}^n$, then

The dimensions check: $J_f(x) \in \mathbb{R}^{n \times m}$, $J_g(f(x)) \in \mathbb{R}^{p \times n}$, and the product lives in $\mathbb{R}^{p \times m}$, which is the correct shape for the Jacobian of $g \circ f$. This is the single line that backs backpropagation in ML-201 lesson 4: every backprop step is a matrix product along a path in the computational graph, and the matrix product is literally the chain rule, not an analogy.

The scalar case is recovered when $n = 1$. Then $J_f(x) \in \mathbb{R}^{1 \times m}$, which is conventionally written as a row vector and identified with the gradient $\nabla f(x)^\top$ when $f$ is scalar-valued. The chain rule for scalar $f: \mathbb{R} \to \mathbb{R}$ reduces to $(f \circ g)'(x) = f'(g(x)) \cdot g'(x)$, which is the form taught in ML-101 lesson 11. The matrix form is the correct generalisation; the scalar form is what survives when the intermediate space is one-dimensional.

import numpy as np

def numerical_jacobian(f, x, h=1e-5):
    """Central-difference estimate of the Jacobian of vector-valued f at x.

    f: R^m -> R^n. Returns an (n, m) matrix. The estimate is O(h^2) per entry.
    """
    x = np.asarray(x, dtype=float)
    n = len(f(x))
    m = len(x)
    J = np.zeros((n, m))
    for j in range(m):
        ej = np.zeros_like(x); ej[j] = h
        J[:, j] = (f(x + ej) - f(x - ej)) / (2 * h)
    return J

# Example: f(x) = (x_0^2, x_0 * x_1, exp(x_1)).
f = lambda x: np.array([x[0] ** 2, x[0] * x[1], np.exp(x[1])])
x0 = np.array([0.7, -0.4])
J_num = numerical_jacobian(f, x0)
J_ana = np.array([
    [2 * x0[0], 0.0],        # d/dx of x_0^2
    [x0[1],    x0[0]],       # d/dx of x_0 * x_1
    [0.0,      np.exp(x0[1])]
])
print("max abs error:", float(np.abs(J_num - J_ana).max()))

The assumption whose violation breaks the chain rule is differentiability at the evaluation point. If $g$ has a kink at $x$ (a ReLU evaluated exactly at zero, for instance), the Jacobian does not exist there and the chain rule has nothing to multiply. The 201-level fix is sub-derivative or Clarke generalised Jacobian, which is a different theory and a different product rule; the 101-level fix is "treat the kink as one-sided" and pretend the chain rule still holds — it does not.

Gradient as steepest ascent

For a scalar function $f: \mathbb{R}^m \to \mathbb{R}$, the gradient is the vector

The first-order Taylor expansion gives $f(x + h) \approx f(x) + \nabla f(x)^\top h$. To maximise this linear approximation over unit vectors $h$ with $\|h\|_2 = 1$, write $h = (\nabla f / \|\nabla f\|) \cos\theta + (\nabla f^\perp / \|\nabla f^\perp\|) \sin\theta$, where $\theta$ is the angle between $h$ and $\nabla f$. The linear term becomes $\|\nabla f\| \cos\theta$, which is maximised at $\theta = 0$, giving the maximiser $h^\star = \nabla f / \|\nabla f\|$ and the maximum value $\|\nabla f\|$. So $\nabla f$ points uphill, and $-\nabla f$ points downhill; the gradient descent step $x_{k+1} = x_k - \alpha \nabla f(x_k)$ is exactly steepest descent.

The argument is more subtle than ML-101 presented. The conclusion "gradient points uphill" is true under the Euclidean norm. If the descent step uses a different norm — say the dual norm of a weighted $\ell_2$, or the $\ell_1$ norm — then the direction of steepest descent is not the negative gradient; it is $-\nabla^L H(x)$, where $\nabla^L H(x)$ is the vector solving $\langle \nabla^L H(x), v\rangle \le \|v\|$ for all $v$, with equality on the minimiser. This is the entire mathematical content of mirror descent, natural-gradient descent, and Hessian-aware optimisation, all of which would be incoherent under the 101-level claim "the negative gradient is always the steepest direction".

import numpy as np

def gradient_descent(f, grad_f, x0, lr=1e-2, n_steps=200):
    """Plain Euclidean gradient descent: x_{k+1} = x_k - lr * grad_f(x_k)."""
    x = np.asarray(x0, dtype=float).copy()
    history = [x.copy()]
    for _ in range(n_steps):
        x = x - lr * grad_f(x)        # negative gradient = direction of steepest DESCENT
        history.append(x.copy())
    return x, np.asarray(history)

# Toy quadratic f(x) = 0.5 x^T A x, with grad f(x) = A x.
A = np.array([[3.0, 1.0], [1.0, 2.0]])
f      = lambda x: 0.5 * x @ A @ x
grad_f = lambda x: A @ x

x_star, path = gradient_descent(f, grad_f, x0=np.array([4.0, -3.0]), lr=0.2)
print("final x:", x_star.round(6))   # should approach [0, 0] (the unique minimiser)

The deeper 201-level point is that the gradient is the Riesz representer of the differential. For every continuous linear functional $L: \mathbb{R}^m \to \mathbb{R}$, there is a unique vector $v \in \mathbb{R}^m$ such that $L(h) = v^\top h$ for all $h$. The differential $df(x): \mathbb{R}^m \to \mathbb{R}$ is such a linear functional; its Riesz representer is the gradient. This definition is independent of basis, and that basis-independence is why the gradient is the same object in $\mathbb{R}^m$ and in the parameter space of a model, and why the chain rule can be expressed without choosing coordinates.

Conditional probability and Bayes' theorem, derived

Conditional probability is defined, in the measure-theoretic foundations, by

provided $p(B) > 0$. The conditional probability $p(A \mid B)$ is not an independent primitive; it is defined in terms of the joint. ML-101 used this definition to compute $p(A \mid B)$ in tabular examples; ML-201 uses it to derive Bayes' theorem, which is a corollary of the definition:

The numerator is the joint; the denominator is the marginal of the joint over $y$. This is not a profound identity. It is a reweighting of the prior $p(y)$ by the likelihood ratio $p(x \mid y) / p(x)$. Every "Bayesian" calculation is this reweighting, and the only hard part is computing the marginal in the denominator.

import numpy as np

def bayes_predict(prior: np.ndarray, likelihood: np.ndarray) -> np.ndarray:
    """Posterior p(y | x) given prior p(y) (shape (k,)) and likelihood p(x | y) (shape (n, k)).

    Each column j is the n-vector of p(x | y=j). Returns posterior of shape (n, k).
    """
    joint = likelihood * prior[None, :]                    # shape (n, k), elementwise
    marginal = joint.sum(axis=1, keepdims=True)            # shape (n, 1)
    return joint / marginal                                  # shape (n, k), normalised

# Toy: 3 classes, 5 input bins.
prior     = np.array([0.5, 0.3, 0.2])                     # p(y)
likelihood = np.array([[0.10, 0.20, 0.30],
                       [0.30, 0.20, 0.10],
                       [0.40, 0.30, 0.20],
                       [0.10, 0.20, 0.30],
                       [0.10, 0.10, 0.10]])               # p(x | y)
posterior = bayes_predict(prior, likelihood)
print("posterior row sums (should all be 1):", posterior.sum(axis=1).round(6))

The intractable marginal is the reason variational methods exist. When $y$ is continuous, the sum becomes an integral, and $\int p(x \mid y) p(y)\, dy$ has no closed form for almost any model of practical interest. The expectation $\mathbb{E}_{p(y)}[f(y)]$ under that integral is what the sampler or the variational approximation is computing. The derivation is one line; the work is the integral.

The assumption whose violation breaks Bayes' theorem is $p(x) = 0$ for the observed $x$. If the support of $p(x)$ does not contain the observation, then $p(x \mid y)$ is undefined on that $x$ for every $y$, and the conditional $p(y \mid x)$ is not a probability at all. In practice this is rare; in theory it matters, because a model whose likelihood vanishes on the test set is a model that cannot be evaluated on the test set. The 101-level mistake is to compute $p(y \mid x)$ by plugging in $x$ without checking that $x$ is in the support; the 201-level fix is to assert the support assumption explicitly.

The five shape rules that catch every bug

The chain rule in matrix form has five shape rules that, memorised once, catch every "shape mismatch" bug in every later derivation in ML-201.

These five rules are stated here as a reference; the lesson 4 derivation of backprop uses every one of them. A learner who can read a derivation and verify each rule against the table is a learner who will not need a debugger to find the shape mismatch. A learner who cannot is a learner who will spend a working day on what is actually a one-minute check.

The deeper statement, which is not on the table but which the table makes concrete, is that the chain rule is always the same chain rule. Every "advanced" gradient — through attention, through convolution, through a recurrent cell — is the same matrix product along the path of the computation, with the same shape rules. There is no special gradient for attention; the attention gradient is the chain rule applied to a particular factorisation of the matrix product. ML-201 lesson 4 makes this explicit by deriving backprop for a two-layer network and then generalising to arbitrary graphs.

Key Takeaways

• A matrix $A \in \mathbb{R}^{n \times m}$ is a linear function $A: \mathbb{R}^m \to \mathbb{R}^n$, not a grid of numbers. Read it as a function and the rest of the lesson follows.
• The Jacobian is the matrix form of the derivative, and the chain rule is the matrix product. Every backprop step is one matrix product along one path in the computational graph.
• The gradient is the Riesz representer of the differential, and points uphill only under the Euclidean norm. Mirror descent uses a different norm and gets a different "steepest" direction.
• Bayes' theorem is the reweighting of a prior by a likelihood ratio, with a marginal in the denominator that is intractable for almost any model of practical interest. The integral is the work.
• Memorise the five shape rules. They catch every "shape mismatch" bug in every later derivation, and they are the only piece of memorisation the lesson asks for.