Estimation Theory

<- Back to Chapter 7: Statistics | Next: Hypothesis Testing ->

---

"The problem of statistical estimation is one of the most fundamental in all of science: given observations drawn from some unknown process, what can we infer about that process?"

• Sir Ronald A. Fisher, Statistical Methods and Scientific Inference (1956)

Overview

Estimation theory answers the central question of applied statistics: given a finite, noisy sample from an unknown population, how do we recover the population's parameters? This inversion - from data to model - is not merely an academic exercise. It is the mathematical foundation of every machine learning training algorithm ever written. When a language model minimises cross-entropy loss on a text corpus, it is performing maximum likelihood estimation. When a researcher reports model accuracy with error bars, those bars are frequentist confidence intervals. When the natural gradient optimizer adjusts step sizes using curvature information, it is inverting the Fisher information matrix.

The discipline divides into three interconnected pillars. Point estimation asks: given a sample, what single value should we report for the unknown parameter? We want estimators that are unbiased (correct on average), consistent (converge to the truth as sample size grows), and efficient (achieve the minimum possible variance). The Cramer-Rao lower bound establishes a fundamental limit - no unbiased estimator can have variance smaller than the reciprocal of Fisher information, a quantity measuring how much data "inform" the parameter. Maximum likelihood estimation is the workhorse of both classical statistics and modern machine learning: find the parameter value that makes the observed data most probable. It is consistent, asymptotically efficient, and invariant under reparametrisation - properties that make it the default choice across science. Confidence intervals extend point estimates to ranges, quantifying the uncertainty inherent in estimation from finite samples.

This section builds the complete theory systematically, from formal definitions through asymptotic theory, culminating in concrete ML applications: MLE as cross-entropy, Fisher information in natural gradient descent, bootstrap confidence intervals for benchmark evaluation, and Fisher information matrices in elastic weight consolidation for catastrophic forgetting prevention.

Prerequisites

• Sample statistics (mean, variance, covariance matrix) - Section01 Descriptive Statistics
• Expectation, variance, MGF ($\mathbb{E}[X]$, $\operatorname{Var}(X)$, $M_X(t)$) - Ch6 Section04 Expectation and Moments
• Common distributions (Gaussian, Bernoulli, Poisson, Exponential, Beta) - Ch6 Section02 Common Distributions
• Bayes' theorem and conditional distributions - Ch6 Section03 Joint Distributions
• Law of large numbers and CLT - Ch6 Section06 Stochastic Processes
• Matrix inverse, SVD, positive definiteness - Ch3 Advanced Linear Algebra
• Partial derivatives and optimisation conditions - Ch4 Calculus Fundamentals

Companion Notebooks

Learning Objectives

After completing this section, you will:

• Define an estimator as a statistic and explain why it is a random variable with a sampling distribution
• Decompose the MSE of any estimator into bias-squared plus variance, and construct examples demonstrating the trade-off
• State and prove the Cramer-Rao lower bound using the Cauchy-Schwarz inequality
• Compute the Fisher information for Bernoulli, Gaussian, Poisson, and Exponential models
• Derive the MLE for scalar and multivariate Gaussian parameters from first principles
• Prove that MLE of Gaussian variance is biased and state the corrected estimator
• Explain why minimising NLL/cross-entropy loss is equivalent to performing MLE
• Derive confidence intervals using both the pivoting method and the asymptotic normal approximation
• Construct bootstrap confidence intervals and explain when they outperform asymptotic CIs
• State the asymptotic normality theorem for MLE and explain its rate $O(1/\sqrt{n})$
• Apply the delta method to derive asymptotic distributions of transformed estimators
• Connect Fisher information to natural gradient descent, K-FAC, and elastic weight consolidation

---

Table of Contents

• 1. Intuition
  • 1.1 What Is Statistical Estimation?
  • 1.2 The Estimation Landscape
  • 1.3 Historical Timeline
  • 1.4 Why Estimation Theory Matters for AI
• 2. The Formal Estimation Problem
  • 2.1 Statistical Model and Parametric Family
  • 2.2 Point Estimators
  • 2.3 The Loss Framework: MSE, Bias, and Variance
  • 2.4 Examples of Biased and Unbiased Estimators
• 3. Properties of Estimators
  • 3.1 Bias and Unbiasedness
  • 3.2 Consistency
  • 3.3 Efficiency
  • 3.4 Sufficiency and the Rao-Blackwell Theorem
  • 3.5 Completeness and the Lehmann-Scheffe Theorem
• 4. Fisher Information and the Cramer-Rao Bound
  • 4.1 The Score Function
  • 4.2 Fisher Information
  • 4.3 The Cramer-Rao Lower Bound
  • 4.4 Fisher Information Matrix in Practice
• 5. Maximum Likelihood Estimation
  • 5.1 The Likelihood Principle
  • 5.2 Deriving MLEs: Scalar Parameters
  • 5.3 Deriving MLEs: Multivariate Gaussian
  • 5.4 MLE as Cross-Entropy Minimisation
  • 5.5 Properties of MLE
  • 5.6 Numerical MLE
• 6. Method of Moments
  • 6.1 The Method of Moments
  • 6.2 Generalised Method of Moments
  • 6.3 MoM vs MLE
• 7. Confidence Intervals
  • 7.1 The Frequentist Interpretation
  • 7.2 Exact Confidence Intervals via Pivoting
  • 7.3 Asymptotic Confidence Intervals
  • 7.4 Bootstrap Confidence Intervals
  • 7.5 Confidence Intervals in ML Evaluation
• 8. Asymptotic Theory
  • 8.1 Convergence Modes
  • 8.2 The Delta Method
  • 8.3 Asymptotic Normality of MLE
  • 8.4 Misspecified Models
• 9. Applications in Machine Learning
  • 9.1 MLE Is Cross-Entropy Training
  • 9.2 Natural Gradient and Fisher Information
  • 9.3 Confidence Intervals for Model Evaluation
  • 9.4 Estimating Scaling Laws
  • 9.5 Fisher Information in Fine-Tuning
• 10. Common Mistakes
• 11. Exercises
• 12. Why This Matters for AI (2026 Perspective)
• 13. Conceptual Bridge

---

1. Intuition

1.1 What Is Statistical Estimation?

Suppose you flip a coin 100 times and observe 63 heads. The coin's true bias $p$ - the probability of heads on any single flip - is unknown. What is your best guess for $p$? The obvious answer is $\hat{p} = 0.63$. But why is this the right answer? What makes it "best"? Could a different estimate be better in some sense? And how confident should you be - could the true $p$ be 0.5 (a fair coin) despite observing 63 heads?

These are precisely the questions estimation theory addresses. The statistical estimation problem has three ingredients:

1. An unknown parameter $\theta \in \Theta$ governing some probability distribution $p(\mathbf{x}; \theta)$
2. Observed data $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}$ drawn independently from $p(\cdot; \theta)$
3. An estimator $\hat{\theta} = T(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)})$: a function of the data that produces a guess for $\theta$

The key insight, often overlooked by beginners, is that the estimator is a random variable. Before we collect data, $\hat{\theta}$ is uncertain - its value depends on which random sample we happen to observe. If we repeated the experiment with a fresh sample, we would get a different $\hat{\theta}$. The sampling distribution of $\hat{\theta}$ - the distribution of $\hat{\theta}$ across all possible samples - is the object of study. A good estimator has a sampling distribution tightly concentrated around the true $\theta$.

For AI: This perspective matters directly. When you train a neural network on a dataset $\mathcal{D}$ and obtain weights $\hat{\boldsymbol{\theta}}$, those weights are a realisation of an estimator - a function of the random training data. Training on a different shuffle or subsample gives different $\hat{\boldsymbol{\theta}}$. The variation you see across training runs, seeds, and dataset splits is the sampling distribution of the MLE manifesting in practice.

1.2 The Estimation Landscape

Estimation theory spans three major frameworks, each answering a different question:

Point estimation produces a single value $\hat{\theta}$ for the unknown parameter. The challenge is specifying what "best" means. Three criteria dominate:

• Unbiasedness: $\mathbb{E}[\hat{\theta}] = \theta$ - on average, the estimator is correct
• Minimum variance: among unbiased estimators, prefer the one with smallest $\operatorname{Var}(\hat{\theta})$
• Consistency: $\hat{\theta}_n \xrightarrow{P} \theta$ as $n \to \infty$ - with enough data, the estimator converges to the truth

These criteria can conflict. The Cramer-Rao lower bound establishes a hard floor on variance for unbiased estimators, and the estimators that achieve this floor are called efficient. Maximum likelihood estimation (MLE) is the dominant method: it is consistent, asymptotically efficient, and invariant under smooth reparametrisations.

Interval estimation goes beyond a point to a range $[\hat{\theta}_L, \hat{\theta}_U]$ with a coverage guarantee: in repeated experiments, $1 - \alpha$ fraction of such intervals will contain the true $\theta$. Frequentist confidence intervals capture estimation uncertainty without requiring a prior distribution on $\theta$.

Asymptotic theory studies estimator behaviour as $n \to \infty$. The flagship result is the asymptotic normality of MLE: under regularity conditions,

$$\sqrt{n}\left(\hat{\theta}_{\text{MLE}} - \theta^*\right) \xrightarrow{d} \mathcal{N}\!\left(0,\, \mathcal{I}(\theta^*)^{-1}\right)$$

where $\mathcal{I}(\theta^*)$ is the Fisher information. This single result simultaneously justifies MLE as an estimation method, provides the basis for asymptotic confidence intervals, and connects estimation theory to information geometry.

For AI: The three frameworks map directly to ML practice. Point estimation = model training (find $\hat{\boldsymbol{\theta}}$). Interval estimation = evaluation with error bars (report accuracy $\pm$ 95% CI). Asymptotic theory = understanding why larger datasets produce more reliable models (the $1/\sqrt{n}$ convergence rate).

1.3 Historical Timeline

ESTIMATION THEORY - HISTORICAL TIMELINE
========================================================================

  1795  Gauss (age 18) uses least squares to predict asteroid orbits
        - the first systematic estimation method

  1805  Legendre publishes least squares formally (Gauss priority dispute)

  1809  Gauss derives least squares from the assumption of Gaussian errors
        - first connection between MLE and squared-error minimisation

  1894  Pearson introduces method of moments - systematic moment matching

  1912  Fisher introduces "maximum likelihood" as a term; begins systematic
        study of estimation properties

  1922  Fisher: "On the Mathematical Foundations of Theoretical Statistics"
        - defines sufficiency, efficiency, consistency; derives properties of MLE

  1925  Fisher introduces Fisher information and the information inequality
        (precursor to Cramer-Rao)

  1945  Cramer proves the Cramer-Rao lower bound (independently of Rao)

  1945  Rao independently proves the same bound + Rao-Blackwell theorem

  1947  Lehmann & Scheffe: completeness and UMVUE theory

  1950s Wald develops statistical decision theory - unified framework for
        estimation as minimising expected loss

  1979  Efron introduces the bootstrap - non-parametric confidence intervals
        without distributional assumptions

  1998  Amari: natural gradient via Fisher information geometry
        - direct connection to modern ML optimisation

  2015  Martens & Grosse: K-FAC - tractable approximation of FIM for DNNs

  2017  Kirkpatrick et al.: Elastic Weight Consolidation (EWC) uses FIM
        to prevent catastrophic forgetting in continual learning

  2022  Hoffmann et al. (Chinchilla): scaling laws estimated via MLE on
        (N, D, L) data - estimation theory at trillion-parameter scale

========================================================================

1.4 Why Estimation Theory Matters for AI

Every major component of a modern ML pipeline rests on estimation theory:

Training objectives. Minimising cross-entropy loss is equivalent to performing MLE. The connection is exact: $\arg\min_{\boldsymbol{\theta}} \sum_i -\log p(y^{(i)} \mid \mathbf{x}^{(i)}; \boldsymbol{\theta}) = \arg\max_{\boldsymbol{\theta}} \prod_i p(y^{(i)} \mid \mathbf{x}^{(i)}; \boldsymbol{\theta})$. Every training run of GPT-4, LLaMA-3, Gemini, or any neural network trained with NLL loss is performing MLE on the conditional distribution.

Evaluation uncertainty. When a benchmark reports "Model A achieves 73.2% accuracy", this is a point estimate. How uncertain is it? With $n = 1000$ test examples, the 95% CI is approximately $\pm 2.8\%$ - meaning Model A and Model B with 73.5% accuracy are statistically indistinguishable. Estimation theory makes this precise.

Second-order optimisation. Adam, K-FAC, and natural gradient all incorporate curvature information. The natural gradient multiplies the gradient by $\mathcal{I}(\boldsymbol{\theta})^{-1}$, where $\mathcal{I}$ is the Fisher information matrix. This reparametrises the optimisation in the geometry of the statistical manifold, making gradient steps invariant to reparametrisation. K-FAC (Martens & Grosse, 2015) approximates $\mathcal{I}^{-1}$ tractably for deep networks.

Continual learning. Elastic weight consolidation (Kirkpatrick et al., 2017) protects important weights during sequential task learning. "Importance" is measured by the diagonal of the FIM: parameters with high Fisher information are those to which the likelihood is most sensitive, and perturbing them most damages performance.

Calibration. A well-calibrated model's confidence scores match empirical frequencies: when it says "80% confident", it's right 80% of the time. Temperature scaling - dividing logits by a scalar $\tau$ - is an MLE problem: find the $\tau$ that maximises likelihood on a held-out calibration set.

---

2. The Formal Estimation Problem

2.1 Statistical Model and Parametric Family

Definition 2.1 (Parametric Statistical Model). A parametric statistical model is a collection of probability distributions indexed by a parameter:

$$\mathcal{M} = \{p(\cdot\,;\boldsymbol{\theta}) : \boldsymbol{\theta} \in \Theta\}$$

where $\Theta \subseteq \mathbb{R}^k$ is the parameter space. The model is:

• Correctly specified if the true data-generating distribution $p^*$ satisfies $p^* = p(\cdot\,;\boldsymbol{\theta}^*)$ for some $\boldsymbol{\theta}^* \in \Theta$
• Misspecified if $p^* \notin \mathcal{M}$ - the model family does not contain the truth (most practical ML models are misspecified)
• Identifiable if different parameters give different distributions: $\boldsymbol{\theta} \neq \boldsymbol{\theta}' \Rightarrow p(\cdot\,;\boldsymbol{\theta}) \neq p(\cdot\,;\boldsymbol{\theta}')$

Identifiability is necessary for consistent estimation - you cannot consistently estimate a parameter that leaves the data distribution unchanged.

Standard examples of parametric families:

Non-examples (non-identifiable):

• $\mathcal{N}(\mu_1 + \mu_2, 1)$ with parameter $(\mu_1, \mu_2)$: we can only estimate $\mu_1 + \mu_2$, not the individual components. The model is not identifiable.
• A neural network with two hidden layers of width 1 using $\tanh$: swapping the two neurons gives the same function, so the parameterisation is not identifiable (though the function class may be).

The iid assumption. Throughout this section, observations $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}$ are assumed independent and identically distributed (iid) from $p(\cdot\,;\boldsymbol{\theta}^*)$. The iid assumption enables the joint log-likelihood to factorise as a sum:

$$\log p(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)};\boldsymbol{\theta}) = \sum_{i=1}^n \log p(\mathbf{x}^{(i)};\boldsymbol{\theta})$$

This factorisation is what makes MLE computationally tractable and theoretically tractable. In practice, this assumption is violated whenever data points are correlated (time series, text sequences, spatially clustered data), and estimation theory has extensions to handle dependent data.

2.2 Point Estimators

Definition 2.2 (Estimator). An estimator of $\boldsymbol{\theta}$ is any measurable function $\hat{\boldsymbol{\theta}}_n = T(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)})$ of the sample. The key points:

1. $\hat{\boldsymbol{\theta}}_n$ is a random variable - before observing data, it is uncertain
2. A specific value $T(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}) = t$ after observing data is called an estimate (not estimator)
3. The sampling distribution of $\hat{\boldsymbol{\theta}}_n$ is the distribution of $T$ across all possible samples of size $n$

This distinction - estimator (random variable) vs. estimate (specific realisation) - is pedantic but consequential. Confidence interval coverage is a property of the estimator (random), not the estimate (fixed). When we say "95% CI", we mean that the random interval contains $\boldsymbol{\theta}^*$ with probability 95%, not that this specific computed interval has a 95% chance of containing $\boldsymbol{\theta}^*$ (after observing data, $\boldsymbol{\theta}^*$ is either in the interval or not - there is no remaining randomness).

Examples of estimators for the Gaussian mean $\mu$:

Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$.

• Sample mean: $\hat{\mu}_1 = \frac{1}{n}\sum_{i=1}^n x^{(i)}$ - unbiased, consistent, efficient
• Constant zero: $\hat{\mu}_2 = 0$ - biased unless $\mu = 0$, inconsistent
• First observation: $\hat{\mu}_3 = x^{(1)}$ - unbiased! but inconsistent (variance $\sigma^2$ regardless of $n$)
• Trimmed mean: $\hat{\mu}_4 =$ mean of middle 90% - slightly biased for Gaussian, but robust to outliers
• Constant $c$: $\hat{\mu}_5 = c$ - biased by $|c - \mu|$, inconsistent; can have lower MSE than the sample mean when $|\mu - c|$ is small (illustrates bias-variance trade-off)

The existence of unbiased but inconsistent estimators ($\hat{\mu}_3$) and consistent but biased estimators (trimmed mean) shows these properties are logically independent.

2.3 The Loss Framework: MSE, Bias, and Variance

To compare estimators, we need a criterion. The most widely used is Mean Squared Error (MSE):

Definition 2.3 (MSE, Bias, Variance). For an estimator $\hat{\theta}$ of a scalar parameter $\theta$:

$$\operatorname{MSE}(\hat{\theta}) = \mathbb{E}\left[(\hat{\theta} - \theta)^2\right]$$

The bias-variance decomposition is the central identity of estimation theory:

$$\operatorname{MSE}(\hat{\theta}) = \operatorname{Bias}^2(\hat{\theta}) + \operatorname{Var}(\hat{\theta})$$

Proof:

$$\operatorname{MSE}(\hat{\theta}) = \mathbb{E}\left[(\hat{\theta} - \theta)^2\right]$$

Add and subtract $\mathbb{E}[\hat{\theta}]$:

$$= \mathbb{E}\left[\left((\hat{\theta} - \mathbb{E}[\hat{\theta}]) + (\mathbb{E}[\hat{\theta}] - \theta)\right)^2\right]$$ $$= \mathbb{E}\left[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2\right] + 2(\mathbb{E}[\hat{\theta}] - \theta)\underbrace{\mathbb{E}[\hat{\theta} - \mathbb{E}[\hat{\theta}]]}_{=0} + (\mathbb{E}[\hat{\theta}] - \theta)^2$$ $$= \operatorname{Var}(\hat{\theta}) + \operatorname{Bias}(\hat{\theta})^2 \qquad \square$$

where $\operatorname{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$.

Geometric interpretation:

BIAS-VARIANCE DECOMPOSITION - GEOMETRIC VIEW
========================================================================

  True parameter \\theta* = 0 (target)

  Case 1: Low bias, low variance (good estimator)
    ---    Estimates cluster tightly around \\theta*
    ----   MSE \\approx small
    ---

  Case 2: Low bias, high variance (unbiased but noisy)
        -             -
    -       -   x   -       -
        -             -
    Centred on \\theta* but spread out.  MSE = Var is large.

  Case 3: High bias, low variance (biased but stable)
              ---
             ----    x  (\\theta*)
              ---
    Centred off target.  MSE = Bias^2 + small Var.

  Case 4: High bias, high variance (worst case)
    -           -
            -       x  (\\theta*)
        -       -
    Both centred off target AND spread out.

  Key trade-off: Reducing variance by shrinkage introduces bias.
  Increasing bias via regularisation often reduces variance more than
  it increases bias-squared, giving lower MSE overall.

========================================================================

The bias-variance trade-off in ML is the exact same phenomenon: regularisation (L2, dropout, early stopping) introduces bias (the model is pulled away from the MLE toward a constrained region) but reduces variance (the model is less sensitive to the particular training sample), often reducing test MSE overall.

Minimax estimators. MSE is not the only loss criterion. The minimax estimator minimises the worst-case risk: $\min_T \max_\theta \mathbb{E}[(\hat{\theta} - \theta)^2]$. The James-Stein estimator (1961) famously showed that in $\mathbb{R}^k$ for $k \geq 3$, the sample mean is inadmissible - there exists an estimator with strictly lower MSE at every $\theta$. This is the Stein paradox: shrinkage toward the origin (a biased estimator!) uniformly dominates the unbiased sample mean in 3+ dimensions.

2.4 Examples of Biased and Unbiased Estimators

Example 1: Sample mean is unbiased.

Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} p(\cdot;\theta)$ with $\mathbb{E}[x^{(i)}] = \mu$. Then:

$$\mathbb{E}\!\left[\bar{x}\right] = \mathbb{E}\!\left[\frac{1}{n}\sum_{i=1}^n x^{(i)}\right] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[x^{(i)}] = \frac{n\mu}{n} = \mu$$

The sample mean is an unbiased estimator of $\mu$ for any distribution with finite mean.

Example 2: MLE of Gaussian variance is biased.

Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$. The MLE of $\sigma^2$ is:

$$\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n (x^{(i)} - \bar{x})^2$$

Computing the bias: Using $x^{(i)} - \bar{x} = (x^{(i)} - \mu) - (\bar{x} - \mu)$:

$$\sum_{i=1}^n (x^{(i)} - \bar{x})^2 = \sum_{i=1}^n (x^{(i)} - \mu)^2 - n(\bar{x} - \mu)^2$$

Taking expectations:

$$\mathbb{E}\!\left[\sum_{i=1}^n (x^{(i)} - \bar{x})^2\right] = n\sigma^2 - n \cdot \frac{\sigma^2}{n} = (n-1)\sigma^2$$

Therefore:

$$\mathbb{E}[\hat{\sigma}^2_{\text{MLE}}] = \frac{(n-1)\sigma^2}{n} = \sigma^2 - \frac{\sigma^2}{n}$$

The bias is $-\sigma^2/n$ - the MLE underestimates variance. The correction is Bessel's correction: $s^2 = \frac{1}{n-1}\sum(x^{(i)} - \bar{x})^2$ is unbiased.

Recall from Section01: We saw Bessel's correction $s^2 = \frac{1}{n-1}\sum(x^{(i)}-\bar{x})^2$ introduced as the "standard" sample variance formula. Now we understand why: MLE gives $1/n$, which is biased. The $n-1$ denominator corrects for the one degree of freedom lost in estimating $\mu$ by $\bar{x}$.

Example 3: MLE is biased but consistent.

The Gaussian variance MLE $\hat{\sigma}^2_{\text{MLE}}$ has bias $-\sigma^2/n \to 0$ as $n \to \infty$. It is asymptotically unbiased and consistent. This illustrates that bias can be acceptable if it vanishes with sample size.

Example 4: When bias reduces MSE.

Consider estimating $\mu$ with the shrinkage estimator $\hat{\mu}_c = c \bar{x}$ for some $c \in (0,1)$. Then:

$$\operatorname{MSE}(\hat{\mu}_c) = (c-1)^2 \mu^2 + c^2 \frac{\sigma^2}{n}$$

The optimal $c^* = \frac{\mu^2}{\mu^2 + \sigma^2/n}$, which gives lower MSE than $c = 1$ (unbiased sample mean) whenever $\sigma^2/n > 0$. The key insight: when $|\mu|$ is small relative to $\sigma/\sqrt{n}$, shrinking toward zero introduces small bias but large variance reduction.

---

3. Properties of Estimators

3.1 Bias and Unbiasedness

Definition 3.1 (Bias). The bias of an estimator $\hat{\theta}$ for parameter $\theta$ is:

$$\operatorname{Bias}(\hat{\theta};\theta) = \mathbb{E}[\hat{\theta}] - \theta$$

The estimator is unbiased if $\operatorname{Bias}(\hat{\theta};\theta) = 0$ for all $\theta \in \Theta$, and asymptotically unbiased if $\operatorname{Bias}(\hat{\theta}_n;\theta) \to 0$ as $n \to \infty$.

Why unbiasedness is desirable - but not sacred. An unbiased estimator is correct on average over infinite repetitions of the experiment. This is a natural property to want. However, unbiasedness alone is insufficient:

• The estimator $\hat{\theta} = x^{(1)}$ (use only the first observation) is unbiased but throws away $n-1$ observations - clearly wasteful
• No unbiased estimator exists for some parameters: there is no unbiased estimator of $e^\mu$ based on a Gaussian sample (by the Lehmann-Scheffe theorem, for this to exist $e^\mu$ would need to be a polynomial in $\bar{x}$, which it is not)
• The Stein phenomenon (Section2.3) shows that in 3+ dimensions, unbiased estimators can be uniformly dominated by biased ones

Bias correction. When an estimator $\hat{\theta}$ has known bias $b(\theta) = \mathbb{E}[\hat{\theta}] - \theta$, we can correct it: $\hat{\theta}_{\text{corr}} = \hat{\theta} - b(\hat{\theta})$. This is the idea behind Bessel's correction ($n-1$ vs $n$ in the variance formula) and jackknife bias reduction.

For AI: Batch normalisation computes $\hat{\mu}_B = \frac{1}{B}\sum_{i \in \text{batch}} x_i$ as the batch mean during training. This is an unbiased estimator of the feature mean. During inference, PyTorch uses the running mean accumulated over training, which is a biased estimator of the current data mean if the distribution has shifted - illustrating that unbiasedness depends on whether the estimation scenario matches the training scenario.

3.2 Consistency

Definition 3.2 (Consistency). An estimator $\hat{\theta}_n$ is:

• Weakly consistent if $\hat{\theta}_n \xrightarrow{P} \theta$ for all $\theta \in \Theta$ (convergence in probability)
• Strongly consistent if $\hat{\theta}_n \xrightarrow{a.s.} \theta$ for all $\theta \in \Theta$ (almost-sure convergence)

Weak consistency: for every $\varepsilon > 0$, $P(|\hat{\theta}_n - \theta| > \varepsilon) \to 0$ as $n \to \infty$.

Sufficient conditions for consistency:

• If $\operatorname{Bias}(\hat{\theta}_n;\theta) \to 0$ and $\operatorname{Var}(\hat{\theta}_n) \to 0$ as $n \to \infty$, then $\hat{\theta}_n$ is weakly consistent. (By Chebyshev + the bias decomposition: $P(|\hat{\theta}_n - \theta| > \varepsilon) \leq \operatorname{MSE}(\hat{\theta}_n)/\varepsilon^2 \to 0$.)

Example: Sample mean consistency.

$\bar{x}_n = \frac{1}{n}\sum_{i=1}^n x^{(i)}$ satisfies $\operatorname{Bias} = 0$ and $\operatorname{Var}(\bar{x}_n) = \sigma^2/n \to 0$. Therefore $\bar{x}_n \xrightarrow{P} \mu$. This is exactly the Weak Law of Large Numbers - consistency of the sample mean is the LLN.

Example: MLE of Gaussian variance is consistent.

$\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum(x^{(i)} - \bar{x})^2$ has bias $-\sigma^2/n \to 0$ and variance $O(1/n)$, so it is consistent even though it is biased for finite $n$.

Inconsistent estimator example: $\hat{\theta} = x^{(1)}$ (take only the first observation) has $\operatorname{Var} = \sigma^2$ regardless of $n$. It never concentrates - it is inconsistent.

Why consistency matters: Consistency is the minimal requirement for an estimator to be scientifically useful. A method that doesn't converge to the right answer with infinite data is fundamentally broken. Consistency does not guarantee that the estimator is good for small $n$ - convergence can be arbitrarily slow - but it at least ensures the method is correct in the large-data limit.

For AI: The "double descent" phenomenon in over-parameterised neural networks challenges classical bias-variance thinking. A massively over-parameterised model ($d \gg n$) can still be consistent for the Bayes-optimal predictor if trained with appropriate implicit regularisation (e.g., gradient descent from zero initialisation). This is an active area connecting statistical estimation theory to modern deep learning theory.

3.3 Efficiency

Among all unbiased estimators of $\theta$, which has the smallest variance? This question is answered by the Cramer-Rao bound (Section 4), but we can define efficiency as a property here.

Definition 3.3 (Efficiency and Relative Efficiency).

• An unbiased estimator $\hat{\theta}$ is efficient if its variance equals the Cramer-Rao lower bound: $\operatorname{Var}(\hat{\theta}) = 1/\mathcal{I}(\theta)$.
• The relative efficiency of two unbiased estimators $\hat{\theta}_1$ and $\hat{\theta}_2$ is $e(\hat{\theta}_1, \hat{\theta}_2) = \operatorname{Var}(\hat{\theta}_2)/\operatorname{Var}(\hat{\theta}_1)$. Values greater than 1 mean $\hat{\theta}_1$ is more efficient.

Examples:

1. Gaussian mean: The sample mean $\bar{x}$ is efficient - it achieves the CRB $\sigma^2/n$. Relative efficiency of the sample median vs. mean is $2/\pi \approx 0.637$ for Gaussian data - the median throws away ~36% of efficiency.

2. Gaussian variance: The unbiased sample variance $s^2$ is not efficient (the MLE $\hat{\sigma}^2$ achieves the CRB but is biased; correcting for bias loses the CRB).

Asymptotic efficiency. For large $n$, MLE achieves the CRB asymptotically - it is asymptotically efficient. This is one of its key advantages.

For AI: The sample mean is the most efficient estimator of the mean for Gaussian data. Batch gradient descent using all $n$ samples computes the exact gradient (equivalent to efficient estimation), while SGD uses a single sample or mini-batch (less efficient, higher variance, but much faster per update). The trade-off between statistical efficiency and computational efficiency is fundamental to modern ML training.

3.4 Sufficiency and the Rao-Blackwell Theorem

A sufficient statistic captures all the information in the data that is relevant to estimating $\theta$.

Definition 3.4 (Sufficient Statistic). A statistic $T = T(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)})$ is sufficient for $\theta$ if the conditional distribution $p(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)} \mid T = t; \theta)$ does not depend on $\theta$ for any $t$.

Intuitively: once you know $T$, the raw data $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}$ provide no additional information about $\theta$.

Fisher-Neyman Factorisation Theorem. $T$ is sufficient for $\theta$ if and only if the likelihood factors as:

$$p(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)};\theta) = g(T(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}), \theta) \cdot h(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)})$$

where $g$ depends on data only through $T$, and $h$ does not depend on $\theta$.

Examples of sufficient statistics:

Rao-Blackwell Theorem. If $\hat{\theta}$ is any unbiased estimator and $T$ is a sufficient statistic, then $\tilde{\theta} = \mathbb{E}[\hat{\theta} \mid T]$ satisfies:

$$\operatorname{Var}(\tilde{\theta}) \leq \operatorname{Var}(\hat{\theta})$$

with equality iff $\hat{\theta}$ is already a function of $T$. The "Rao-Blackwellised" estimator is at least as good.

Proof sketch: By the law of total variance, $\operatorname{Var}(\hat{\theta}) = \operatorname{Var}(\mathbb{E}[\hat{\theta}|T]) + \mathbb{E}[\operatorname{Var}(\hat{\theta}|T)] \geq \operatorname{Var}(\tilde{\theta})$.

For AI: Sufficient statistics are the foundation of exponential families, which include Gaussian, Bernoulli, Poisson, Gamma, and most common distributions. The natural parameterisation of exponential families (used in generalised linear models and variational inference) exploits sufficient statistics to simplify computation. In the VAE (Kingma & Welling, 2014), the encoder $q_\phi(\mathbf{z}|\mathbf{x})$ outputs sufficient statistics $(\boldsymbol{\mu}_\phi(\mathbf{x}), \boldsymbol{\sigma}_\phi(\mathbf{x}))$ of the Gaussian posterior approximation.

3.5 Completeness and the Lehmann-Scheffe Theorem

Definition 3.5 (Complete Statistic). A sufficient statistic $T$ is complete if for any measurable function $g$: $\mathbb{E}[g(T); \theta] = 0$ for all $\theta \in \Theta$ implies $g(T) = 0$ a.s. for all $\theta$.

Completeness means the only function of $T$ that has zero expectation everywhere is the zero function - there are no "hidden" unbiasedness conditions.

Lehmann-Scheffe Theorem. If $T$ is a complete sufficient statistic and $\tilde{\theta} = h(T)$ is unbiased for $\theta$, then $\tilde{\theta}$ is the unique minimum variance unbiased estimator (UMVUE) of $\theta$.

The UMVUE is the "best possible" unbiased estimator - it has the lowest variance among all unbiased estimators, and it is unique.

Example: For $\mathcal{N}(\mu, \sigma^2)$ with $\sigma^2$ known, $T = \bar{x}$ is complete sufficient, and $\bar{x}$ itself is unbiased for $\mu$. By Lehmann-Scheffe, $\bar{x}$ is the UMVUE of $\mu$ - there is no unbiased estimator with smaller variance. This also confirms that $\bar{x}$ achieves the CRB $\sigma^2/n$.

---

4. Fisher Information and the Cramer-Rao Bound

4.1 The Score Function

The score function is the fundamental quantity linking estimation and information theory.

Definition 4.1 (Score Function). For a single observation $x$ from $p(x;\theta)$, the score is:

$$s(\theta; x) = \frac{\partial}{\partial \theta} \log p(x; \theta)$$

For a sample of $n$ iid observations, the total score is:

$$S_n(\theta) = \sum_{i=1}^n \frac{\partial}{\partial \theta} \log p(x^{(i)};\theta)$$

The MLE solves $S_n(\hat{\theta}) = 0$ (the score equation).

Zero-mean property of the score:

$$\mathbb{E}\!\left[s(\theta; X)\right] = \int \frac{\partial \log p(x;\theta)}{\partial \theta} p(x;\theta)\, dx = \int \frac{\frac{\partial p(x;\theta)}{\partial \theta}}{p(x;\theta)} p(x;\theta)\, dx = \frac{\partial}{\partial \theta}\int p(x;\theta)\, dx = \frac{\partial}{\partial \theta}(1) = 0$$

under regularity conditions permitting interchange of differentiation and integration.

Interpretation: The score measures how sensitively the log-likelihood responds to perturbations of $\theta$. At the true parameter value, the score has zero mean - on average, the likelihood is at a stationary point. But it has positive variance, measuring how much information the data carries about $\theta$.

SCORE FUNCTION - GEOMETRIC INTUITION
========================================================================

  Log-likelihood \\ell(\\theta) = \\Sigma_i log p(x_i;\\theta) as a function of \\theta:

         \\ell(\\theta)
          |
          |         +-----+
          |       +-+     +-+
          |     +-+         +-+
          |---+-+             +-+----
          |                        \\theta
                    ^
                 \\theta* (true parameter)
                 slope = score = 0 at maximum

  Score s(\\theta; data) = d\\ell/d\\theta is the slope of the log-likelihood.
  High variance of score at \\theta* means the data sharply identifies
  \\theta* (the likelihood is "peaky"). Low variance means the data
  are relatively uninformative about \\theta.

========================================================================

4.2 Fisher Information

Definition 4.2 (Fisher Information). The Fisher information for parameter $\theta$ in model $p(x;\theta)$ is:

$$\mathcal{I}(\theta) = \mathbb{E}\!\left[\left(\frac{\partial \log p(X;\theta)}{\partial \theta}\right)^2\right] = \operatorname{Var}\!\left(\frac{\partial \log p(X;\theta)}{\partial \theta}\right)$$

(The second equality uses $\mathbb{E}[s] = 0$.)

Alternative form (under regularity conditions):

$$\mathcal{I}(\theta) = -\mathbb{E}\!\left[\frac{\partial^2 \log p(X;\theta)}{\partial \theta^2}\right]$$

Proof of equivalence: Differentiating $\mathbb{E}[s(\theta;X)] = 0$ with respect to $\theta$:

$$0 = \frac{\partial}{\partial\theta}\int \frac{\partial \log p}{\partial \theta} p\, dx = \int \left[\frac{\partial^2 \log p}{\partial \theta^2} p + \frac{\partial \log p}{\partial \theta} \cdot \frac{\partial p}{\partial \theta}\right] dx$$ $$= \mathbb{E}\!\left[\frac{\partial^2 \log p}{\partial \theta^2}\right] + \mathbb{E}\!\left[\left(\frac{\partial \log p}{\partial\theta}\right)^2\right]$$

Therefore $\mathcal{I}(\theta) = -\mathbb{E}\!\left[\frac{\partial^2 \log p}{\partial\theta^2}\right]$. $\square$

The second-derivative form has intuitive content: $\mathcal{I}(\theta)$ is the expected (negative) curvature of the log-likelihood. High curvature = the likelihood has a sharp peak = the data strongly identifies $\theta$. Low curvature = flat likelihood = data is relatively uninformative.

Fisher information for an iid sample of $n$ observations:

$$\mathcal{I}_n(\theta) = n \cdot \mathcal{I}_1(\theta)$$

Information accumulates linearly with sample size - doubling data doubles information, halving uncertainty (in variance terms: $1/\mathcal{I}_n = (1/\mathcal{I}_1)/n$).

Computed examples:

Bernoulli$(p)$: $\log p(x;p) = x\log p + (1-x)\log(1-p)$. Score: $s = x/p - (1-x)/(1-p)$. Score squared: $s^2 = \frac{(x-p)^2}{p^2(1-p)^2}$.

$$\mathcal{I}(p) = \mathbb{E}[s^2] = \frac{\operatorname{Var}(X)}{p^2(1-p)^2} = \frac{p(1-p)}{p^2(1-p)^2} = \frac{1}{p(1-p)}$$

$\mathcal{N}(\mu, \sigma^2)$ with $\sigma^2$ known: $\log p(x;\mu) = -\frac{(x-\mu)^2}{2\sigma^2} + \text{const}$. Score: $s = (x-\mu)/\sigma^2$.

$$\mathcal{I}(\mu) = \mathbb{E}\!\left[\frac{(X-\mu)^2}{\sigma^4}\right] = \frac{\operatorname{Var}(X)}{\sigma^4} = \frac{\sigma^2}{\sigma^4} = \frac{1}{\sigma^2}$$

Poisson$(\lambda)$: $\log p(x;\lambda) = x\log\lambda - \lambda - \log(x!)$. Score: $s = x/\lambda - 1$.

$$\mathcal{I}(\lambda) = \mathbb{E}\!\left[\left(\frac{X}{\lambda} - 1\right)^2\right] = \frac{\operatorname{Var}(X)}{\lambda^2} = \frac{\lambda}{\lambda^2} = \frac{1}{\lambda}$$

Pattern: For exponential family distributions, $\mathcal{I}(\theta) = 1/\operatorname{Var}(\hat{\theta}_{\text{MLE}})$ in scalar cases - information is the reciprocal of the natural parameter variance.

Multivariate Fisher Information Matrix. For $\boldsymbol{\theta} \in \mathbb{R}^k$:

$$\mathcal{I}(\boldsymbol{\theta})_{jl} = \mathbb{E}\!\left[\frac{\partial \log p}{\partial \theta_j} \cdot \frac{\partial \log p}{\partial \theta_l}\right] = -\mathbb{E}\!\left[\frac{\partial^2 \log p}{\partial \theta_j \partial \theta_l}\right]$$

In matrix form: $\mathcal{I}(\boldsymbol{\theta}) = \mathbb{E}[\mathbf{s}\mathbf{s}^\top] = -\mathbb{E}[H_{\log p}]$ where $H_{\log p}$ is the Hessian of the log-likelihood.

The FIM is always positive semi-definite (as an outer product expectation), and positive definite under identifiability.

4.3 The Cramer-Rao Lower Bound

The Cramer-Rao Lower Bound (CRB) is the fundamental limit on estimation accuracy.

Theorem 4.3 (Cramer-Rao Lower Bound). Let $\hat{\theta}$ be any unbiased estimator of $\theta$ based on $n$ iid observations. Under regularity conditions:

$$\operatorname{Var}(\hat{\theta}) \geq \frac{1}{n\mathcal{I}(\theta)}$$

For the multivariate case with unbiased estimator $\hat{\boldsymbol{\theta}}$ of $\boldsymbol{\theta} \in \mathbb{R}^k$:

$$\operatorname{Cov}(\hat{\boldsymbol{\theta}}) \succeq \frac{1}{n}\mathcal{I}(\boldsymbol{\theta})^{-1}$$

(meaning $\operatorname{Cov}(\hat{\boldsymbol{\theta}}) - \frac{1}{n}\mathcal{I}(\boldsymbol{\theta})^{-1}$ is positive semidefinite).

Proof (scalar case, single observation):

We need to show $\operatorname{Var}(\hat{\theta}) \cdot \mathcal{I}(\theta) \geq 1$.

Since $\hat{\theta}$ is unbiased: $\mathbb{E}[\hat{\theta}] = \theta$. Differentiating with respect to $\theta$:

$$1 = \frac{\partial}{\partial\theta}\int \hat{\theta}(x)\, p(x;\theta)\, dx = \int \hat{\theta}(x)\, \frac{\partial \log p(x;\theta)}{\partial\theta}\, p(x;\theta)\, dx = \mathbb{E}[\hat{\theta} \cdot s(\theta;X)]$$

Now apply the Cauchy-Schwarz inequality to $\mathbb{E}[\hat{\theta} \cdot s]$:

$$1 = \mathbb{E}[\hat{\theta} \cdot s] = \mathbb{E}[(\hat{\theta} - \theta) \cdot s] \quad (\text{since } \mathbb{E}[s] = 0)$$

By Cauchy-Schwarz: $\mathbb{E}[(\hat{\theta} - \theta) \cdot s]^2 \leq \mathbb{E}[(\hat{\theta}-\theta)^2] \cdot \mathbb{E}[s^2]$

$$1 \leq \operatorname{Var}(\hat{\theta}) \cdot \mathcal{I}(\theta)$$ $$\operatorname{Var}(\hat{\theta}) \geq \frac{1}{\mathcal{I}(\theta)} \qquad \square$$

When is the bound tight? The CRB is achieved (with equality in Cauchy-Schwarz) if and only if $(\hat{\theta} - \theta) = c(\theta) \cdot s(\theta; X)$ for some function $c(\theta)$. This occurs precisely for exponential family distributions, and the efficient estimator is the MLE.

Biased estimator CRB. For biased estimators with $\mathbb{E}[\hat{\theta}] = \theta + b(\theta)$:

$$\operatorname{Var}(\hat{\theta}) \geq \frac{(1 + b'(\theta))^2}{\mathcal{I}(\theta)}$$

This shows that introducing bias (via regularisation) can actually reduce the CRB - a biased estimator faces a different, potentially less demanding bound.

Examples of efficient estimators:

The sample mean is efficient for all these single-parameter exponential families. This is not a coincidence - it follows from the structure of exponential families.

4.4 Fisher Information Matrix in Practice

For a neural network with parameters $\boldsymbol{\theta} \in \mathbb{R}^p$ defining a conditional distribution $p(y|\mathbf{x};\boldsymbol{\theta})$, the FIM is:

$$\mathcal{I}(\boldsymbol{\theta}) = \mathbb{E}_{(\mathbf{x},y)\sim p_{\text{data}}}\!\left[\nabla_{\boldsymbol{\theta}} \log p(y|\mathbf{x};\boldsymbol{\theta})\, \nabla_{\boldsymbol{\theta}} \log p(y|\mathbf{x};\boldsymbol{\theta})^\top\right]$$

This is an outer product of gradient vectors, making it a $p \times p$ PSD matrix. For modern LLMs with $p \sim 10^{11}$ parameters, this matrix is completely intractable to store (would require $10^{22}$ bytes).

Empirical FIM. In practice, approximate by sampling:

$$\hat{\mathcal{I}}(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^n \nabla_{\boldsymbol{\theta}} \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta})\, \nabla_{\boldsymbol{\theta}} \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta})^\top$$

This is the average squared gradient - exactly what is accumulated in gradient variance estimates.

Natural gradient (Amari, 1998). The ordinary gradient $\nabla_{\boldsymbol{\theta}} \mathcal{L}$ is the steepest ascent direction in Euclidean parameter space $\mathbb{R}^p$. But parameters do not live in Euclidean space - they live in the statistical manifold of distributions, where the natural metric is the Fisher information. The natural gradient is:

$$\tilde{\nabla}_{\boldsymbol{\theta}} \mathcal{L} = \mathcal{I}(\boldsymbol{\theta})^{-1} \nabla_{\boldsymbol{\theta}} \mathcal{L}$$

This is the steepest ascent direction in the metric defined by the FIM. Natural gradient is invariant to reparametrisation: if we change from $\boldsymbol{\theta}$ to $\boldsymbol{\phi} = g(\boldsymbol{\theta})$, the natural gradient update gives the same parameter change.

K-FAC approximation (Martens & Grosse, 2015). Full FIM inversion costs $O(p^3)$. K-FAC approximates $\mathcal{I}^{-1}$ by assuming layer-wise independence and Kronecker factorisation of each layer's FIM: $\mathcal{I}^{[l]} \approx A^{[l]} \otimes G^{[l]}$, where $A^{[l]} = \mathbb{E}[\mathbf{a}^{[l-1]}(\mathbf{a}^{[l-1]})^\top]$ (input covariance) and $G^{[l]} = \mathbb{E}[\delta^{[l]}(\delta^{[l]})^\top]$ (gradient covariance). Inverting the Kronecker product: $(A \otimes G)^{-1} = A^{-1} \otimes G^{-1}$, reducing cost from $O(p^3)$ to $O(d_{\text{in}}^3 + d_{\text{out}}^3)$ per layer.

Jeffreys prior. In Bayesian statistics (preview of Section04), the Jeffreys prior $p(\theta) \propto \sqrt{\mathcal{I}(\theta)}$ is the "uninformative" prior that is invariant to reparametrisation. For $\theta = p$ in Bernoulli: $\mathcal{I}(p) = 1/(p(1-p))$, so $p_J(p) \propto p^{-1/2}(1-p)^{-1/2}$, which is $\operatorname{Beta}(1/2, 1/2)$.

Preview: MAP Estimation. Adding a prior $p(\boldsymbol{\theta})$ and finding $\arg\max_{\boldsymbol{\theta}} [p(\mathcal{D}|\boldsymbol{\theta}) p(\boldsymbol{\theta})]$ gives the Maximum A Posteriori (MAP) estimate - MLE regularised by the log-prior. MAP with a Gaussian prior $\mathcal{N}(0, \lambda^{-1}I)$ gives L2-regularised MLE. The full Bayesian treatment of priors, posteriors, and conjugate families is in Section04 Bayesian Inference.

---

5. Maximum Likelihood Estimation

5.1 The Likelihood Principle

Definition 5.1 (Likelihood Function). Given observed data $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}$ and a parametric model $p(\cdot;\boldsymbol{\theta})$, the likelihood function is:

$$L(\boldsymbol{\theta}; \mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}) = \prod_{i=1}^n p(\mathbf{x}^{(i)};\boldsymbol{\theta})$$

The log-likelihood is:

$$\ell(\boldsymbol{\theta}) = \log L(\boldsymbol{\theta}) = \sum_{i=1}^n \log p(\mathbf{x}^{(i)};\boldsymbol{\theta})$$

The Maximum Likelihood Estimator (MLE) is:

$$\hat{\boldsymbol{\theta}}_{\text{MLE}} = \arg\max_{\boldsymbol{\theta} \in \Theta} \ell(\boldsymbol{\theta}) = \arg\max_{\boldsymbol{\theta} \in \Theta} \sum_{i=1}^n \log p(\mathbf{x}^{(i)};\boldsymbol{\theta})$$

The likelihood is not a probability. $L(\boldsymbol{\theta}; \mathbf{x})$ is a function of $\boldsymbol{\theta}$ for fixed data $\mathbf{x}$, not a probability distribution over $\boldsymbol{\theta}$. It does not integrate to 1 over $\boldsymbol{\theta}$. The statement "$L(\boldsymbol{\theta}) = 0.3$" means "the data has probability density 0.3 under parameter $\boldsymbol{\theta}$", not "there is a 30% chance $\boldsymbol{\theta}$ is the true parameter".

Why log? Three reasons:

1. Computational: $\prod_{i=1}^n p_i$ underflows to zero for large $n$ (e.g., $0.5^{1000} \approx 10^{-301}$); sums are numerically stable
2. Mathematical: sums are easier to differentiate and optimise than products
3. Statistical: log is a monotone transformation, so $\arg\max L = \arg\max \log L$

The likelihood principle states that all information about $\boldsymbol{\theta}$ in the data is contained in the likelihood function. Two datasets with proportional likelihoods should lead to the same inferences about $\boldsymbol{\theta}$. This is a philosophical principle - frequentists may not accept it fully (confidence intervals depend on the sampling procedure, not just the likelihood), but it underlies MLE and Bayesian inference alike.

5.2 Deriving MLEs: Scalar Parameters

The standard procedure for finding the MLE:

1. Write the log-likelihood $\ell(\theta) = \sum_i \log p(x^{(i)};\theta)$
2. Differentiate and set $\ell'(\theta) = 0$ (the score equation)
3. Verify it is a maximum (second derivative $< 0$ or global structure)
4. Check boundary cases (the maximum might be at $\Theta$'s boundary)

Bernoulli MLE. Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \operatorname{Bern}(p)$.

$$\ell(p) = \sum_{i=1}^n \left[x^{(i)}\log p + (1-x^{(i)})\log(1-p)\right] = S\log p + (n-S)\log(1-p)$$

where $S = \sum_{i=1}^n x^{(i)}$ is the total number of successes. Setting $\ell'(p) = 0$:

$$\frac{S}{p} - \frac{n-S}{1-p} = 0 \implies S(1-p) = p(n-S) \implies \hat{p}_{\text{MLE}} = \frac{S}{n} = \bar{x}$$

The MLE is the sample proportion - the intuitive answer.

Poisson MLE. Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \operatorname{Poisson}(\lambda)$.

$$\ell(\lambda) = \sum_{i=1}^n \left[x^{(i)}\log\lambda - \lambda - \log(x^{(i)}!)\right] = S\log\lambda - n\lambda + \text{const}$$

Setting $\ell'(\lambda) = S/\lambda - n = 0$: $\hat{\lambda}_{\text{MLE}} = S/n = \bar{x}$.

The MLE of the Poisson rate is the sample mean - the estimator of the mean equals the MLE because $\mathbb{E}[X] = \lambda$.

Exponential MLE. Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \operatorname{Exp}(\lambda)$, i.e., $p(x;\lambda) = \lambda e^{-\lambda x}$ for $x > 0$.

$$\ell(\lambda) = n\log\lambda - \lambda\sum_{i=1}^n x^{(i)}$$

Setting $\ell'(\lambda) = n/\lambda - \sum x^{(i)} = 0$: $\hat{\lambda}_{\text{MLE}} = n/\sum x^{(i)} = 1/\bar{x}$.

The MLE of the rate is the reciprocal of the sample mean - since $\mathbb{E}[X] = 1/\lambda$, this is natural.

Uniform MLE. Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{U}(0,\theta)$, $p(x;\theta) = 1/\theta$ for $x \in [0,\theta]$.

$$\ell(\theta) = -n\log\theta \cdot \mathbb{1}[\theta \geq \max_i x^{(i)}]$$

The likelihood is zero for $\theta < \max_i x^{(i)}$ (some observation would be outside $[0,\theta]$) and decreasing in $\theta$ for $\theta \geq \max_i x^{(i)}$. The MLE is at the boundary: $\hat{\theta}_{\text{MLE}} = \max_i x^{(i)}$.

This is an example where the score equation gives no solution (the log-likelihood has no interior critical point); the MLE is found by reasoning about the likelihood's shape. Also, the MLE here is biased: $\mathbb{E}[\max_i x^{(i)}] = n\theta/(n+1) < \theta$.

5.3 Deriving MLEs: Multivariate Gaussian

Let $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\boldsymbol{\mu}, \Sigma)$ with $\boldsymbol{\mu} \in \mathbb{R}^d$ and $\Sigma \in \mathbb{S}^d_{++}$.

The log-likelihood is:

$$\ell(\boldsymbol{\mu}, \Sigma) = -\frac{n}{2}\log\det(\Sigma) - \frac{1}{2}\sum_{i=1}^n (\mathbf{x}^{(i)} - \boldsymbol{\mu})^\top \Sigma^{-1}(\mathbf{x}^{(i)} - \boldsymbol{\mu}) - \frac{nd}{2}\log(2\pi)$$

MLE of $\boldsymbol{\mu}$: Taking the gradient with respect to $\boldsymbol{\mu}$ and setting to zero:

$$\nabla_{\boldsymbol{\mu}} \ell = \sum_{i=1}^n \Sigma^{-1}(\mathbf{x}^{(i)} - \boldsymbol{\mu}) = 0 \implies \hat{\boldsymbol{\mu}}_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n \mathbf{x}^{(i)} = \bar{\mathbf{x}}$$

MLE of $\Sigma$: Substituting $\hat{\boldsymbol{\mu}} = \bar{\mathbf{x}}$ and differentiating with respect to $\Sigma^{-1}$ (using matrix calculus identities $\nabla_A \log\det A = A^{-\top}$ and $\nabla_A \operatorname{tr}(BA) = B^\top$):

$$\frac{\partial \ell}{\partial \Sigma^{-1}} = \frac{n}{2}\Sigma - \frac{1}{2}\sum_{i=1}^n (\mathbf{x}^{(i)} - \bar{\mathbf{x}})(\mathbf{x}^{(i)} - \bar{\mathbf{x}})^\top = 0$$ $$\hat{\Sigma}_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n (\mathbf{x}^{(i)} - \bar{\mathbf{x}})(\mathbf{x}^{(i)} - \bar{\mathbf{x}})^\top$$

Bias of $\hat{\Sigma}_{\text{MLE}}$: By the same calculation as in the scalar case, $\mathbb{E}[\hat{\Sigma}_{\text{MLE}}] = \frac{n-1}{n}\Sigma$. The unbiased estimator is $S = \frac{1}{n-1}\sum_i (\mathbf{x}^{(i)} - \bar{\mathbf{x}})(\mathbf{x}^{(i)} - \bar{\mathbf{x}})^\top$.

For AI: Fitting a Gaussian to activations or embeddings is done in numerous ML methods. The empirical covariance of a layer's activations, used in whitening transforms, weight initialisation (Xavier/He initialisation matches the second moment), and covariance-regularised fine-tuning, all use this MLE formula. The multivariate Gaussian MLE is also the building block for Gaussian mixture models (EM algorithm) and linear discriminant analysis.

5.4 MLE as Cross-Entropy Minimisation

This connection is perhaps the most important in all of applied ML.

Theorem 5.4. For a classification model $p(y|\mathbf{x};\boldsymbol{\theta})$ and iid data $\{(\mathbf{x}^{(i)}, y^{(i)})\}_{i=1}^n$:

$$\hat{\boldsymbol{\theta}}_{\text{MLE}} = \arg\max_{\boldsymbol{\theta}} \sum_{i=1}^n \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta}) = \arg\min_{\boldsymbol{\theta}} \left[-\frac{1}{n}\sum_{i=1}^n \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta})\right]$$

The right-hand side is the negative log-likelihood (NLL) or cross-entropy loss:

$$\mathcal{L}_{\text{CE}}(\boldsymbol{\theta}) = -\frac{1}{n}\sum_{i=1}^n \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta})$$

Connection to KL divergence: Define the empirical distribution $p_{\text{data}}(y|\mathbf{x}) = \frac{1}{n}\sum_i \delta_{(x^{(i)}, y^{(i)})}$. Then:

$$\mathcal{L}_{\text{CE}} = -\frac{1}{n}\sum_i \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta}) = H(p_{\text{data}}) + D_{\mathrm{KL}}(p_{\text{data}} \| p_\theta)$$

Since $H(p_{\text{data}})$ does not depend on $\boldsymbol{\theta}$:

$$\arg\min_{\boldsymbol{\theta}} \mathcal{L}_{\text{CE}} = \arg\min_{\boldsymbol{\theta}} D_{\mathrm{KL}}(p_{\text{data}} \| p_{\boldsymbol{\theta}})$$

MLE minimises the KL divergence from the data distribution to the model. This is the information-theoretic interpretation of MLE.

For language models: An LLM defines $p_{\boldsymbol{\theta}}(x_t | x_1, \ldots, x_{t-1})$. Training by NLL is MLE:

$$\hat{\boldsymbol{\theta}} = \arg\min_{\boldsymbol{\theta}} -\frac{1}{T}\sum_{t=1}^T \log p_{\boldsymbol{\theta}}(x_t | x_{<t})$$

where the sum is over all tokens in the training corpus. This objective is used verbatim in every large language model - GPT-4 (OpenAI, 2023), LLaMA-3 (Meta, 2024), Gemini (Google, 2023), Claude (Anthropic, 2024). The perplexity $\exp(\mathcal{L}_{\text{CE}})$ is the standard evaluation metric.

Label smoothing as regularised MLE. Standard MLE uses hard labels $y^{(i)} \in \{0,1\}^K$ (one-hot). Label smoothing (Szegedy et al., 2016) uses soft targets $\tilde{y}^{(i)} = (1-\varepsilon)y^{(i)} + \varepsilon/K$, adding a small weight to incorrect classes. This is equivalent to regularising MLE by mixing the empirical distribution with a uniform prior - reducing overconfidence and improving calibration.

5.5 Properties of MLE

Under regularity conditions (the "Cramer-Rao conditions" on the model), MLE satisfies:

1. Consistency: $\hat{\boldsymbol{\theta}}_{\text{MLE}} \xrightarrow{P} \boldsymbol{\theta}^*$. The proof uses the fact that the expected log-likelihood $\mathbb{E}[\ell(\boldsymbol{\theta})/n]$ is uniquely maximised at the true parameter $\boldsymbol{\theta}^*$ (by the strict convexity of KL divergence), and the LLN ensures $\ell(\boldsymbol{\theta})/n \to \mathbb{E}[\log p(X;\boldsymbol{\theta})]$ uniformly.

2. Asymptotic normality:

$$\sqrt{n}\left(\hat{\boldsymbol{\theta}}_{\text{MLE}} - \boldsymbol{\theta}^*\right) \xrightarrow{d} \mathcal{N}\!\left(\mathbf{0},\, \mathcal{I}(\boldsymbol{\theta}^*)^{-1}\right)$$

For large $n$: $\hat{\boldsymbol{\theta}}_{\text{MLE}} \approx \mathcal{N}(\boldsymbol{\theta}^*, \frac{1}{n}\mathcal{I}(\boldsymbol{\theta}^*)^{-1})$.

3. Asymptotic efficiency: The asymptotic covariance $\frac{1}{n}\mathcal{I}^{-1}$ achieves the CRB. No consistent estimator can have smaller asymptotic variance (at every $\boldsymbol{\theta}$). MLE is asymptotically optimal.

4. Invariance under reparametrisation: If $\hat{\boldsymbol{\theta}}$ is the MLE of $\boldsymbol{\theta}$, then $g(\hat{\boldsymbol{\theta}})$ is the MLE of $g(\boldsymbol{\theta})$ for any function $g$ (not required to be injective). This is the invariance property of MLE.

Examples of invariance:

• MLE of $\sigma$ is $\hat{\sigma} = \sqrt{\hat{\sigma}^2_{\text{MLE}}}$ (not $s$, which is the unbiased estimator)
• MLE of $1/\lambda$ in Exponential is $\bar{x}$ (consistent with $\hat{\lambda} = 1/\bar{x}$)
• MLE of $p^2$ in Bernoulli is $\hat{p}^2 = \bar{x}^2$ (biased, but the MLE)

Regularity conditions (Cramer-Rao conditions). These are the sufficient conditions for the above properties:

1. The parameter space $\Theta$ is an open subset of $\mathbb{R}^k$
2. The model is identifiable
3. The support $\{x: p(x;\boldsymbol{\theta}) > 0\}$ does not depend on $\boldsymbol{\theta}$ (excludes Uniform)
4. The log-likelihood is three times differentiable in $\boldsymbol{\theta}$
5. The Fisher information matrix is positive definite at $\boldsymbol{\theta}^*$

When these fail (as in the Uniform example), MLE may still be the natural estimator but its properties differ.

5.6 Numerical MLE

For complex models, the score equation $\ell'(\boldsymbol{\theta}) = 0$ has no closed-form solution and must be solved numerically.

Gradient ascent on log-likelihood. The simplest approach:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t + \eta \nabla_{\boldsymbol{\theta}} \ell(\boldsymbol{\theta}_t)$$

For neural networks trained with mini-batches: stochastic gradient ascent on the mini-batch log-likelihood. This is exactly SGD on NLL loss with a negated gradient.

Newton-Raphson / Fisher scoring. Use second-order information:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - [H_\ell(\boldsymbol{\theta}_t)]^{-1} \nabla \ell(\boldsymbol{\theta}_t)$$

where $H_\ell$ is the Hessian of the log-likelihood. Replacing the Hessian with the expected negative Hessian (Fisher information) gives Fisher scoring:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t + \frac{1}{n}\mathcal{I}(\boldsymbol{\theta}_t)^{-1} \nabla \ell(\boldsymbol{\theta}_t)$$

This is exactly the natural gradient update. Newton-Raphson converges quadratically near the maximum - much faster than gradient ascent - but requires inverting a $k \times k$ matrix per step.

EM algorithm. For models with latent variables (mixtures, HMMs, VAEs), the EM algorithm alternates between:

• E-step: compute $Q(\boldsymbol{\theta};\boldsymbol{\theta}_t) = \mathbb{E}_{z|\mathbf{x};\boldsymbol{\theta}_t}[\log p(\mathbf{x},z;\boldsymbol{\theta})]$
• M-step: $\boldsymbol{\theta}_{t+1} = \arg\max_{\boldsymbol{\theta}} Q(\boldsymbol{\theta};\boldsymbol{\theta}_t)$

EM guarantees that $\ell(\boldsymbol{\theta}_{t+1}) \geq \ell(\boldsymbol{\theta}_t)$ - the log-likelihood is non-decreasing. EM is a specialised case of variational inference (to be developed in Section04 Bayesian Inference).

Numerical pitfalls:

• Overflow/underflow: compute $\ell = \sum \log p_i$, not $\log \prod p_i$ (avoid multiplying probabilities)
• Log-sum-exp trick: for softmax-based likelihoods, $\log \sum_k e^{z_k} = z_{\max} + \log\sum_k e^{z_k - z_{\max}}$
• Flat log-likelihoods: when $\mathcal{I}(\boldsymbol{\theta}) \approx 0$ near the solution, gradient methods converge extremely slowly; natural gradient methods help
• Local maxima: the log-likelihood may be multimodal for mixture models and neural networks; multiple restarts are necessary

---

6. Method of Moments

6.1 The Method of Moments

The method of moments (MoM), introduced by Karl Pearson (1894), is the oldest systematic estimation procedure. The idea is simple: match sample moments to population moments and solve for $\boldsymbol{\theta}$.

Definition 6.1 (Method of Moments Estimator). For a $k$-parameter model, the MoM estimator solves the system:

$$\hat{\mu}_j = \mathbb{E}[X^j;\boldsymbol{\theta}], \quad j = 1, \ldots, k$$

where $\hat{\mu}_j = \frac{1}{n}\sum_{i=1}^n (x^{(i)})^j$ is the $j$-th sample moment.

Example: Normal distribution. For $\mathcal{N}(\mu, \sigma^2)$ with 2 parameters:

• 1st moment equation: $\bar{x} = \mathbb{E}[X] = \mu \implies \hat{\mu} = \bar{x}$
• 2nd moment equation: $\frac{1}{n}\sum x_i^2 = \mathbb{E}[X^2] = \mu^2 + \sigma^2 \implies \hat{\sigma}^2 = \frac{1}{n}\sum x_i^2 - \bar{x}^2 = \frac{1}{n}\sum(x_i - \bar{x})^2$

For the Gaussian, MoM and MLE coincide.

Example: Gamma distribution. $X \sim \text{Gamma}(\alpha, \beta)$ with mean $\alpha/\beta$ and variance $\alpha/\beta^2$.

• $\hat{\mu}_1 = \bar{x} = \alpha/\beta$
• $\hat{\mu}_2 - \hat{\mu}_1^2 = s^2 = \alpha/\beta^2$

Solving: $\hat{\beta} = \bar{x}/s^2$, $\hat{\alpha} = \bar{x}^2/s^2$. Unlike the Gaussian case, the Gamma MLE requires numerical optimisation, making MoM attractive for quick estimation.

Example: Beta distribution. $X \sim \text{Beta}(\alpha, \beta)$ with mean $\alpha/(\alpha+\beta)$ and variance $\alpha\beta/[(\alpha+\beta)^2(\alpha+\beta+1)]$.

Setting $m = \bar{x}$ and $v = s^2$, solving the moment equations:

$$\hat{\alpha} = m\left(\frac{m(1-m)}{v} - 1\right), \quad \hat{\beta} = (1-m)\left(\frac{m(1-m)}{v} - 1\right)$$

This requires $v < m(1-m)$ (the variance must be less than the theoretical maximum for the Beta).

Properties of MoM:

• Consistent: by the LLN, sample moments converge to population moments; moment equations are continuous, so MoM estimators are consistent
• Asymptotically normal: by the multivariate CLT and delta method
• Inefficient: generally does not achieve the CRB; MLE has smaller asymptotic variance when it exists in closed form

6.2 Generalised Method of Moments

The Generalised Method of Moments (GMM), introduced by Hansen (1982), allows for more moment conditions than parameters.

Suppose we have $m$ moment conditions $\mathbb{E}[g(\mathbf{x};\boldsymbol{\theta})] = \mathbf{0}$, where $g: \mathbb{R}^d \times \Theta \to \mathbb{R}^m$ and $m \geq k$ (over-identified system).

Sample moment conditions: $\hat{\mathbf{g}}_n(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^n g(\mathbf{x}^{(i)};\boldsymbol{\theta})$

When $m > k$, we cannot set all $m$ conditions to zero simultaneously. The GMM estimator minimises the weighted quadratic form:

$$\hat{\boldsymbol{\theta}}_{\text{GMM}} = \arg\min_{\boldsymbol{\theta}} \hat{\mathbf{g}}_n(\boldsymbol{\theta})^\top W \hat{\mathbf{g}}_n(\boldsymbol{\theta})$$

for some positive-definite weighting matrix $W$. The optimal weight matrix (minimising asymptotic variance) is $W^* = S^{-1}$ where $S = \operatorname{Var}(g(\mathbf{x};\boldsymbol{\theta}^*))$.

GMM is widely used in econometrics. In ML, it appears implicitly when models are estimated by matching distributional moments rather than maximising likelihood.

6.3 MoM vs MLE

When to prefer MoM:

• The likelihood is intractable (latent variable models without EM)
• Speed matters more than efficiency
• Only the first few moments are of interest
• The distributional assumption may be wrong (moments are more robust)

When to prefer MLE:

• Full likelihood is tractable and the model is well-specified
• Maximum statistical efficiency is needed
• Regularity conditions are satisfied
• A standard reference distribution is being fitted

---

7. Confidence Intervals

7.1 The Frequentist Interpretation

Definition 7.1 (Confidence Interval). A random interval $[\hat{\theta}_L(\mathbf{X}), \hat{\theta}_U(\mathbf{X})]$ is a $(1-\alpha)$ confidence interval for $\theta$ if:

$$P_\theta\!\left(\hat{\theta}_L(\mathbf{X}) \leq \theta \leq \hat{\theta}_U(\mathbf{X})\right) \geq 1-\alpha \quad \text{for all } \theta \in \Theta$$

The key is that $\hat{\theta}_L$ and $\hat{\theta}_U$ are random (functions of the data $\mathbf{X}$), while $\theta$ is fixed (unknown but not random in the frequentist framework).

The correct interpretation: If we repeat the experiment many times and construct a 95% CI each time, 95% of those intervals will contain the true $\theta$. This specific computed interval either contains $\theta$ or it does not - there is no 95% probability attached to this particular interval after observing data.

Common misconceptions:

The Bayesian analogue - "there is a 95% probability that $\theta$ is in this interval" - is a credible interval and requires a prior distribution on $\theta$. The full treatment is in Section04 Bayesian Inference.

7.2 Exact Confidence Intervals via Pivoting

The pivotal method constructs CIs by finding a pivot: a function of data and parameter whose distribution is known and does not depend on $\theta$.

Gaussian mean, $\sigma^2$ known. Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$.

The pivot is $Z = \frac{\bar{x} - \mu}{\sigma/\sqrt{n}} \sim \mathcal{N}(0,1)$.

$$P\!\left(-z_{\alpha/2} \leq \frac{\bar{x}-\mu}{\sigma/\sqrt{n}} \leq z_{\alpha/2}\right) = 1 - \alpha$$

Solving for $\mu$: the $(1-\alpha)$ CI is $\left[\bar{x} - z_{\alpha/2}\frac{\sigma}{\sqrt{n}},\; \bar{x} + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right]$.

Gaussian mean, $\sigma^2$ unknown. Replacing $\sigma$ with the sample standard deviation $s$ changes the distribution: $T = \frac{\bar{x}-\mu}{s/\sqrt{n}} \sim t_{n-1}$ (Student's $t$ with $n-1$ degrees of freedom).

$(1-\alpha)$ CI: $\left[\bar{x} - t_{n-1,\alpha/2}\frac{s}{\sqrt{n}},\; \bar{x} + t_{n-1,\alpha/2}\frac{s}{\sqrt{n}}\right]$

As $n \to \infty$, $t_{n-1,\alpha/2} \to z_{\alpha/2}$ (Student's $t$ converges to standard normal).

STUDENT'S t DISTRIBUTION vs. NORMAL
========================================================================

  p(t)
   |
   |            --- Normal(0,1)
   |          ==== t(df=5)
   |        - - - t(df=1) [Cauchy]
   |
   |          +-----+         Heavier tails of t distribution
   |         ++     ++        reflect additional uncertainty
   |        ++       ++       from estimating \\sigma.
   |      --+         +--
   |--------------------------- t
       -3  -2  -1   0   1   2   3

  For df \\geq 30, t \\approx Normal - practical "large sample" threshold.

========================================================================

Gaussian variance CI. The pivot is $(n-1)s^2/\sigma^2 \sim \chi^2_{n-1}$, giving the CI:

$$\left[\frac{(n-1)s^2}{\chi^2_{n-1, \alpha/2}},\; \frac{(n-1)s^2}{\chi^2_{n-1, 1-\alpha/2}}\right]$$

Note this CI is asymmetric because the $\chi^2$ distribution is skewed.

7.3 Asymptotic Confidence Intervals

When exact pivots are unavailable, use the asymptotic normality of MLE: for large $n$, $\hat{\theta}_{\text{MLE}} \approx \mathcal{N}(\theta^*, \frac{1}{n}\mathcal{I}(\theta^*)^{-1})$.

Wald interval. Plugging in the MLE for the unknown $\theta^*$ in the Fisher information:

$$\hat{\theta}_{\text{MLE}} \pm z_{\alpha/2} \cdot \frac{1}{\sqrt{n\mathcal{I}(\hat{\theta}_{\text{MLE}})}}$$

For Bernoulli with $n$ observations: $\hat{p} \pm z_{\alpha/2}\sqrt{\hat{p}(1-\hat{p})/n}$.

Limitations of the Wald interval near boundaries. For $p$ near 0 or 1, the Wald interval can extend outside $[0,1]$. The Wilson score interval is better-behaved:

$$\frac{\hat{p} + z^2/(2n)}{1 + z^2/n} \pm \frac{z\sqrt{\hat{p}(1-\hat{p})/n + z^2/(4n^2)}}{1 + z^2/n}$$

where $z = z_{\alpha/2}$. This is the standard CI for proportions in ML evaluation.

Likelihood ratio interval. Based on the likelihood ratio statistic $\Lambda = 2[\ell(\hat{\theta}) - \ell(\theta_0)]$, which satisfies $\Lambda \xrightarrow{d} \chi^2_1$ under $H_0: \theta = \theta_0$. The CI is the set of $\theta_0$ values that pass the test:

$$\text{CI}_{1-\alpha} = \{\theta : 2[\ell(\hat{\theta}) - \ell(\theta)] \leq \chi^2_{1,\alpha}\}$$

This is often more accurate than the Wald interval for small $n$.

7.4 Bootstrap Confidence Intervals

The bootstrap (Efron, 1979) is a non-parametric resampling method for constructing CIs without distributional assumptions.

Non-parametric bootstrap algorithm:

1. From the original sample $\mathcal{X} = \{x^{(1)}, \ldots, x^{(n)}\}$, draw $B$ bootstrap samples $\mathcal{X}^*_1, \ldots, \mathcal{X}^*_B$ (each of size $n$, with replacement)
2. Compute the statistic $\hat{\theta}^*_b = T(\mathcal{X}^*_b)$ for each bootstrap sample
3. Estimate the sampling distribution of $\hat{\theta}$ by the empirical distribution of $\{\hat{\theta}^*_1, \ldots, \hat{\theta}^*_B\}$

Percentile bootstrap CI: Use the $(\alpha/2)$ and $(1-\alpha/2)$ quantiles of $\{\hat{\theta}^*_b\}$ as CI endpoints.

BCa (bias-corrected and accelerated) bootstrap: Adjusts for bias in the bootstrap distribution and non-constant acceleration. Typically more accurate than the percentile method for small $n$.

When to use bootstrap:

• The sampling distribution of $\hat{\theta}$ is non-standard (no closed-form pivot)
• The distributional assumptions for parametric CIs are suspect
• The statistic is complex (e.g., ratio of two MLEs, AUC, BLEU score)
• $n$ is moderate and asymptotic approximations are poor

Bootstrap for ML evaluation (example). To compute a 95% CI for test accuracy:

1. Let the test set be $\{(x^{(i)}, y^{(i)})\}_{i=1}^n$; accuracy $\hat{a} = \frac{1}{n}\sum \mathbb{1}[\hat{y}^{(i)} = y^{(i)}]$
2. Bootstrap: resample with replacement $B = 2000$ times, compute accuracy on each bootstrap test set
3. CI = [2.5th percentile, 97.5th percentile] of bootstrap accuracies

7.5 Confidence Intervals in ML Evaluation

Understanding CI coverage for ML evaluation prevents erroneous conclusions.

Binary accuracy. Test accuracy $\hat{a}$ with $n$ examples and $k$ correct. Wilson score CI:

$$\text{CI}_{95\%} \approx \hat{a} \pm 1.96\sqrt{\hat{a}(1-\hat{a})/n}$$

For $n = 1000$, $\hat{a} = 0.85$: CI = $[0.828, 0.872]$ - width $\pm 2.2\%$. For $n = 100$: CI = $[0.780, 0.920]$ - width $\pm 7\%$.

This shows why benchmark results on $n < 500$ test examples are statistically unreliable.

Comparing two models. If Model A achieves $\hat{a}_A = 0.852$ and Model B achieves $\hat{a}_B = 0.847$ on the same $n = 1000$ test examples, is A significantly better? The difference $\Delta = 0.005$ with standard error $\approx \sqrt{\hat{a}_A(1-\hat{a}_A)/n + \hat{a}_B(1-\hat{a}_B)/n} \approx 0.011$ gives a 95% CI for $\Delta$ of $[-0.017, +0.027]$, which includes zero. The models are not statistically distinguishable.

McNemar's test is more powerful than comparing independent CIs when models are evaluated on the same examples (the paired structure reduces variance). See Section03 Hypothesis Testing for the full treatment.

Multiple benchmark correction. When evaluating on $k$ benchmarks, the probability of at least one spuriously significant result rises. This is the multiple comparison problem - addressed fully in Section03. For reporting CIs, Bonferroni correction replaces $\alpha$ with $\alpha/k$, widening each CI to maintain family-wise coverage.

---

8. Asymptotic Theory

8.1 Convergence Modes

Before stating the asymptotic normality theorem, we review the convergence modes used.

Almost-sure (a.s.) convergence ($\xrightarrow{a.s.}$): $P(\lim_{n\to\infty} X_n = X) = 1$. The sequence converges on every sample path except a set of probability zero. This is the strongest mode; the Strong LLN gives $\bar{x}_n \xrightarrow{a.s.} \mu$.

Convergence in probability ($\xrightarrow{P}$): For every $\varepsilon > 0$, $P(|X_n - X| > \varepsilon) \to 0$. Weaker than a.s. convergence. The Weak LLN gives $\bar{x}_n \xrightarrow{P} \mu$. Sufficient for consistency.

Convergence in distribution ($\xrightarrow{d}$): $F_{X_n}(t) \to F_X(t)$ at every continuity point of $F_X$. The weakest mode; the CLT gives $\sqrt{n}(\bar{x}_n - \mu)/\sigma \xrightarrow{d} \mathcal{N}(0,1)$. Used for asymptotic normality.

Key theorems for manipulating convergence:

Slutsky's Theorem. If $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{P} c$ (a constant), then:

• $X_n + Y_n \xrightarrow{d} X + c$
• $X_n \cdot Y_n \xrightarrow{d} c \cdot X$
• $X_n / Y_n \xrightarrow{d} X/c$ (if $c \neq 0$)

Continuous Mapping Theorem. If $X_n \xrightarrow{P} X$ and $g$ is continuous, then $g(X_n) \xrightarrow{P} g(X)$. Same holds for $\xrightarrow{d}$.

Application: The MLE $\hat{\theta} \xrightarrow{P} \theta^*$ (consistency) implies $\mathcal{I}(\hat{\theta}) \xrightarrow{P} \mathcal{I}(\theta^*)$ by the continuous mapping theorem (assuming $\mathcal{I}$ is continuous). Then by Slutsky: $\sqrt{n\mathcal{I}(\hat{\theta})}(\hat{\theta} - \theta^*) \xrightarrow{d} \mathcal{N}(0,1)$.

8.2 The Delta Method

The delta method extends the CLT to smooth functions of asymptotically normal estimators.

Theorem 8.2 (Delta Method - Scalar). Suppose $\sqrt{n}(\hat{\theta} - \theta^*) \xrightarrow{d} \mathcal{N}(0, \sigma^2)$ and $g$ is differentiable at $\theta^*$ with $g'(\theta^*) \neq 0$. Then:

$$\sqrt{n}\left(g(\hat{\theta}) - g(\theta^*)\right) \xrightarrow{d} \mathcal{N}\!\left(0,\, [g'(\theta^*)]^2 \sigma^2\right)$$

Proof sketch: By differentiability, $g(\hat{\theta}) \approx g(\theta^*) + g'(\theta^*)(\hat{\theta} - \theta^*)$. Multiply through by $\sqrt{n}$: $\sqrt{n}(g(\hat{\theta}) - g(\theta^*)) \approx g'(\theta^*) \cdot \sqrt{n}(\hat{\theta} - \theta^*) \xrightarrow{d} g'(\theta^*)\mathcal{N}(0,\sigma^2) = \mathcal{N}(0, [g'(\theta^*)]^2\sigma^2)$.

Examples:

1. CI for log-odds. For Bernoulli with $\hat{p} = \bar{x}$, the log-odds are $\theta = \log[p/(1-p)]$. The MLE is $\hat{\theta} = \log[\hat{p}/(1-\hat{p})]$. The asymptotic variance: $g(p) = \log[p/(1-p)]$, $g'(p) = 1/[p(1-p)]$. So:

$$\operatorname{Var}(\hat{\theta}) \approx \frac{1}{n} \cdot \frac{1}{[p(1-p)]^2} \cdot p(1-p) = \frac{1}{np(1-p)}$$

1. CI for standard deviation. Given $\hat{\sigma}^2_{\text{MLE}} \approx \mathcal{N}(\sigma^2, 2\sigma^4/n)$ (asymptotic variance of variance estimator), the delta method with $g(v) = \sqrt{v}$, $g'(v) = 1/(2\sqrt{v})$:

$$\operatorname{Var}(\hat{\sigma}) \approx \frac{1}{4\sigma^2} \cdot \frac{2\sigma^4}{n} = \frac{\sigma^2}{2n}$$

Multivariate delta method. If $\sqrt{n}(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta}^*) \xrightarrow{d} \mathcal{N}(\mathbf{0}, \Sigma)$ and $g: \mathbb{R}^k \to \mathbb{R}^m$ is differentiable:

$$\sqrt{n}(g(\hat{\boldsymbol{\theta}}) - g(\boldsymbol{\theta}^*)) \xrightarrow{d} \mathcal{N}\!\left(\mathbf{0},\, J_g \Sigma J_g^\top\right)$$

where $J_g$ is the $m \times k$ Jacobian of $g$ at $\boldsymbol{\theta}^*$.

8.3 Asymptotic Normality of MLE

Theorem 8.3 (Asymptotic Normality of MLE). Under the Cramer-Rao regularity conditions, if $\hat{\boldsymbol{\theta}}_n$ is the MLE based on $n$ iid observations:

$$\sqrt{n}\left(\hat{\boldsymbol{\theta}}_n - \boldsymbol{\theta}^*\right) \xrightarrow{d} \mathcal{N}\!\left(\mathbf{0},\, \mathcal{I}(\boldsymbol{\theta}^*)^{-1}\right)$$

Proof sketch for the scalar case. Taylor-expand the score equation around $\boldsymbol{\theta}^*$:

$$0 = \frac{1}{\sqrt{n}} S_n(\hat{\theta}) \approx \frac{1}{\sqrt{n}} S_n(\theta^*) + \frac{1}{n} S_n'(\theta^*) \cdot \sqrt{n}(\hat{\theta} - \theta^*)$$

Rearranging:

$$\sqrt{n}(\hat{\theta} - \theta^*) \approx -\frac{\frac{1}{\sqrt{n}}S_n(\theta^*)}{\frac{1}{n}S_n'(\theta^*)}$$

Numerator: $\frac{1}{\sqrt{n}}S_n(\theta^*) = \frac{1}{\sqrt{n}}\sum_i s(\theta^*; x^{(i)})$. By CLT (scores are iid with mean 0 and variance $\mathcal{I}(\theta^*)$): $\frac{1}{\sqrt{n}}S_n(\theta^*) \xrightarrow{d} \mathcal{N}(0, \mathcal{I}(\theta^*))$.

Denominator: $\frac{1}{n}S_n'(\theta^*) = \frac{1}{n}\sum_i \frac{\partial^2 \log p}{\partial\theta^2}\big|_{\theta^*}$. By LLN: $\frac{1}{n}S_n'(\theta^*) \xrightarrow{P} \mathbb{E}\!\left[\frac{\partial^2 \log p}{\partial\theta^2}\right] = -\mathcal{I}(\theta^*)$.

By Slutsky: $\sqrt{n}(\hat{\theta} - \theta^*) \xrightarrow{d} \mathcal{N}(0, \mathcal{I}(\theta^*)) \cdot \frac{-1}{-\mathcal{I}(\theta^*)} = \mathcal{N}(0, \mathcal{I}(\theta^*)^{-1})$. $\square$

Implications for practice:

• Large sample CI: For $n \geq 50$ (rule of thumb), the $(1-\alpha)$ CI is $\hat{\theta} \pm z_{\alpha/2}/\sqrt{n\mathcal{I}(\hat{\theta})}$
• Convergence rate: The MLE error is $O(1/\sqrt{n})$ - doubling data halves the standard error
• Efficiency: The asymptotic covariance $\mathcal{I}^{-1}/n$ achieves the CRB - no consistent estimator is asymptotically better
• Standard errors in ML: The standard errors reported in logistic regression, GLMs, and survival models are all based on this theorem, using the observed or expected Fisher information

8.4 Misspecified Models

In practice, the model $\{p(\cdot;\boldsymbol{\theta})\}$ rarely contains the true data-generating distribution $p^*$. What happens when the model is misspecified?

Theorem 8.4 (MLE under Misspecification). Under regularity conditions, even when $p^* \notin \mathcal{M}$, the MLE converges to:

$$\hat{\boldsymbol{\theta}}_{\text{MLE}} \xrightarrow{P} \boldsymbol{\theta}^{**} = \arg\min_{\boldsymbol{\theta} \in \Theta} D_{\mathrm{KL}}(p^* \| p(\cdot;\boldsymbol{\theta}))$$

The "pseudo-true parameter" $\boldsymbol{\theta}^{**}$ is the closest point in the model family to the truth in KL divergence. The asymptotic distribution under misspecification is the sandwich estimator:

$$\sqrt{n}(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta}^{**}) \xrightarrow{d} \mathcal{N}\!\left(\mathbf{0},\, \mathcal{I}^{-1} V \mathcal{I}^{-1}\right)$$

where $V = \operatorname{Var}[s(\boldsymbol{\theta}^{**}; X)]$ is the actual variance of the score (not equal to $\mathcal{I}$ when misspecified). Under correct specification, $V = \mathcal{I}$ and the sandwich collapses to $\mathcal{I}^{-1}$.

Implications for LLM training: Language models trained by NLL minimisation are nearly always misspecified (the model cannot perfectly represent natural language). The trained model converges to the KL-minimising distribution within the model class. This is why language models exhibit characteristic failure modes related to the model architecture's inductive biases - they converge to the nearest representable distribution, not the true distribution.

---

9. Applications in Machine Learning

9.1 MLE Is Cross-Entropy Training

We derived in Section5.4 that MLE = cross-entropy minimisation. Here we examine the consequences.

NLL loss for common model types:

| Task | Model $p(y|\mathbf{x};\boldsymbol{\theta})$ | NLL loss | Standard name | | --- | --- | --- | --- | | Binary classification | $\operatorname{Bern}(\sigma(\mathbf{w}^\top\mathbf{x}))$ | $-y\log\hat{p} - (1-y)\log(1-\hat{p})$ | Binary cross-entropy | | Multi-class | $\operatorname{Cat}(\operatorname{softmax}(\mathbf{W}\mathbf{x}))$ | $-\sum_k y_k\log\hat{p}_k$ | Categorical cross-entropy | | Regression | $\mathcal{N}(\mathbf{w}^\top\mathbf{x}, \sigma^2)$ | $\frac{(y-\hat{y})^2}{2\sigma^2}$ | MSE (up to constant) | | Language model | $\prod_t p_\theta(x_t|x_{<t})$ | $-\frac{1}{T}\sum_t\log p_\theta(x_t|x_{<t})$ | NLL / perplexity |

Temperature scaling for calibration. After training a classifier, its softmax probabilities are often overconfident (Guo et al., 2017). Temperature scaling finds $T^* = \arg\min_T \mathcal{L}_{\text{NLL}}(\boldsymbol{\theta}, T)$ where logits are divided by $T$. This is a 1-parameter MLE problem on a calibration set, provably maintaining accuracy while improving Expected Calibration Error (ECE).

Label smoothing. Standard MLE on one-hot targets drives $\log p(y|\mathbf{x}) \to 0$ (i.e., $p \to 1$) for the correct class, making the model overconfident. Label smoothing (Szegedy et al., 2016) uses target $\tilde{y}_k = (1-\varepsilon)\mathbb{1}[k=y] + \varepsilon/K$ for small $\varepsilon$ (typically 0.1). This regularises MLE toward a mixture of the data distribution and uniform - reducing overconfidence without sacrificing accuracy.

9.2 Natural Gradient and Fisher Information

The problem with Euclidean gradient descent. The ordinary gradient $\nabla_{\boldsymbol{\theta}} \mathcal{L}$ is the direction of steepest ascent in Euclidean space. But this is parameterisation-dependent: rescaling parameters changes the gradient direction. Different parameterisations of the same model family give different gradient descent trajectories.

Information geometry. The space of probability distributions has a natural Riemannian metric - the Fisher information metric $g_{jk} = \mathcal{I}(\boldsymbol{\theta})_{jk}$. In this curved space, the steepest ascent direction is:

$$\tilde{\nabla}_{\boldsymbol{\theta}} \mathcal{L} = \mathcal{I}(\boldsymbol{\theta})^{-1} \nabla_{\boldsymbol{\theta}} \mathcal{L}$$

Natural gradient descent (Amari, 1998):

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \mathcal{I}(\boldsymbol{\theta}_t)^{-1} \nabla_{\boldsymbol{\theta}} \mathcal{L}$$

Properties:

• Parameterisation-invariant: changing parameterisation gives the same update in the function space
• Adaptive: steps are small in directions of high curvature (where the loss changes rapidly), large in flat directions
• Connection to Newton's method: natural gradient = Newton's method when the Hessian equals the FIM (true for GLMs at the maximum)

K-FAC (Martens & Grosse, 2015). Full FIM inversion is $O(p^3)$ - impossible for neural networks. K-FAC approximates $\mathcal{I}^{-1}$ layer-wise using the Kronecker structure:

$$\mathcal{I}^{[l]} \approx A^{[l-1]} \otimes G^{[l]}$$

where $A^{[l-1]} = \mathbb{E}[\mathbf{a}^{[l-1]}(\mathbf{a}^{[l-1]})^\top]$ and $G^{[l]} = \mathbb{E}[\delta^{[l]}(\delta^{[l]})^\top]$. This reduces the cost from $O(p^3)$ to $O(d_{\text{in}}^3 + d_{\text{out}}^3)$ per layer, making natural gradient feasible for moderate-sized networks.

Shampoo (Gupta et al., 2018) is a related second-order optimiser that maintains full-matrix preconditioners per parameter group, used at Google-scale training.

9.3 Confidence Intervals for Model Evaluation

The evaluation uncertainty problem. A model is evaluated on a test set of $n$ examples. The reported accuracy $\hat{a}$ is an estimate of the true accuracy $a^*$. Without a CI, it is impossible to determine if two models are genuinely different or differ only by chance.

Standard practice for ML papers:

• Wilson score CI for binary metrics (accuracy, F1 per class)
• Bootstrap CI for non-standard metrics (BLEU, ROUGE, perplexity)
• Correct for multiple benchmarks using Bonferroni or BH correction

Confidence interval for AUC. The Area Under the ROC Curve is a function of ranks, not a simple proportion. Its asymptotic distribution follows the Wilcoxon-Mann-Whitney form, giving a CI:

$$\widehat{\text{AUC}} \pm z_{\alpha/2} \sqrt{\frac{\widehat{\text{AUC}}(1-\widehat{\text{AUC}}) + (n_+-1)(Q_1-\widehat{\text{AUC}}^2) + (n_--1)(Q_2-\widehat{\text{AUC}}^2)}{n_+ n_-}}$$

For complex metrics like this, bootstrap is typically preferred.

LLM benchmark confidence. Major benchmarks (MMLU, HumanEval, HellaSwag) have 1000-14000 questions. For MMLU ($n \approx 14000$), the 95% Wilson CI for an accuracy of 70% is approximately $\pm 0.76\%$. For HumanEval ($n = 164$), the CI is $\pm 7.0\%$ - models within 7 percentage points cannot be reliably ranked.

9.4 Estimating Scaling Laws

Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) describe how model performance $L$ depends on model size $N$ (parameters), training tokens $D$, and compute $C$. These are regression problems - a direct application of estimation theory.

Chinchilla scaling law (Hoffmann et al., 2022). The loss function is modelled as:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

where $E, A, B, \alpha, \beta$ are estimated by nonlinear least squares MLE on $(N, D, L)$ observations from hundreds of training runs. The fitted parameters ($\alpha \approx 0.34$, $\beta \approx 0.28$) are MLE estimates with associated uncertainty.

The optimal allocation. For a fixed compute budget $C = 6ND$, the optimal $(N^*, D^*)$ is found by constrained optimisation of the fitted loss model. Chinchilla showed that LLaMA-1 (Touvron et al., 2023) and GPT-3 were undertrained relative to their model size - the optimal allocation uses smaller models trained on more tokens.

For AI: The Chinchilla scaling law computation is MLE applied at the scale of frontier AI research. The "parameters" being estimated ($\alpha, \beta, A, B, E$) are 5 numbers that determine the optimal architecture and training strategy for models costing millions of dollars to train.

9.5 Fisher Information in Fine-Tuning

Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2017). In continual learning, a model trained sequentially on tasks $T_1, T_2, \ldots$ tends to forget earlier tasks - catastrophic forgetting. EWC prevents this by penalising changes to weights that were important for task $T_1$:

$$\mathcal{L}_{\text{EWC}}(\boldsymbol{\theta}) = \mathcal{L}_{T_2}(\boldsymbol{\theta}) + \frac{\lambda}{2} \sum_j F_j (\theta_j - \theta^*_{1,j})^2$$

where $F_j$ is the $j$-th diagonal element of the Fisher information matrix computed on task $T_1$'s data: $F_j = \frac{1}{n}\sum_i (\partial \log p(y^{(i)}|\mathbf{x}^{(i)};\boldsymbol{\theta}^*_1)/\partial\theta_j)^2$.

Parameters with high $F_j$ are those to which the task-1 likelihood is most sensitive - changing them most damages task-1 performance. EWC protects these parameters with strong quadratic penalties.

LoRA as low-rank MLE (Hu et al., 2022). LoRA fine-tunes large models by constraining weight updates to be low-rank: $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$. This is MLE with a rank constraint on the parameter update - the MLE over a restricted parameter manifold. The rank $r$ controls the trade-off between expressivity and the number of estimated parameters.

Optimal Brain Damage (LeCun et al., 1990). Neural network pruning by removing parameters $\theta_j$ that, according to a second-order Taylor expansion, increase the loss least. The per-parameter importance is approximately $\frac{1}{2} H_{jj} \theta_j^2$, where $H_{jj}$ is the diagonal Hessian. Replacing $H_{jj}$ with $F_{jj}$ (diagonal FIM) recovers FIM-based pruning - estimation theory meets efficient neural architecture.

---

10. Common Mistakes

---

11. Exercises

Exercise 1 [*] MLE for Bernoulli and Poisson.

(a) Given $n = 10$ coin flips with outcomes $\{H, H, T, H, T, T, H, H, H, T\}$ (6 heads, 4 tails), compute $\hat{p}_{\text{MLE}}$ and its standard error $\sqrt{\hat{p}(1-\hat{p})/n}$.

(b) For the Poisson model $\operatorname{Poisson}(\lambda)$ with $n$ iid observations summing to $S$, derive the MLE $\hat{\lambda}$ by solving the score equation and verify it is a maximum (second derivative < 0).

(c) Using the MLE from (b) and the Fisher information $\mathcal{I}(\lambda) = 1/\lambda$, verify the MLE achieves the CRB: $\operatorname{Var}(\hat{\lambda}) = \lambda/n = 1/(n\mathcal{I}(\lambda))$.

Exercise 2 [*] Bias of the MLE variance estimator.

Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$ with both parameters unknown.

(a) Prove that $\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n (x^{(i)} - \bar{x})^2$ has $\mathbb{E}[\hat{\sigma}^2_{\text{MLE}}] = \frac{n-1}{n}\sigma^2$. (Hint: write $x^{(i)} - \bar{x}$ in terms of $x^{(i)} - \mu$ and $\bar{x} - \mu$.)

(b) Show that the corrected estimator $s^2 = \frac{n}{n-1}\hat{\sigma}^2_{\text{MLE}}$ is unbiased.

(c) By MLE invariance, $\hat{\sigma}_{\text{MLE}} = \sqrt{\hat{\sigma}^2_{\text{MLE}}}$. Is this an unbiased estimator of $\sigma$? If not, what is the sign of its bias? Use Jensen's inequality.

Exercise 3 [*] Cramer-Rao bound verification.

For $n$ iid observations from $\operatorname{Exp}(\lambda)$ with $p(x;\lambda) = \lambda e^{-\lambda x}$:

(a) Compute the score $s(\lambda; x) = \partial \log p(x;\lambda)/\partial \lambda$ and verify $\mathbb{E}[s] = 0$.

(b) Compute the Fisher information $\mathcal{I}(\lambda)$ using both the variance-of-score and negative-expected-Hessian formulas and verify they are equal.

(c) Show that $\hat{\lambda} = n/\sum_{i=1}^n x^{(i)}$ is the MLE and compute $\operatorname{Var}(\hat{\lambda})$ using the delta method applied to $g(\bar{x}) = 1/\bar{x}$ with $\operatorname{Var}(\bar{x}) = (1/\lambda^2)/n$. Verify that $\operatorname{Var}(\hat{\lambda}) = \lambda^2/n = 1/(n\mathcal{I}(\lambda))$ - the CRB is achieved.

Exercise 4 [**] Method of Moments for the Beta distribution.

Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \operatorname{Beta}(\alpha, \beta)$ with moments $\mathbb{E}[X] = \alpha/(\alpha+\beta)$ and $\operatorname{Var}(X) = \alpha\beta/[(\alpha+\beta)^2(\alpha+\beta+1)]$.

(a) Derive the MoM estimators $\hat{\alpha}$ and $\hat{\beta}$ in terms of $\bar{x}$ and $s^2 = \frac{1}{n}\sum(x_i - \bar{x})^2$.

(b) Show that the MoM estimates are only valid when $s^2 < \bar{x}(1-\bar{x})$. What is the statistical interpretation of this constraint?

(c) Generate $n = 200$ samples from $\operatorname{Beta}(2, 5)$ and compute both the MoM and the MLE (via numerical optimisation). Compare the estimates and their standard errors. Which is more efficient?

Exercise 5 [**] Bootstrap confidence interval.

Let $\mathbf{x} = \{3.1, 1.4, 2.5, 4.2, 1.8, 3.7, 2.1, 4.8, 1.2, 3.3\}$ (a sample of size 10).

(a) Compute the sample median $\tilde{x}$ and the 95% asymptotic CI for the median using the fact that, for a distribution with density $f(\tilde{x})$ at the true median, $\operatorname{Var}(\text{sample median}) \approx \frac{1}{4nf(\tilde{x})^2}$.

(b) Implement the non-parametric bootstrap with $B = 2000$ resamples and compute the percentile bootstrap CI for the median. Compare the width to the asymptotic CI.

(c) Is the bootstrap CI or the asymptotic CI more reliable here, and why?

Exercise 6 [**] Asymptotic normality simulation.

Verify the asymptotic normality of the Bernoulli MLE empirically.

(a) For true parameter $p = 0.3$ and sample sizes $n \in \{10, 30, 100, 500\}$, simulate $M = 5000$ MLEs $\hat{p}^{(m)} = \bar{x}^{(m)}$ and plot their distributions.

(b) For each $n$, standardise: $Z^{(m)} = \frac{\hat{p}^{(m)} - 0.3}{\sqrt{0.3 \cdot 0.7/n}}$ and overlay the $\mathcal{N}(0,1)$ PDF. At what $n$ does the approximation appear adequate?

(c) Compute the empirical coverage of the 95% Wald CI at each $n$. At what $n$ does coverage first reach 94%-96%? Does the CRB bound hold: is $n \cdot \operatorname{Var}(\hat{p})$ always $\geq 1/\mathcal{I}(0.3) = p(1-p)$?

Exercise 7 [***] Fisher information for a neural network layer.

Consider a one-layer softmax classifier $p(y|\mathbf{x};\mathbf{W}) = \operatorname{softmax}(\mathbf{W}\mathbf{x})_y$ for $y \in \{1,\ldots,K\}$, where $\mathbf{W} \in \mathbb{R}^{K \times d}$.

(a) Derive the score $\nabla_{\mathbf{W}} \log p(y|\mathbf{x};\mathbf{W}) = (\mathbf{e}_y - \hat{\mathbf{p}}) \mathbf{x}^\top$ where $\hat{\mathbf{p}} = \operatorname{softmax}(\mathbf{W}\mathbf{x})$.

(b) Show that the Fisher information matrix (as a $Kd \times Kd$ matrix) is:

$$\mathcal{I}(\mathbf{W}) = \mathbb{E}_{\mathbf{x}}\!\left[(\operatorname{Diag}(\hat{\mathbf{p}}) - \hat{\mathbf{p}}\hat{\mathbf{p}}^\top) \otimes \mathbf{x}\mathbf{x}^\top\right]$$

(c) Identify the two factors in the K-FAC approximation: $A = \mathbb{E}[\mathbf{x}\mathbf{x}^\top]$ (input covariance) and $G = \mathbb{E}[\operatorname{Diag}(\hat{\mathbf{p}}) - \hat{\mathbf{p}}\hat{\mathbf{p}}^\top]$ (output covariance). Simulate this for $d = 10$, $K = 5$, and compare $A \otimes G$ to the full FIM.

Exercise 8 [***] Chinchilla scaling law estimation.

The Chinchilla model (Hoffmann et al., 2022) estimates language model loss as $L(N,D) = E + A/N^\alpha + B/D^\beta$.

(a) Given synthetic training run data (simulate 50 runs with $(N, D)$ pairs and losses with added Gaussian noise), fit the 5 parameters $(E, A, B, \alpha, \beta)$ by nonlinear least squares using scipy.optimize.curve_fit.

(b) Compute 95% CIs for each parameter using the covariance matrix returned by curve_fit (which estimates $\sigma^2(J^\top J)^{-1}$, where $J$ is the Jacobian at the solution).

(c) For a fixed compute budget $C = 6ND$ (FLOPs), find the optimal $(N^*, D^*)$ by minimising $L(N, C/(6N))$ over $N$. How sensitive is this optimal allocation to uncertainty in $\alpha$ and $\beta$?

---

12. Why This Matters for AI (2026 Perspective)

---

13. Conceptual Bridge

Estimation theory is the bridge between probability and action. Probability theory (Chapter 6) defines random variables, distributions, and their properties - it answers "if we know the model, what data should we expect?" Estimation theory inverts this: "given data, what can we infer about the model?" This inversion from data to parameters is the core operation of every machine learning training algorithm.

The section you have just completed builds on Descriptive Statistics (Section01) in a crucial way: sample statistics like $\bar{x}$ and $s^2$ were introduced there as data summaries; here they are elevated to estimators - random variables with sampling distributions, bias, variance, and convergence properties. The concept of Bessel's correction ($n-1$ denominator) was motivated in Section01 by practicality; here it is derived from the requirement of unbiasedness.

Looking forward to Hypothesis Testing (Section03): the confidence intervals of this section have a duality with hypothesis tests - rejecting $H_0: \theta = \theta_0$ at level $\alpha$ is equivalent to $\theta_0$ falling outside the $(1-\alpha)$ CI. The sampling distributions developed here (Gaussian, Student's $t$, chi-squared, $F$) are the exact distributions that hypothesis tests use.

The next major extension is Bayesian Inference (Section04), which reframes the estimation problem by treating $\boldsymbol{\theta}$ as a random variable with a prior distribution $p(\boldsymbol{\theta})$. The posterior $p(\boldsymbol{\theta}|\mathcal{D}) \propto L(\boldsymbol{\theta};\mathcal{D})\, p(\boldsymbol{\theta})$ combines the likelihood (covered here) with the prior. MAP estimation - maximising the posterior - is MLE regularised by the log-prior. L2 regularisation is MAP with a Gaussian prior; L1 regularisation is MAP with a Laplace prior. The full treatment of conjugate priors, credible intervals, and MCMC is in Section04.

ESTIMATION THEORY IN THE CURRICULUM
========================================================================

  Chapter 6 - Probability Theory
  |  Section01 Random Variables (distributions, CDF/PDF)
  |  Section04 Expectation and Moments (E[X], Var(X), CLT, LLN)
  |  Section03 Joint Distributions (Bayes' theorem)
  +------------------------------------------+
                                             | prerequisites
  Chapter 7 - Statistics                     v
  |  Section01 Descriptive Statistics    ----------------------------------
  |       sample mean, variance,   data summaries -> estimators
  |       covariance, correlation
  |
  +--> Section02 ESTIMATION THEORY (this section)
  |       - Point estimators: bias, variance, MSE
  |       - Cramer-Rao bound, Fisher information
  |       - MLE: derivation, properties, examples
  |       - Asymptotic normality, delta method
  |       - Confidence intervals (frequentist)
  |       - MLE = cross-entropy training
  |
  +--> Section03 Hypothesis Testing      (duality: CIs <-> tests)
  |       p-values, power, t/z/\\chi^2 tests, A/B testing
  |
  +--> Section04 Bayesian Inference      (extension: add prior to likelihood)
  |       posterior = likelihood x prior, MAP = regularised MLE
  |       conjugate priors, MCMC, variational inference
  |
  +--> Section05 Time Series             (sequential estimation, Kalman filter)
  |
  +--> Section06 Regression Analysis     (applied MLE: OLS, Ridge, Lasso)

  Chapter 8 - Optimisation         (natural gradient uses FIM)
  Chapter 9 - Information Theory   (FIM and Shannon information)

========================================================================

The Fisher information matrix is the central object connecting estimation theory outward: it bounds estimation variance (CRB), defines the geometry of natural gradient optimisation (Ch8), underpins the Laplace approximation in Bayesian inference (Section04), and is related to the Shannon capacity of statistical experiments (Ch9). Understanding the FIM deeply is understanding the mathematical infrastructure of modern ML training.

---

End of Section02 Estimation Theory

<- Back to Chapter 7: Statistics | Next: Hypothesis Testing ->

---

Appendix A: Exponential Families

Exponential families are the most important class of parametric models in statistics, unifying the Gaussian, Bernoulli, Poisson, Exponential, Gamma, Beta, and dozens of other distributions under one framework.

Definition A.1 (Exponential Family). A parametric family $\{p(x;\boldsymbol{\eta})\}$ is an exponential family if it can be written in the form:

$$p(x;\boldsymbol{\eta}) = h(x) \exp\!\left(\boldsymbol{\eta}^\top T(x) - A(\boldsymbol{\eta})\right)$$

where:

• $\boldsymbol{\eta} \in \mathbb{R}^k$ is the natural (canonical) parameter
• $T(x) \in \mathbb{R}^k$ is the sufficient statistic (vector of natural statistics)
• $h(x) \geq 0$ is the base measure
• $A(\boldsymbol{\eta}) = \log \int h(x) e^{\boldsymbol{\eta}^\top T(x)} dx$ is the log-partition function (ensures normalisation)

The log-partition function encodes all moments: $\nabla_{\boldsymbol{\eta}} A(\boldsymbol{\eta}) = \mathbb{E}[T(X)]$ and $\nabla^2_{\boldsymbol{\eta}} A(\boldsymbol{\eta}) = \operatorname{Cov}(T(X))$.

Verification examples:

Bernoulli. $p(x;p) = p^x(1-p)^{1-x}$. Write as $\exp(x\log(p/(1-p)) + \log(1-p))$. Natural parameter $\eta = \log(p/(1-p))$ (log-odds), sufficient statistic $T(x) = x$, $A(\eta) = \log(1 + e^\eta)$.

Gaussian. $\mathcal{N}(\mu,\sigma^2)$: natural parameters $\boldsymbol{\eta} = (\mu/\sigma^2, -1/(2\sigma^2))$, sufficient statistics $T(x) = (x, x^2)$.

Key properties of exponential families for MLE:

1. MLE score equation: $\hat{\boldsymbol{\eta}}_{\text{MLE}}$ satisfies $\mathbb{E}_{\hat{\boldsymbol{\eta}}}[T(X)] = \frac{1}{n}\sum_i T(x^{(i)})$ - the MLE moment-matches the sufficient statistics.

2. MLE = MoM: For exponential families, MLE and method of moments coincide (they both set $\mathbb{E}[T] = \hat{T}$).

3. Efficient MLE: The MLE achieves the CRB for all exponential families - it is always efficient.

4. Log-likelihood is concave: For minimal exponential families, $\ell(\boldsymbol{\eta})$ is strictly concave - the MLE is the unique global maximum, and Newton-Raphson converges from any starting point.

5. Complete sufficient statistics: $T(x^{(1)}, \ldots, x^{(n)}) = \sum_i T(x^{(i)})$ is a complete sufficient statistic, so the MLE is the UMVUE by Lehmann-Scheffe.

Importance for ML: The categorical distribution (softmax output of neural networks), Gaussian (regression, VAE latent space), Bernoulli (binary classification), and Poisson (count data, Poisson regression) are all exponential families. The VAE encoder outputs the natural parameters $(\boldsymbol{\mu}, \boldsymbol{\sigma})$ of the Gaussian posterior - the sufficient statistics. Generalised linear models (GLMs, of which logistic regression is a special case) are defined precisely as: exponential family response + linear natural parameter $\boldsymbol{\eta} = \boldsymbol{W}\mathbf{x}$.

---

Appendix B: Sufficient Statistics - Detailed Examples

B.1 The Exponential Distribution

For $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \operatorname{Exp}(\lambda)$:

$$p(\mathbf{x};\lambda) = \lambda^n \exp\!\left(-\lambda \sum_{i=1}^n x^{(i)}\right) \cdot \mathbb{1}[x^{(i)} \geq 0 \; \forall i]$$

By the factorisation theorem: $T = \sum_i x^{(i)}$ (total waiting time) is a sufficient statistic. It factors as $g(T,\lambda) \cdot h(\mathbf{x})$ with $g(T,\lambda) = \lambda^n e^{-\lambda T}$ and $h(\mathbf{x}) = \prod \mathbb{1}[x^{(i)}\geq 0]$.

MLE: $\hat{\lambda} = n/T = 1/\bar{x}$, which depends on the data only through $T$ - consistent with sufficiency.

B.2 Order Statistics and the Uniform Distribution

For $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{U}(0,\theta)$:

$$p(\mathbf{x};\theta) = \frac{1}{\theta^n} \prod_{i=1}^n \mathbb{1}[0 \leq x^{(i)} \leq \theta] = \frac{1}{\theta^n}\mathbb{1}[\max_i x^{(i)} \leq \theta]\mathbb{1}[\min_i x^{(i)} \geq 0]$$

The sufficient statistic is $T = X_{(n)} = \max_i x^{(i)}$ (the maximum order statistic). The MLE $\hat{\theta} = X_{(n)}$ is a function of $T$ as expected.

Bias correction. Since $\mathbb{E}[X_{(n)}] = n\theta/(n+1)$, the unbiased estimator is $\tilde{\theta} = \frac{n+1}{n}X_{(n)}$. By Rao-Blackwell applied to $\tilde{\theta}' = \frac{n+1}{n}x^{(1)}$ (use only first observation): conditioning on $T = X_{(n)}$ gives $\mathbb{E}[\tilde{\theta}' | T] = \frac{n+1}{n}X_{(n)}\cdot\frac{1}{n} \cdot n = \frac{n+1}{n}X_{(n)}$, so the Rao-Blackwell estimator is indeed $\tilde{\theta}$.

---

Appendix C: The EM Algorithm - Connecting MLE to Latent Variable Models

Many models of interest (Gaussian mixture models, HMMs, topic models, VAEs) have latent variables $Z$ in addition to observed data $X$. The complete-data log-likelihood $\log p(X, Z; \boldsymbol{\theta})$ would be easy to maximise if $Z$ were observed, but we only observe $X$.

The EM algorithm maximises the marginal log-likelihood $\ell(\boldsymbol{\theta}) = \log p(X;\boldsymbol{\theta}) = \log \sum_Z p(X,Z;\boldsymbol{\theta})$ by iterating two steps:

E-step: Compute the expected complete-data log-likelihood under the current posterior over $Z$:

$$Q(\boldsymbol{\theta};\boldsymbol{\theta}^{(t)}) = \mathbb{E}_{Z \mid X;\boldsymbol{\theta}^{(t)}}[\log p(X, Z;\boldsymbol{\theta})]$$

M-step: Maximise:

$$\boldsymbol{\theta}^{(t+1)} = \arg\max_{\boldsymbol{\theta}} Q(\boldsymbol{\theta};\boldsymbol{\theta}^{(t)})$$

Why EM guarantees $\ell(\boldsymbol{\theta}^{(t+1)}) \geq \ell(\boldsymbol{\theta}^{(t)})$: The evidence lower bound (ELBO):

$$\ell(\boldsymbol{\theta}) = Q(\boldsymbol{\theta};\boldsymbol{\theta}^{(t)}) + H(\boldsymbol{\theta}^{(t)})$$

where $H$ is the entropy term not involving $\boldsymbol{\theta}$. The M-step maximises $Q$, and the E-step tightens the bound at the new $\boldsymbol{\theta}^{(t+1)}$.

Gaussian Mixture Model (GMM). For $K$-component GMM with means $\boldsymbol{\mu}_k$, covariances $\Sigma_k$, and mixing weights $\pi_k$:

• E-step: Compute responsibilities $r_{ik} = \frac{\pi_k \mathcal{N}(\mathbf{x}^{(i)};\boldsymbol{\mu}_k,\Sigma_k)}{\sum_j \pi_j \mathcal{N}(\mathbf{x}^{(i)};\boldsymbol{\mu}_j,\Sigma_j)}$
• M-step: Update parameters:

$$\hat{\pi}_k = \frac{\sum_i r_{ik}}{n}, \quad \hat{\boldsymbol{\mu}}_k = \frac{\sum_i r_{ik}\mathbf{x}^{(i)}}{\sum_i r_{ik}}, \quad \hat{\Sigma}_k = \frac{\sum_i r_{ik}(\mathbf{x}^{(i)}-\hat{\boldsymbol{\mu}}_k)(\mathbf{x}^{(i)}-\hat{\boldsymbol{\mu}}_k)^\top}{\sum_i r_{ik}}$$

Each M-step is a weighted MLE for a Gaussian - exactly the closed-form solutions from Section5.3, but with fractional weights.

For AI: The EM algorithm is the algorithmic prototype for variational inference in VAEs (Section04). In the VAE, the E-step is replaced by the encoder $q_\phi(\mathbf{z}|\mathbf{x})$ (an amortised posterior approximation) and the M-step is replaced by joint gradient ascent on the ELBO with respect to both encoder parameters $\phi$ and decoder parameters $\boldsymbol{\theta}$.

---

Appendix D: Cramer-Rao Bound - Multivariate Proof and Geometry

Multivariate CRB (full proof). Let $\hat{\boldsymbol{\theta}}$ be an unbiased estimator of $\boldsymbol{\theta} \in \mathbb{R}^k$. Define the score vector $\mathbf{s}(\boldsymbol{\theta};\mathbf{x}) = \nabla_{\boldsymbol{\theta}} \log p(\mathbf{x};\boldsymbol{\theta}) \in \mathbb{R}^k$.

Since $\hat{\boldsymbol{\theta}}$ is unbiased: $\mathbb{E}[\hat{\boldsymbol{\theta}}] = \boldsymbol{\theta}$. Differentiating with respect to $\theta_j$:

$$I_j = \frac{\partial}{\partial\theta_j}\mathbb{E}[\hat{\boldsymbol{\theta}}] = \mathbb{E}[\hat{\boldsymbol{\theta}} \cdot s_j(\boldsymbol{\theta};\mathbf{x})] = \mathbb{E}[(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta}) s_j]$$

In matrix form: $I_k = \mathbb{E}[(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta})\mathbf{s}^\top]$ (a $k \times k$ identity).

By the matrix Cauchy-Schwarz (Lowner ordering):

$$\begin{pmatrix} \operatorname{Cov}(\hat{\boldsymbol{\theta}}) & I_k \\ I_k & \mathcal{I}(\boldsymbol{\theta}) \end{pmatrix} \succeq 0$$

The Schur complement condition gives: $\operatorname{Cov}(\hat{\boldsymbol{\theta}}) - \mathcal{I}(\boldsymbol{\theta})^{-1} \succeq 0$, i.e., $\operatorname{Cov}(\hat{\boldsymbol{\theta}}) \succeq \mathcal{I}(\boldsymbol{\theta})^{-1}$. $\square$

Information-geometric interpretation. The statistical manifold $\mathcal{M} = \{p(\cdot;\boldsymbol{\theta}) : \boldsymbol{\theta} \in \Theta\}$ is a Riemannian manifold with the Fisher-Rao metric $g_{jl} = \mathcal{I}(\boldsymbol{\theta})_{jl}$. The CRB states that the covariance of any unbiased estimator is at least as large as the inverse metric - the "volume" of estimation uncertainty is bounded below by the curvature of the statistical manifold.

This geometric view motivates the natural gradient: gradient descent in the Fisher-Rao metric on the statistical manifold corresponds to the natural gradient update $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta \mathcal{I}^{-1}\nabla\ell$.

---

Appendix E: The Score Test, Wald Test, and Likelihood Ratio Test

These three test statistics form the classical trinity of asymptotic tests in statistics. While their full treatment belongs in Section03 Hypothesis Testing, their derivations from MLE theory belong here.

For testing $H_0: \boldsymbol{\theta} = \boldsymbol{\theta}_0$:

Score (Rao) test: $\Lambda_S = \frac{1}{n}\mathbf{s}_n(\boldsymbol{\theta}_0)^\top \mathcal{I}(\boldsymbol{\theta}_0)^{-1} \mathbf{s}_n(\boldsymbol{\theta}_0) \xrightarrow{d} \chi^2_k$ under $H_0$.

Uses only the restricted model - does not require fitting the unrestricted MLE.

Wald test: $\Lambda_W = n(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta}_0)^\top \mathcal{I}(\hat{\boldsymbol{\theta}})(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta}_0) \xrightarrow{d} \chi^2_k$ under $H_0$.

Uses only the unrestricted MLE.

Likelihood ratio test: $\Lambda_{LR} = 2[\ell(\hat{\boldsymbol{\theta}}) - \ell(\boldsymbol{\theta}_0)] \xrightarrow{d} \chi^2_k$ under $H_0$.

Requires fitting both restricted and unrestricted models. By Wilks' theorem (1938), this converges to the chi-squared distribution.

Asymptotic equivalence. Under $H_0$: $\Lambda_S = \Lambda_W = \Lambda_{LR} + O_P(n^{-1/2})$ - all three are first-order equivalent. They differ in finite samples. The LRT is generally most accurate for small $n$.

These tests are developed fully in Section03 Hypothesis Testing, including the Neyman-Pearson lemma, p-values, and power calculations.

---

---

Appendix F: Sampling Distributions of Classical Statistics

Understanding the exact (finite-sample) distributions of key estimators is essential for constructing valid hypothesis tests and confidence intervals when $n$ is small.

F.1 Distribution of the Sample Mean

Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$.

$$\bar{x} = \frac{1}{n}\sum_{i=1}^n x^{(i)} \sim \mathcal{N}\!\left(\mu, \frac{\sigma^2}{n}\right)$$

This is an exact result (not asymptotic) - the sum of independent Gaussians is Gaussian. The standardised version $Z = \frac{\bar{x}-\mu}{\sigma/\sqrt{n}} \sim \mathcal{N}(0,1)$ gives exact z-tests and z-intervals.

F.2 Distribution of the Sample Variance: Chi-Squared

Theorem F.1. Let $x^{(1)}, \ldots, x^{(n)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$. Then:

$$\frac{(n-1)s^2}{\sigma^2} = \frac{\sum_{i=1}^n (x^{(i)} - \bar{x})^2}{\sigma^2} \sim \chi^2_{n-1}$$

and this is independent of $\bar{x}$.

Proof sketch. Write $\sum(x^{(i)}-\bar{x})^2/\sigma^2 = \sum(z^{(i)}-\bar{z})^2$ where $z^{(i)} = (x^{(i)}-\mu)/\sigma \overset{\text{iid}}{\sim} \mathcal{N}(0,1)$. The quadratic form $\sum(z^{(i)}-\bar{z})^2$ equals $\mathbf{z}^\top(I - \frac{1}{n}\mathbf{1}\mathbf{1}^\top)\mathbf{z}$, a projection of $\mathbf{z}$ onto the $(n-1)$-dimensional subspace orthogonal to $\mathbf{1}$. By the spectral theorem for Gaussian quadratic forms, this has a $\chi^2_{n-1}$ distribution. Independence from $\bar{x}$ (a function of $\mathbf{1}^\top\mathbf{z}$) follows from the orthogonality.

The $n-1$ degrees of freedom arise because estimating $\mu$ by $\bar{x}$ removes one degree of freedom from the $n$ deviations $x^{(i)}-\bar{x}$ (they sum to zero, so only $n-1$ are free).

F.3 Student's $t$ Distribution

Definition. If $Z \sim \mathcal{N}(0,1)$ and $V \sim \chi^2_\nu$ independently, then $T = Z/\sqrt{V/\nu} \sim t_\nu$.

Application to Gaussian mean. With $Z = (\bar{x}-\mu)/(\sigma/\sqrt{n}) \sim \mathcal{N}(0,1)$ and $V = (n-1)s^2/\sigma^2 \sim \chi^2_{n-1}$ (independent of $Z$):

$$T = \frac{\bar{x}-\mu}{s/\sqrt{n}} = \frac{Z}{\sqrt{V/(n-1)}} \sim t_{n-1}$$

This is exact for all $n \geq 2$ - no large-sample approximation needed. The $t$-distribution is symmetric, bell-shaped, heavier-tailed than the Gaussian, and converges to $\mathcal{N}(0,1)$ as $\nu \to \infty$.

Quantiles comparison:

The $t$-distribution with small df requires wider intervals to achieve the same coverage - reflecting the additional uncertainty from estimating $\sigma$.

F.4 The $F$ Distribution

Definition. If $U \sim \chi^2_m$ and $V \sim \chi^2_n$ independently, then $F = (U/m)/(V/n) \sim F_{m,n}$.

Application. Comparing variances: $s_1^2/s_2^2 \cdot (\sigma_2^2/\sigma_1^2) \sim F_{n_1-1, n_2-1}$. Testing equality of variances $H_0: \sigma_1^2 = \sigma_2^2$: $F_{\text{obs}} = s_1^2/s_2^2 \sim F_{n_1-1, n_2-1}$ under $H_0$.

The $F$-distribution also underlies ANOVA (testing equality of multiple means), regression significance tests, and model comparison - fully developed in Section03 Hypothesis Testing and Section06 Regression Analysis.

---

Appendix G: Numerical Example - Fitting a Gaussian Mixture Model via EM

This worked example demonstrates MLE for a latent variable model.

Setup. Suppose we observe $n = 200$ observations believed to come from a mixture of two Gaussians: $p(x;\boldsymbol{\theta}) = \pi_1 \mathcal{N}(x;\mu_1,\sigma_1^2) + (1-\pi_1)\mathcal{N}(x;\mu_2,\sigma_2^2)$.

The parameters are $\boldsymbol{\theta} = (\pi_1, \mu_1, \sigma_1^2, \mu_2, \sigma_2^2)$.

Direct MLE is hard. The log-likelihood is $\ell(\boldsymbol{\theta}) = \sum_{i=1}^n \log[\pi_1\mathcal{N}(x^{(i)};\mu_1,\sigma_1^2) + (1-\pi_1)\mathcal{N}(x^{(i)};\mu_2,\sigma_2^2)]$, which contains a log of a sum - non-convex, no closed form.

EM solution. Introduce latent indicators $Z^{(i)} \in \{1,2\}$ indicating which component generated $x^{(i)}$.

E-step: Compute responsibilities

$$r^{(i)}_1 = \frac{\pi_1\mathcal{N}(x^{(i)};\mu_1,\sigma_1^2)}{\pi_1\mathcal{N}(x^{(i)};\mu_1,\sigma_1^2) + (1-\pi_1)\mathcal{N}(x^{(i)};\mu_2,\sigma_2^2)}, \quad r^{(i)}_2 = 1 - r^{(i)}_1$$

M-step: Update parameters (weighted MLEs for each component):

$$\hat{\pi}_1 = \frac{\sum_i r^{(i)}_1}{n}, \quad \hat{\mu}_k = \frac{\sum_i r^{(i)}_k x^{(i)}}{\sum_i r^{(i)}_k}, \quad \hat{\sigma}_k^2 = \frac{\sum_i r^{(i)}_k (x^{(i)} - \hat{\mu}_k)^2}{\sum_i r^{(i)}_k}$$

Convergence. EM guarantees $\ell(\boldsymbol{\theta}^{(t+1)}) \geq \ell(\boldsymbol{\theta}^{(t)})$ at every iteration, converging to a local maximum. Multiple random initialisations are needed to find the global maximum.

For AI: The EM algorithm is the prototype for alternating optimisation in deep learning. The BERT pre-training objective (masked language modelling) alternates between predicting masked tokens (like the E-step computing responsibilities) and updating parameters (like the M-step). Expectation-Maximisation underlies the theory of the EM algorithm, which is used for fitting GMMs, HMMs, and topic models in NLP preprocessing pipelines.

---

Appendix H: Regularised MLE and the Bias-Variance Trade-Off in Practice

Ridge regression (L2-regularised linear regression) is the canonical example of regularised MLE.

Setup. For linear regression $y^{(i)} = \boldsymbol{\theta}^\top \mathbf{x}^{(i)} + \varepsilon^{(i)}$ with $\varepsilon^{(i)} \overset{\text{iid}}{\sim} \mathcal{N}(0,\sigma^2)$, the MLE is OLS: $\hat{\boldsymbol{\theta}}_{\text{OLS}} = (X^\top X)^{-1}X^\top\mathbf{y}$.

Ridge regression: Add L2 penalty:

$$\hat{\boldsymbol{\theta}}_{\text{ridge}}(\lambda) = \arg\min_{\boldsymbol{\theta}} \lVert \mathbf{y} - X\boldsymbol{\theta} \rVert_2^2 + \lambda\lVert\boldsymbol{\theta}\rVert_2^2 = (X^\top X + \lambda I)^{-1}X^\top\mathbf{y}$$

Bias: $\operatorname{Bias}(\hat{\boldsymbol{\theta}}_{\text{ridge}}) = -\lambda(X^\top X + \lambda I)^{-1}\boldsymbol{\theta}^* \neq \mathbf{0}$ - ridge is biased.

Variance: $\operatorname{Cov}(\hat{\boldsymbol{\theta}}_{\text{ridge}}) = \sigma^2(X^\top X + \lambda I)^{-1}X^\top X(X^\top X + \lambda I)^{-1} \preceq \sigma^2(X^\top X)^{-1} = \operatorname{Cov}(\hat{\boldsymbol{\theta}}_{\text{OLS}})$.

Ridge has smaller covariance than OLS - introducing bias reduces variance. The bias-variance trade-off: for well-chosen $\lambda$, MSE of ridge < MSE of OLS.

MAP interpretation. Ridge is MAP estimation with Gaussian prior $\boldsymbol{\theta} \sim \mathcal{N}(\mathbf{0}, \sigma^2/\lambda \cdot I)$. The prior pulls $\boldsymbol{\theta}$ toward zero, introducing bias toward zero. The full development of MAP (adding a prior to MLE) is in Section04 Bayesian Inference.

Optimal $\lambda$ selection. Cross-validation estimates the prediction MSE for each $\lambda$, selecting the one that minimises the bias-variance trade-off from a prediction perspective. Alternatively, analytical formula: the MSE-minimising $\lambda^* = k\sigma^2/\lVert\boldsymbol{\theta}^*\rVert_2^2$ (depends on unknown $\boldsymbol{\theta}^*$, so must be estimated).

Connection to weight decay in LLMs. AdamW (Loshchilov & Hutter, 2019) adds weight decay $\lambda\lVert\boldsymbol{\theta}\rVert_2^2$ to the Adam update. This is regularised MLE with a Gaussian prior on all parameters - the standard training procedure for all frontier language models. The weight decay coefficient $\lambda$ (typically 0.1) controls the strength of the prior, determining how much parameters are regularised toward zero.

---

Appendix I: Non-Parametric Estimation

When the parametric model is uncertain or clearly wrong, non-parametric estimation makes minimal distributional assumptions.

Empirical Distribution Function (EDF). Given sample $x^{(1)}, \ldots, x^{(n)}$:

$$\hat{F}_n(t) = \frac{1}{n}\sum_{i=1}^n \mathbb{1}[x^{(i)} \leq t]$$

The EDF is the non-parametric MLE of the CDF $F$. By the Glivenko-Cantelli theorem: $\sup_t |\hat{F}_n(t) - F(t)| \xrightarrow{a.s.} 0$ - the EDF converges uniformly to the true CDF.

The Dvoretzky-Kiefer-Wolfowitz (DKW) inequality gives a finite-sample bound:

$$P\!\left(\sup_t |\hat{F}_n(t) - F(t)| > \varepsilon\right) \leq 2e^{-2n\varepsilon^2}$$

This is the non-parametric Cramer-Rao analogue - it bounds estimation error for the CDF without any parametric assumption.

Kernel density estimation. Estimating the PDF $f$ non-parametrically:

$$\hat{f}_h(x) = \frac{1}{nh}\sum_{i=1}^n K\!\left(\frac{x - x^{(i)}}{h}\right)$$

where $K$ is a kernel (e.g., Gaussian) and $h$ is the bandwidth. The bias-variance trade-off applies: small $h$ -> low bias, high variance; large $h$ -> high bias, low variance. Optimal bandwidth: $h^* \propto n^{-1/5}$ (mean integrated squared error).

Bootstrap as non-parametric MLE. The bootstrap resamples from the EDF $\hat{F}_n$ - effectively replacing the unknown $F$ with its non-parametric MLE. Bootstrap CI validity follows from the closeness of $\hat{F}_n$ to $F$ (Glivenko-Cantelli) and the continuity of the statistic of interest.

---

---

Appendix J: Selected Proofs and Derivations

J.1 Asymptotic Variance of the Sample Median

For a distribution with PDF $f$ and CDF $F$, let $m$ denote the population median ($F(m) = 1/2$). The sample median $\tilde{x}_n$ satisfies:

$$\sqrt{n}(\tilde{x}_n - m) \xrightarrow{d} \mathcal{N}\!\left(0, \frac{1}{4f(m)^2}\right)$$

Proof sketch. The sample median is the MLE for the double-exponential (Laplace) distribution, and the asymptotic variance of the $p$-th quantile estimator $\hat{q}_p$ is $p(1-p)/[nf(q_p)^2]$. For $p = 1/2$: $(1/2)(1/2)/[n f(m)^2] = 1/(4nf(m)^2)$.

Comparison to mean (for Gaussian data). For $\mathcal{N}(\mu,\sigma^2)$: $f(\mu) = 1/(\sigma\sqrt{2\pi})$, so the median's asymptotic variance is $\pi\sigma^2/(2n)$ vs. $\sigma^2/n$ for the mean. The relative efficiency is $2/\pi \approx 0.637$ - the mean extracts $100/0.637 \approx 57\%$ more information from Gaussian data than the median. However, for heavy-tailed distributions (e.g., Cauchy), the mean has infinite variance while the median remains consistent - robustness matters.

J.2 The Neyman-Scott Paradox - Inconsistency of MLE

MLE is not always consistent. The Neyman-Scott example (1948) provides a famous cautionary case.

Setup. Observe $n$ pairs $(x_j^{(1)}, x_j^{(2)})$ for $j = 1, \ldots, n$, where $x_j^{(k)} \overset{\text{iid}}{\sim} \mathcal{N}(\mu_j, \sigma^2)$. The parameters are $(\mu_1, \ldots, \mu_n, \sigma^2)$ - $n+1$ parameters for $2n$ observations.

MLE. For each pair: $\hat{\mu}_j = (x_j^{(1)} + x_j^{(2)})/2$. For $\sigma^2$: $\hat{\sigma}^2_{\text{MLE}} = \frac{1}{2n}\sum_{j=1}^n \left[(x_j^{(1)} - \hat{\mu}_j)^2 + (x_j^{(2)} - \hat{\mu}_j)^2\right] = \frac{1}{2n}\sum_j \frac{(x_j^{(1)}-x_j^{(2)})^2}{2}$.

Inconsistency. $\mathbb{E}[\hat{\sigma}^2_{\text{MLE}}] = \sigma^2/2$ - the MLE permanently underestimates $\sigma^2$ by a factor of 2, regardless of $n$. The bias does not vanish because the number of parameters grows with $n$.

Root cause. The regularity conditions for MLE consistency require the number of parameters to be fixed as $n \to \infty$. When nuisance parameters grow with $n$ (an "incidental parameters" problem), MLE can fail. The fix: use a conditional or marginal likelihood that eliminates the nuisance parameters.

For AI. This paradox is relevant to neural network generalisation. A model with $p \gg n$ parameters (overparameterised regime) has more parameters than observations. Classical MLE theory breaks down - yet in practice, overparameterised networks often generalise well due to implicit regularisation (gradient descent inductive bias). This remains an active research area.

J.3 Locally Most Powerful Tests and the Neyman-Pearson Lemma

(Preview - full treatment in Section03 Hypothesis Testing)

The Neyman-Pearson lemma provides the most powerful test for simple hypotheses $H_0: \theta = \theta_0$ vs. $H_1: \theta = \theta_1$. The rejection region is:

$$\{x : L(\theta_1;x)/L(\theta_0;x) > c\} = \{\text{reject } H_0\}$$

where $c$ is chosen to control type-I error at level $\alpha$. The likelihood ratio $L(\theta_1;x)/L(\theta_0;x)$ is a sufficient statistic for the test - no other test can be more powerful at the same level. The connection to MLE: the most powerful test uses the likelihood, and the MLE is the point that maximises the likelihood.

For composite alternatives $H_1: \theta \in \Theta_1$, the likelihood ratio test statistic $\Lambda = 2[\ell(\hat{\theta}) - \ell(\hat{\theta}_{\text{restricted}})] \xrightarrow{d} \chi^2_k$ is the standard generalisation, and connects to the Wald and score tests via asymptotic equivalence.

---

Appendix K: Reference Tables

K.1 Common MLE Formulas

K.2 Fisher Information Summary

K.3 Confidence Interval Reference

---

End of Appendices

---

Appendix L: Worked Examples - End-to-End Estimation

L.1 Estimating the Parameters of a Sentiment Classifier

Problem. A BERT-based sentiment classifier is evaluated on a test set of $n = 500$ examples. It classifies 418 correctly ($\hat{a} = 0.836$). Report (a) the point estimate with standard error, (b) the 95% Wilson CI, (c) determine if this differs significantly from a baseline accuracy of 80%.

Step 1: Point estimate and SE.

$\hat{a} = 418/500 = 0.836$. Standard error: $\text{SE} = \sqrt{\hat{a}(1-\hat{a})/n} = \sqrt{0.836 \times 0.164/500} = \sqrt{0.000274} \approx 0.0166$.

Step 2: Wilson score CI.

With $z = z_{0.025} = 1.96$, $n = 500$, $\hat{a} = 0.836$:

$$\text{CI} = \left[\frac{0.836 + 1.96^2/(2 \cdot 500)}{1 + 1.96^2/500} \pm \frac{1.96\sqrt{0.836 \times 0.164/500 + 1.96^2/(4 \cdot 500^2)}}{1 + 1.96^2/500}\right]$$

Numerically: centre $\approx (0.836 + 0.00384)/1.00768 \approx 0.8364$; margin $\approx 0.0323$. CI $\approx [0.804, 0.869]$.

Step 3: Comparison to baseline 80%.

The null hypothesis $H_0: a^* = 0.80$ corresponds to checking if 0.80 is inside the CI. Since 0.80 is inside $[0.804, 0.869]$? No - 0.80 is just outside. Alternatively, $z$-test: $z = (0.836 - 0.80)/\text{SE} = 0.036/0.0166 = 2.17 > 1.96$. The improvement is statistically significant at $\alpha = 0.05$.

Statistical interpretation. The classifier significantly outperforms the 80% baseline, but uncertainty spans from 80.4% to 86.9% - real-world performance could be anywhere in this range, highlighting the importance of CIs even for significant results.

L.2 MLE for a Gaussian Mixture - Worked Iteration

Problem. Two clusters are observed: $\{1.1, 1.3, 0.9, 1.2\}$ and $\{5.8, 6.2, 5.5, 6.1\}$ (but we don't know the assignments). Fit a 2-component Gaussian mixture.

Initialisation. $\hat{\pi}_1 = 0.5$, $\hat{\mu}_1 = 2.0$, $\hat{\sigma}_1^2 = 1.0$, $\hat{\mu}_2 = 5.0$, $\hat{\sigma}_2^2 = 1.0$.

E-step (iteration 1). For $x = 1.1$:

$$r_1 = \frac{0.5 \cdot \mathcal{N}(1.1;2.0,1.0)}{0.5 \cdot \mathcal{N}(1.1;2.0,1.0) + 0.5 \cdot \mathcal{N}(1.1;5.0,1.0)} = \frac{0.5 \times 0.266}{0.5 \times 0.266 + 0.5 \times 0.000}\approx 0.9999$$

For $x = 5.8$: $r_1 \approx 0.0001$. The algorithm already strongly assigns each point to the correct cluster.

M-step (iteration 1). With $\sum r^{(i)}_1 \approx 4.0$ (the four small values):

$$\hat{\mu}_1 \approx (1.1+1.3+0.9+1.2)/4 = 1.125, \quad \hat{\mu}_2 \approx (5.8+6.2+5.5+6.1)/4 = 5.9$$

After 2-3 iterations, the algorithm converges to the obvious clusters. The key insight: EM separates the cluster assignment (E-step soft assignments $r_{ik}$) from parameter estimation (M-step weighted MLEs), alternating until convergence.

L.3 Bootstrap CI for Median Income - ML Context

Problem. A dataset of $n = 50$ yearly salaries (thousands) for ML engineers is collected. Sample statistics: $\tilde{x} = 145$, $s = 42$. Construct a 95% bootstrap CI for the median.

Why bootstrap? The Gaussian CI for the median requires knowing the PDF at the median - unknown without assuming a distribution. The salary distribution is likely right-skewed (some outliers earn much more), making parametric CIs unreliable.

Bootstrap procedure (pseudocode):

for b = 1 to B=2000:
    x_star = sample(x, n=50, replace=True)
    median_star[b] = median(x_star)

CI_95 = [percentile(median_star, 2.5), percentile(median_star, 97.5)]

Typical result: $\text{CI}_{95} \approx [132, 158]$ thousand. The CI is slightly asymmetric (the right tail is longer due to income skewness), which the bootstrap correctly captures and a symmetric Gaussian CI would miss.

For AI. This exact procedure is used to report confidence intervals for ML engineer compensation surveys, which inform hiring benchmarks and compensation strategy at AI companies. Statistically sound salary benchmarks require bootstrap CIs because income distributions are strongly non-Gaussian.

---

Appendix M: Glossary of Key Terms

---

Appendix N: Further Reading and References

Primary References

1. Casella, G. & Berger, R.L. (2002). Statistical Inference (2nd ed.). Duxbury. - The definitive graduate textbook on classical estimation theory; covers all topics in this section at full rigor.

2. Lehmann, E.L. & Casella, G. (1998). Theory of Point Estimation (2nd ed.). Springer. - Advanced treatment of UMVUE theory, Rao-Blackwell, and sufficiency.

3. Fisher, R.A. (1922). "On the Mathematical Foundations of Theoretical Statistics." Philosophical Transactions of the Royal Society A, 222, 309-368. - The foundational paper defining sufficiency, efficiency, consistency, and MLE.

4. Cramer, H. (1946). Mathematical Methods of Statistics. Princeton University Press. - Original proof of the CRB.

5. Rao, C.R. (1945). "Information and the Accuracy Attainable in the Estimation of Statistical Parameters." Bulletin of the Calcutta Mathematical Society, 37, 81-91. - Independent CRB proof and Rao-Blackwell theorem.

6. Efron, B. (1979). "Bootstrap Methods: Another Look at the Jackknife." Annals of Statistics, 7(1), 1-26. - Original bootstrap paper.

ML-Specific References

1. Amari, S. (1998). "Natural Gradient Works Efficiently in Learning." Neural Computation, 10(2), 251-276. - Foundation of natural gradient methods.

2. Martens, J. & Grosse, R. (2015). "Optimizing Neural Networks with Kronecker-factored Approximate Curvature." ICML. - K-FAC for tractable FIM approximation in deep networks.

3. Kirkpatrick, J. et al. (2017). "Overcoming Catastrophic Forgetting in Neural Networks." PNAS, 114(13), 3521-3526. - EWC using FIM for continual learning.

4. Hoffmann, J. et al. (2022). "Training Compute-Optimal Large Language Models." NeurIPS. - Chinchilla scaling laws via nonlinear MLE.

5. Hu, E. et al. (2022). "LoRA: Low-Rank Adaptation of Large Language Models." ICLR. - Low-rank MLE for efficient fine-tuning.

6. Guo, C. et al. (2017). "On Calibration of Modern Neural Networks." ICML. - Temperature scaling as MLE on calibration set.

7. Goodfellow, I., Bengio, Y. & Courville, A. (2016). Deep Learning. MIT Press. - Chapter 5 covers MLE in the context of deep learning at the appropriate depth for ML practitioners.

Advanced References

1. van der Vaart, A.W. (1998). Asymptotic Statistics. Cambridge. - Graduate-level asymptotic theory; proofs of asymptotic normality, delta method, semiparametric theory.

2. Wasserman, L. (2004). All of Statistics. Springer. - Excellent graduate-level survey connecting classical statistics to modern methods.

3. Bishop, C.M. (2006). Pattern Recognition and Machine Learning. Springer. - Chapters 1-3 cover MLE, MAP, and Bayesian estimation with ML motivation.

---

This section is part of the Math for LLMs curriculum. For corrections or contributions, see CONTRIBUTING.md.

---

Appendix O: Connections to Information Theory

The Fisher information matrix has a deep connection to Shannon information theory, previewing material from Chapter 9 - Information Theory.

O.1 Fisher Information as Local Curvature of KL Divergence

The KL divergence between distributions at nearby parameter values satisfies:

$$D_{\mathrm{KL}}(p(\cdot;\boldsymbol{\theta}) \| p(\cdot;\boldsymbol{\theta}+\mathrm{d}\boldsymbol{\theta})) = \frac{1}{2}\mathrm{d}\boldsymbol{\theta}^\top \mathcal{I}(\boldsymbol{\theta})\, \mathrm{d}\boldsymbol{\theta} + O(\|\mathrm{d}\boldsymbol{\theta}\|^3)$$

The Fisher information matrix is the Hessian of the KL divergence with respect to the second argument, evaluated at the point where both arguments agree. This identifies $\mathcal{I}$ as the local curvature of the divergence surface.

Consequence: Moving in the Fisher metric direction by $\mathrm{d}\boldsymbol{\theta}$ produces the maximal increase in KL divergence from the current distribution - this is why the natural gradient (moving $\mathcal{I}^{-1}\nabla\ell$ in parameter space) corresponds to moving in the direction of maximal KL divergence gain in distribution space.

O.2 Cramer-Rao and Channel Capacity

Shannon's channel capacity theorem and the Cramer-Rao bound are related through the van Trees inequality (a Bayesian version of the CRB for random $\theta$):

$$\operatorname{MSE}(\hat{\theta}) \geq \frac{1}{n\mathcal{I}(\theta) + \mathcal{I}_{\text{prior}}(\theta)}$$

where $\mathcal{I}_{\text{prior}}$ is the Fisher information of the prior. As the prior becomes uninformative ($\mathcal{I}_{\text{prior}} \to 0$), this recovers the standard CRB. The connection: information accumulates (Fisher information adds) across observations and from the prior, exactly as Shannon information adds in a noisy channel.

O.3 Sufficient Statistics and Data Compression

A sufficient statistic $T$ satisfies the data processing inequality: processing data through $T$ cannot increase Fisher information. In fact, for a sufficient statistic, $\mathcal{I}_T(\theta) = \mathcal{I}_{\mathbf{X}}(\theta)$ - no information is lost. This is the estimation-theory analogue of lossless compression: a sufficient statistic compresses the data without losing any information about $\theta$.

Minimal sufficient statistics provide the maximum compression while retaining all information - analogous to the minimum description length (MDL) principle in information theory.

---

Appendix P: Practice Problems

P.1 Identification Problems

For each of the following, state whether the model is identifiable. If not, identify the unidentifiable combination.

1. $X \sim \mathcal{N}(\mu_1 - \mu_2, 1)$ with parameters $(\mu_1, \mu_2) \in \mathbb{R}^2$.
2. $p(x|\theta) = \theta e^{-\theta x}$ for $x > 0$, with $\theta > 0$. (Identifiable?)
3. A two-layer ReLU network $f(x) = \text{ReLU}(w_1 x) + \text{ReLU}(w_2 x)$ with parameters $(w_1, w_2)$. Is the function class identifiable?

P.2 Computational Problems

1. MLE and invariance. For Poisson data with $n = 20$ and $\sum x_i = 48$: (a) find $\hat{\lambda}$; (b) find the MLE of $e^{-\lambda}$ (the probability of observing zero events); (c) find the MLE of $1/\lambda$ (mean inter-arrival time).

2. CRB for a transformed parameter. For $X \sim \mathcal{N}(\mu, 1)$ with $n$ observations, derive the CRB for estimating $g(\mu) = \mu^2$ using the biased-estimator form of the CRB. Show that the MLE $\hat{\mu}^2 = \bar{x}^2$ does not achieve this bound for finite $n$, but does so asymptotically.

3. Bootstrap vs. asymptotic. For $n = 20$ observations from $\mathcal{N}(\mu, 1)$ with $\bar{x} = 3.2$: (a) compute the exact 95% $t$-CI; (b) simulate the bootstrap CI with $B = 1000$; (c) compare coverage by repeating 1000 times and computing the empirical coverage of each CI type.

P.3 Conceptual Problems

1. Stein paradox. For $d = 1$, the sample mean $\bar{x}$ is the admissible MLE. For $d = 3$, the James-Stein estimator $\tilde{\boldsymbol{\mu}} = (1 - \frac{d-2}{n\lVert\bar{\mathbf{x}}\rVert_2^2})\bar{\mathbf{x}}$ dominates $\bar{\mathbf{x}}$ in MSE. Simulate this for $d = 3$, $n = 1$, true $\boldsymbol{\mu} = (2,2,2)$, $\sigma = 1$, and verify $\text{MSE}(\tilde{\boldsymbol{\mu}}) < \text{MSE}(\bar{\mathbf{x}})$.

2. Model misspecification. A logistic regression model is fitted to data from a probit model ($p(y=1|\mathbf{x}) = \Phi(\mathbf{w}^\top\mathbf{x})$). The MLE converges to what? Explain using the misspecified MLE theorem and the KL divergence interpretation.

3. FIM for temperature scaling. For a $K$-class classifier with logits $\mathbf{z}$ and temperature $T$, the softmax is $p_k = e^{z_k/T}/\sum_j e^{z_j/T}$. Derive the Fisher information $\mathcal{I}(T)$ for a single example and explain why Newton-Raphson temperature calibration converges in 1-2 steps.

---

Section Section02 Estimation Theory - Chapter 7 Statistics - Math for LLMs curriculum

Lines: ~2000 | Theory notebook: 50+ cells | Exercises: 8 graded problems