Example 1.2. A measurable target $f$ gives $K^f$: the output is $f(x)$ with probability one.
Example 1.3. Gaussian noise, $y=f(x)+\varepsilon$: $\mathcal{N}\bigl(f(x),\sigma^2\bigr)$ is the normal distribution with mean $f(x)$ and variance $\sigma^2$. As $\sigma\to0$, it concentrates at $f(x)$ and approaches $K^f$.
Drag $x$ and the end points of $B$.
1.3 · The joint distribution of inputs and outputs
Proof. Expand $\bigl((y-f^\ast(x))+(f^\ast(x)-g(x))\bigr)^2$ and integrate (Lemma 1.4); the cross term vanishes, since $\int(y-f^\ast(x))\,K(x,\mathrm{d}y)=0$.□
Theorem 2.3. The regression function $f^\ast$ minimises the prediction error over $L^2(\mu)$, and it is the only minimiser up to $\mu$-null sets. The Bayes error is
Averaging out the fluctuations recovers the target function.
The Bayes error is the noise variance: no learning system does better than $\sigma^2$.
The Bayes error vanishes exactly for deterministic outputs: for $K^f$,$f^\ast=f$ and $\mathcal{L}^\ast=0$.
Section 3 of 3
3
Decomposition of the prediction error
3 · Decomposition of the prediction error
Projecting onto the function class
the output $y$ as an element of $L^2(\nu)$; right angles mark orthogonality
Let $\mathcal{F}=\{f_\theta:\theta\in\Theta\}$ be a closed linear subspace of $L^2(\mu)$.
By Lemma 2.2, $\mathcal{L}(\theta)=\mathcal{L}(f^\ast)+{\|f_\theta-f^\ast\|}^2_{L^2(\mu)}$.
$\theta^\ast$ minimises $\mathcal{L}$ exactly when $f_{\theta^\ast}$ is the orthogonal projection of $f^\ast$ onto $\mathcal{F}$.
The learner's $f_{\theta_{\mathbf{z}}}$ lies somewhere in $\mathcal{F}$.
Drag $f_{\theta_{\mathbf{z}}}$ along $\mathcal{F}$.
3 · Decomposition of the prediction error
Three error terms
Theorem 3.1. Let $\mathcal{F}$ be a closed linear subspace of $L^2(\mu)$ and $f_{\theta^\ast}$ the orthogonal projection of $f^\ast$ onto $\mathcal{F}$. Then, for every learned $f_{\theta_{\mathbf{z}}}\in\mathcal{F}$,
$$\mathcal{L}(\theta_{\mathbf{z}})=\underbrace{\mathcal{L}(f^\ast)}_{\text{Bayes error}}+\underbrace{{\bigl\|f^\ast-f_{\theta^\ast}\bigr\|}^2_{L^2(\mu)}}_{\text{approximation error}}+\underbrace{{\bigl\|f_{\theta^\ast}-f_{\theta_{\mathbf{z}}}\bigr\|}^2_{L^2(\mu)}}_{\text{estimation or sample error}}.$$
Proof. By Lemma 2.2, $\mathcal{L}(\theta_{\mathbf{z}})=\mathcal{L}(f^\ast)+{\|f^\ast-f_{\theta_{\mathbf{z}}}\|}^2_{L^2(\mu)}$. Since $f^\ast-f_{\theta^\ast}\perp\mathcal{F}$ and $f_{\theta^\ast}-f_{\theta_{\mathbf{z}}}\in\mathcal{F}$, Pythagoras splits the last term.□
Lecture 1's $\mathcal{L}(\theta^\ast)$ splits into the Bayes error (zero for $K=K^f$) and ${\|f^\ast-f_{\theta^\ast}\|}^2_{L^2(\mu)}$.
Within $\mathcal{F}$, minimising $\mathcal{L}$ means minimising the excess error, defined for every $\theta$:
No learning algorithm is needed for this; the sample error is the excess error of the learned $\theta_{\mathbf{z}}$.
3 · Decomposition of the prediction error
An example: noisy polynomial regression
Empirical risk minimisation (Lecture 1; in general, Lecture 4): $\theta_{\mathbf{z}}$ minimises ${\mathcal{L}_{\mathbf{z}}(\theta)=\frac{1}{m}\sum_{i=1}^m\bigl(y_i-f_\theta(x_i)\bigr)^2}$.
$\mathcal{F}$: polynomials of the chosen degree; $\mu$ uniform on $[0,1]$;$y_i$ drawn from $\mathcal{N}\bigl(f^\ast(x_i),\sigma^2\bigr)$, so the Bayes error is $\sigma^2$ (Example 2.4).
Minimising $\mathcal{L}_{\mathbf{z}}$ is a linear least-squares problem.
Lecture 2
Summary
A Markov kernel $K$ describes random outputs; with $\mu$ it defines the joint distribution $\nu$.
The regression function $f^\ast$ attains the Bayes error, the least prediction error over $L^2(\mu)$.