Machine Learning II · Prof. Dr. Nihat Ay

Lecture 2
Random outputs and the Bayes error

When outputs are random, some error remains for every predictor.

Lecture 2

Contents

With random outputs, the prediction error gains a third part, the Bayes error.

  1. Random outputs: Markov kernels and the joint distribution
    1.1 The prediction error and its decomposition
    1.2 Markov kernels
    1.3 The joint distribution of inputs and outputs
  2. The regression function and the Bayes error
    2.1 The prediction error for random outputs
    2.2 The regression function
  3. Decomposition of the prediction error
Section 1 of 3
1

Random outputs: Markov kernels and the joint distribution

1.1 The prediction error and its decomposition1.2 Markov kernels1.3 The joint distribution of inputs and outputs

1.1 · The prediction error and its decomposition

The decomposition so far

In Lecture 1, the prediction error of a parameter value $\theta$ was

$$\mathcal{L}(\theta)=\int_D\bigl(f(x)-f_\theta(x)\bigr)^2\,\mu(\mathrm{d}x),\qquad \mathcal{L}(\theta^\ast)=\min_{\theta\in\Theta}\mathcal{L}(\theta).$$

For a sample sequence ${\mathbf{z}=(z_1,\dots,z_m)}$, ${z_i=(x_i,y_i)}$, a learning algorithm yields $\theta_{\mathbf{z}}$:

$$\underbrace{\mathcal{L}(\theta_{\mathbf{z}})}_{\text{prediction error}}\;=\;\underbrace{\mathcal{L}(\theta^\ast)}_{\text{approximation error}}\;+\;\underbrace{\mathcal{L}(\theta_{\mathbf{z}})-\mathcal{L}(\theta^\ast)}_{\text{estimation or sample error}}$$
  • So far, outputs are deterministic, $y_i=f(x_i)$.
  • Noisy labels, measurement errors, incomplete inputs: the same $x$ comes with different $y$.
  • No function predicts perfectly: a third term, the Bayes error.
1.2 · Markov kernels

Random outputs

Definition 1.1. A Markov kernel from $D$ to $\mathbb{R}$ is a map $K:D\times\mathfrak{B}(\mathbb{R})\to[0,1]$ with:

  1. $K(x,\cdot)$ is a probability measure on $\mathbb{R}$ for every $x\in D$;
  2. $K(\cdot,B)$ is measurable for every Borel set $B\subseteq\mathbb{R}$.
  • The target function is replaced by the distribution $K(x,\cdot)$ of the output given the input $x$.
  • $K(x,B)$: the probability that the output lies in $B$.

Drag $x$ along the axis and the end points of $B$.

1.2 · Markov kernels

Examples 1.2, 1.3: deterministic outputs, Gaussian noise

$$K^f(x,B)=\delta_{f(x)}(B)=\begin{cases}1, & \text{if } f(x)\in B,\\ 0, & \text{otherwise,}\end{cases}\qquad\qquad \begin{aligned}K(x,\cdot)&=\mathcal{N}\bigl(f(x),\sigma^2\bigr),\\ K(x,\mathrm{d}y)&=\tfrac{1}{\sqrt{2\pi\sigma^2}}\,e^{-(y-f(x))^2/(2\sigma^2)}\,\mathrm{d}y\end{aligned}$$
  • Example 1.2. A measurable target $f$ gives $K^f$: the output is $f(x)$ with probability one.
  • Example 1.3. Gaussian noise, $y=f(x)+\varepsilon$: $\mathcal{N}\bigl(f(x),\sigma^2\bigr)$ is the normal distribution with mean $f(x)$ and variance $\sigma^2$. As $\sigma\to0$, it concentrates at $f(x)$ and approaches $K^f$.

Drag $x$ and the end points of $B$.

1.3 · The joint distribution of inputs and outputs

A joint distribution of inputs and outputs

$$\nu(A\times B):=\int_A K(x,B)\,\mu(\mathrm{d}x),\qquad A\subseteq D,\ B\subseteq\mathbb{R}\ \text{measurable}$$
  • $\mu$ and $K$ give a probability measure $\nu$ on the space $\mathsf{Z}:=D\times\mathbb{R}$ of input–output pairs.
  • $\nu$ is uniquely determined by this formula (Carathéodory, uniqueness theorem).
  • In the figure, $\mu$ is uniform on $[0,1]$.

Drag the corners of $A\times B$.

1.3 · The joint distribution of inputs and outputs

The prediction error revisited

Lemma 1.4. For every non-negative measurable function $g$ on $\mathsf{Z}$,

$$\int_\mathsf{Z} g(x,y)\,\nu(\mathrm{d}x,\mathrm{d}y)=\int_D\Bigl(\int_{\mathbb{R}}g(x,y)\,K(x,\mathrm{d}y)\Bigr)\,\mu(\mathrm{d}x).$$

Proof. For $g=1_{A\times B}$ this is the definition of $\nu$; then simple functions, monotone convergence.□

For the deterministic kernel $K^f$, with joint distribution $\nu^f$:

$$\begin{aligned}\mathcal{L}(\theta)&=\int_D\bigl(f(x)-f_\theta(x)\bigr)^2\,\mu(\mathrm{d}x)=\int_D\Bigl(\int_{\mathbb{R}}\bigl(y-f_\theta(x)\bigr)^2\,K^f(x,\mathrm{d}y)\Bigr)\,\mu(\mathrm{d}x)\\&=\int_\mathsf{Z}\bigl(y-f_\theta(x)\bigr)^2\,\nu^f(\mathrm{d}x,\mathrm{d}y)\end{aligned}$$
  • The prediction error refers only to the joint distribution $\nu^f$, not to $f$ itself.
  • This form makes sense for every $\nu$.
Section 2 of 3
2

The regression function and the Bayes error

2.1 The prediction error for random outputs2.2 The regression function

2.1 · The prediction error for random outputs

An error that cannot be removed

General $\nu$ from $\mu$ and a Markov kernel $K$: the output $y$ is drawn from $K(x,\cdot)$. For $g\in L^2(\mu)$, the prediction error is

$$\mathcal{L}(g):=\int_\mathsf{Z}\bigl(y-g(x)\bigr)^2\,\nu(\mathrm{d}x,\mathrm{d}y).$$
  • For $g=f_\theta$: $\mathcal{L}(\theta):=\mathcal{L}(f_\theta)$; for $\nu=\nu^f$ the prediction error of Lecture 1.
  • Assumption: finite second moment, $\int_\mathsf{Z} y^2\,\nu(\mathrm{d}x,\mathrm{d}y)<\infty$.
  • In general, the error cannot be reduced to zero, not even over all of $L^2(\mu)$.

Definition 2.1. The Bayes error is the smallest prediction error of all functions in $L^2(\mu)$:

$$\mathcal{L}^\ast:=\inf_{g\in L^2(\mu)}\mathcal{L}(g).$$
2.2 · The regression function

The regression function

$$f^\ast(x):=\int_{\mathbb{R}}y\,K(x,\mathrm{d}y),\qquad x\in D$$
  • $f^\ast(x)$ is the mean of $K(x,\cdot)$.
  • By Jensen's inequality, $f^\ast\in L^2(\mu)$.
  • The spread around $f^\ast$ remains as Bayes error.

Drag the inputs $x$ and $x'$ along the axis.

2.2 · The regression function

The regression function minimises the error

Lemma 2.2. For every $g\in L^2(\mu)$,

$$\mathcal{L}(g)=\mathcal{L}(f^\ast)+{\|g-f^\ast\|}^2_{L^2(\mu)}.$$

Proof. Expand $\bigl((y-f^\ast(x))+(f^\ast(x)-g(x))\bigr)^2$ and integrate (Lemma 1.4); the cross term vanishes, since $\int(y-f^\ast(x))\,K(x,\mathrm{d}y)=0$.□

Theorem 2.3. The regression function $f^\ast$ minimises the prediction error over $L^2(\mu)$, and it is the only minimiser up to $\mu$-null sets. The Bayes error is

$$\mathcal{L}^\ast=\mathcal{L}(f^\ast)=\int_D\Bigl(\int_{\mathbb{R}}\bigl(y-f^\ast(x)\bigr)^2\,K(x,\mathrm{d}y)\Bigr)\,\mu(\mathrm{d}x),$$

the mean, over the inputs, of the variance of $K(x,\cdot)$.

2.2 · The regression function

No function does better

$$\mathcal{L}(g)=\mathcal{L}(f^\ast)+{\|g-f^\ast\|}^2_{L^2(\mu)}\ \geq\ \mathcal{L}(f^\ast)\qquad\text{for all } g\in L^2(\mu)$$
  • For ${g=f^\ast+c\,h}$, the error grows by $c^2{\|h\|}^2_{L^2(\mu)}$.
  • The minimum, the Bayes error, is positive.
2.2 · The regression function

Example 2.4: Gaussian noise, continued

The kernel of Example 1.3, $K(x,\cdot)=\mathcal{N}\bigl(f(x),\sigma^2\bigr)$:

$$f^\ast=f,\qquad \mathcal{L}^\ast=\int_D\Bigl(\int_{\mathbb{R}}\bigl(y-f(x)\bigr)^2\,K(x,\mathrm{d}y)\Bigr)\,\mu(\mathrm{d}x)=\sigma^2.$$
  • Averaging out the fluctuations recovers the target function.
  • The Bayes error is the noise variance: no learning system does better than $\sigma^2$.
  • The Bayes error vanishes exactly for deterministic outputs: for $K^f$, $f^\ast=f$ and $\mathcal{L}^\ast=0$.
Section 3 of 3
3

Decomposition of the prediction error

3 · Decomposition of the prediction error

Projecting onto the function class

the output $y$ as an element of $L^2(\nu)$; right angles mark orthogonality
  • Let $\mathcal{F}=\{f_\theta:\theta\in\Theta\}$ be a closed linear subspace of $L^2(\mu)$.
  • By Lemma 2.2, $\mathcal{L}(\theta)=\mathcal{L}(f^\ast)+{\|f_\theta-f^\ast\|}^2_{L^2(\mu)}$.
  • $\theta^\ast$ minimises $\mathcal{L}$ exactly when $f_{\theta^\ast}$ is the orthogonal projection of $f^\ast$ onto $\mathcal{F}$.
  • The learner's $f_{\theta_{\mathbf{z}}}$ lies somewhere in $\mathcal{F}$.

Drag $f_{\theta_{\mathbf{z}}}$ along $\mathcal{F}$.

3 · Decomposition of the prediction error

Three error terms

Theorem 3.1. Let $\mathcal{F}$ be a closed linear subspace of $L^2(\mu)$ and $f_{\theta^\ast}$ the orthogonal projection of $f^\ast$ onto $\mathcal{F}$. Then, for every learned $f_{\theta_{\mathbf{z}}}\in\mathcal{F}$,

$$\mathcal{L}(\theta_{\mathbf{z}})=\underbrace{\mathcal{L}(f^\ast)}_{\text{Bayes error}}+\underbrace{{\bigl\|f^\ast-f_{\theta^\ast}\bigr\|}^2_{L^2(\mu)}}_{\text{approximation error}}+\underbrace{{\bigl\|f_{\theta^\ast}-f_{\theta_{\mathbf{z}}}\bigr\|}^2_{L^2(\mu)}}_{\text{estimation or sample error}}.$$

Proof. By Lemma 2.2, $\mathcal{L}(\theta_{\mathbf{z}})=\mathcal{L}(f^\ast)+{\|f^\ast-f_{\theta_{\mathbf{z}}}\|}^2_{L^2(\mu)}$. Since $f^\ast-f_{\theta^\ast}\perp\mathcal{F}$ and $f_{\theta^\ast}-f_{\theta_{\mathbf{z}}}\in\mathcal{F}$, Pythagoras splits the last term.□

  • Lecture 1's $\mathcal{L}(\theta^\ast)$ splits into the Bayes error (zero for $K=K^f$) and ${\|f^\ast-f_{\theta^\ast}\|}^2_{L^2(\mu)}$.
  • Within $\mathcal{F}$, minimising $\mathcal{L}$ means minimising the excess error, defined for every $\theta$:
    $$\mathcal{L}(\theta)-\min_{\theta'\in\Theta}\mathcal{L}(\theta')={\bigl\|f_{\theta^\ast}-f_\theta\bigr\|}^2_{L^2(\mu)}.$$
  • No learning algorithm is needed for this; the sample error is the excess error of the learned $\theta_{\mathbf{z}}$.
3 · Decomposition of the prediction error

An example: noisy polynomial regression

Empirical risk minimisation (Lecture 1; in general, Lecture 4): $\theta_{\mathbf{z}}$ minimises ${\mathcal{L}_{\mathbf{z}}(\theta)=\frac{1}{m}\sum_{i=1}^m\bigl(y_i-f_\theta(x_i)\bigr)^2}$.

  • $\mathcal{F}$: polynomials of the chosen degree; $\mu$ uniform on $[0,1]$; $y_i$ drawn from $\mathcal{N}\bigl(f^\ast(x_i),\sigma^2\bigr)$, so the Bayes error is $\sigma^2$ (Example 2.4).
  • Minimising $\mathcal{L}_{\mathbf{z}}$ is a linear least-squares problem.
Lecture 2

Summary

  • A Markov kernel $K$ describes random outputs; with $\mu$ it defines the joint distribution $\nu$.
  • The regression function $f^\ast$ attains the Bayes error, the least prediction error over $L^2(\mu)$.
  • Closed linear $\mathcal{F}$: prediction error = Bayes + approximation + estimation error (Theorem 3.1).

Next: general loss functions and risk functions (Lecture 3).