From the squared error to general losses and their risks.
Introduction
Where we stand
In Lecture 2, the prediction error was decomposed into the Bayes error, the approximation error and the estimation error. We now replace the squared error by general losses, define the corresponding risk terms in general, and discuss three examples that accompany us through the course.
Risk functions
1.1 Loss functions and risk 1.2 Examples
Section 1 of 1
1
Risk functions
1.1 Loss functions and risk1.2 Examples
1.1 · Loss functions and risk
General loss functions
In Lecture 2, the prediction error is the expected squared error. Replacing the squared error by other losses, with outcomes $z=(x,y)$, leads to a general notion:
Definition 1.1. Let $(\mathsf{Z},\mathcal{Z})$ be a measurable space with a probability measure $\nu$. A loss function is a function $\ell:\mathsf{Z}\times\Theta\to\mathbb{R}$,$(z,\theta)\mapsto\ell(z;\theta)$, such that $z\mapsto\ell(z;\theta)$ is measurable and $\nu$-integrable for every $\theta\in\Theta$. Its expectation value, the expected loss, is called a risk:
The outcome $z$ need not be an input–output pair (density estimation, Example 1.8).
The loss may depend on $\theta$ directly, not only through a prediction $f_\theta(x)$.
1.1 · Loss functions and risk
Bayes risk
In Lecture 2, the prediction error of $f_\theta$ was compared with the Bayes error, the smallest prediction error of all functions in $L^2(\mu)$ (Definition 2.1 of Lecture 2). For a general loss:
Definition 1.2. Let $\mathcal{G}$ be a class of hypotheses $h$, such as functions or densities, with a loss $\ell(z;h)$ that is measurable and $\nu$-integrable in $z$ for every $h\in\mathcal{G}$, and let the model be a map $\theta\mapsto h_\theta\in\mathcal{G}$ with $\ell(z;\theta)=\ell(z;h_\theta)$. With the risk $\mathcal{L}(h):=\int_{\mathsf{Z}}\ell(z;h)\,\nu(\mathrm{d}z)$, the Bayes risk of $\mathcal{G}$ is
The excess risk is defined for every $\theta$; for the learned $\theta_{\mathbf{z}}$ of a learning algorithm, it is the estimation or sample risk.
The approximation risk measures how well the model approximates the best hypothesis in $\mathcal{G}$.
If $\ell$ depends on $\theta$ directly, as a regularised loss ${\bigl(y-f_\theta(x)\bigr)}^2+\lambda\|\theta\|^2$, there is no Bayes risk; only the excess risk remains.
1.2 · Examples
Examples of risk functions
Three examples accompany us through the course:
Regression (Example 1.3). In supervised learning, $\mathsf{Z}=D\times\mathsf{Y}$, the hypotheses are measurable predictors $g$,$h_\theta=f_\theta$, and $\ell(z;g)=\ell\bigl(y,g(x)\bigr)$ with a loss $\ell(y,y')$ of the prediction $y'$ of $y$,$\ell(y,y)=0$. Regression: $\mathsf{Y}=\mathbb{R}$,
$$\ell(x,y;\theta)=\bigl(y-f_\theta(x)\bigr)^2;$$
the risk is the prediction error, the Bayes risk of $L^2(\mu)$ is the Bayes error of Lecture 2, and the risk terms are the errors of Theorem 3.1 of Lecture 2. Our standard example.
Pattern recognition:$\mathsf{Y}=\{\pm1\}$, the classification error → Examples 1.4, 1.6
Density estimation: the cross entropy → Example 1.8
1.2 · Examples
Example 1.4: pattern recognition
A class ${C\subseteq D}$ of inputs, say images of dogs, and $\mathsf{Z}=D\times\{\pm1\}$:
In Example 1.4, the risk of an arbitrary classifier $f_C$ with measurable $C\subseteq D$ (other values $g(x)$ always err), where $K_\pm(x):=K(x,\{\pm1\})$:
Theorem 1.5. The Bayes risk $\mathcal{L}^\ast=\inf_C\mathcal{L}(f_C)$ is attained by the classifier $f_{C^\ast}$ with $C^\ast:=\{x:K_+(x)\geq K_-(x)\}$:
Proof. $x$ contributes $K_+(x)$ if $x\notin C$ and $K_-(x)$ if $x\in C$; $C^\ast$ picks the smaller.□
The Bayes classifier $f_{C^\ast}$ predicts the more likely label; it is unique up to the ties $K_+(x)=K_-(x)$.
$\mathcal{L}^\ast>0$ unless the labels are determined by the inputs almost surely.
1.2 · Examples
Example 1.6: threshold classifiers
$D=[0,1]$, $\mu$ uniform, $C_t:=[t,1]$,$f_t:=f_{C_t}$,$\mathcal{L}(t):=\mathcal{L}(f_t)\geq\mathcal{L}^\ast$. For strictly increasing $K_+$,$C^\ast=[t^\ast,1]$ with $K_+(t^\ast)=\frac12$.
Top: the label probabilities $K_\pm(x)$. They cross at $t^\ast$, not at $\tfrac12$.
Bottom: the probability of a wrong label. Its area $\mathcal{L}(t)$ is the Bayes risk (red) plus the excess risk (blue).
or drag $t$ along the axis
1.2 · Examples
Entropy and KL divergence
Definition 1.7. Let $\nu$ and $\nu'$ be probability measures on $(\mathsf{Z},\mathcal{Z})$ with positive densities $p$ and $p'$ with respect to a reference measure $\lambda$. Whenever the integrals exist,
the entropy of $\nu$ is $H(\nu):=-\int_{\mathsf{Z}} p\ln p\,\mathrm{d}\lambda$,
the cross entropy of $\nu$ and $\nu'$ is $H(\nu,\nu'):=-\int_{\mathsf{Z}} p\ln p'\,\mathrm{d}\lambda$,
the KL divergence of $\nu'$ from $\nu$ is $D_{\mathrm{KL}}(\nu\,\|\,\nu'):=\int_{\mathsf{Z}} p\ln\frac{p}{p'}\,\mathrm{d}\lambda$.
For finite $H(\nu)$:$D_{\mathrm{KL}}(\nu\,\|\,\nu')=H(\nu,\nu')-H(\nu)$.
$D_{\mathrm{KL}}(\nu\,\|\,\nu')\geq0$, with equality only for $\nu'=\nu$.
Not a metric: neither symmetric nor satisfying the triangle inequality.
1.2 · Examples
Example 1.8: density estimation
Data and model distributions with positive densities with respect to $\lambda$:
The loss is the surprise at $z$ under $p(\cdot\,;\theta)$; the risk is the mean surprise or cross entropy.
Empirical risk minimisation is maximum likelihood.
Model: $\nu_\theta=\mathcal{N}(a,s^2)$.
1.2 · Examples
Example 1.8: risk terms
$D_{\mathrm{KL}}\geq0$, with equality only for $\nu_\theta=\nu$: the entropy $H(\nu)$ is the least cross entropy, the Bayes risk for the class of positive densities:
The model elements $h_\theta=f_\theta$,$f_{C_\theta}$,$p(\cdot\,;\theta)$ lie in $\mathcal{G}$.
In all three, the excess risk $\mathcal{L}(\theta)-\inf_\Theta\mathcal{L}$ completes the decomposition $\mathcal{L}(\theta)=\mathcal{L}^\ast+(\inf_\Theta\mathcal{L}-\mathcal{L}^\ast)+(\mathcal{L}(\theta)-\inf_\Theta\mathcal{L})$.
Lecture 3
Summary
A loss ${\ell(z;\theta)}$ defines a risk, the expected loss (Definition 1.1); the data distribution $\nu$ is unknown to the learner.
The Bayes risk of a class $\mathcal{G}$ (Definition 1.2) gives Bayes + approximation + excess risk; for $\theta_{\mathbf{z}}$, the excess risk is the estimation or sample risk.
Regression: for the squared error, the risk terms are the errors of Lecture 2.
Pattern recognition: the classification error; the Bayes classifier predicts the more likely label (Theorem 1.5).
Density estimation: the loss is a surprise, the risk the cross entropy = entropy + KL divergence; the Bayes risk is the entropy.
Next: learning algorithms, consistency and the empirical risk minimisation principle (Lecture 4).