Machine Learning II · Prof. Dr. Nihat Ay

Lecture 3
Risk functions

From the squared error to general losses and their risks.

Introduction

Where we stand

In Lecture 2, the prediction error was decomposed into the Bayes error, the approximation error and the estimation error. We now replace the squared error by general losses, define the corresponding risk terms in general, and discuss three examples that accompany us through the course.

  1. Risk functions
    1.1 Loss functions and risk
    1.2 Examples
Section 1 of 1
1

Risk functions

1.1 Loss functions and risk1.2 Examples

1.1 · Loss functions and risk

General loss functions

In Lecture 2, the prediction error is the expected squared error. Replacing the squared error by other losses, with outcomes $z=(x,y)$, leads to a general notion:

Definition 1.1. Let $(\mathsf{Z},\mathcal{Z})$ be a measurable space with a probability measure $\nu$. A loss function is a function $\ell:\mathsf{Z}\times\Theta\to\mathbb{R}$, $(z,\theta)\mapsto\ell(z;\theta)$, such that $z\mapsto\ell(z;\theta)$ is measurable and $\nu$-integrable for every $\theta\in\Theta$. Its expectation value, the expected loss, is called a risk:

$$\mathcal{L}(\theta):=\int_{\mathsf{Z}}\ell(z;\theta)\,\nu(\mathrm{d}z).$$
  • The outcome $z$ need not be an input–output pair (density estimation, Example 1.8).
  • The loss may depend on $\theta$ directly, not only through a prediction $f_\theta(x)$.
1.1 · Loss functions and risk

Bayes risk

In Lecture 2, the prediction error of $f_\theta$ was compared with the Bayes error, the smallest prediction error of all functions in $L^2(\mu)$ (Definition 2.1 of Lecture 2). For a general loss:

Definition 1.2. Let $\mathcal{G}$ be a class of hypotheses $h$, such as functions or densities, with a loss $\ell(z;h)$ that is measurable and $\nu$-integrable in $z$ for every $h\in\mathcal{G}$, and let the model be a map $\theta\mapsto h_\theta\in\mathcal{G}$ with $\ell(z;\theta)=\ell(z;h_\theta)$. With the risk $\mathcal{L}(h):=\int_{\mathsf{Z}}\ell(z;h)\,\nu(\mathrm{d}z)$, the Bayes risk of $\mathcal{G}$ is

$$\mathcal{L}^\ast:=\inf_{h\in\mathcal{G}}\mathcal{L}(h).$$
  • $\mathcal{L}^\ast$ depends on $\mathcal{G}$: the larger $\mathcal{G}$, the smaller $\mathcal{L}^\ast$.
1.1 · Loss functions and risk

Decomposition of the risk

Adding and subtracting $\mathcal{L}^\ast$ and $\inf_\Theta\mathcal{L}$ gives, for every $\theta\in\Theta$,

$$\mathcal{L}(\theta)\;=\;\underbrace{\mathcal{L}^\ast}_{\text{Bayes risk}}\;+\;\underbrace{\inf_\Theta\mathcal{L}-\mathcal{L}^\ast}_{\text{approximation risk}}\;+\;\underbrace{\mathcal{L}(\theta)-\inf_\Theta\mathcal{L}}_{\text{excess risk}}$$
  • The excess risk is defined for every $\theta$; for the learned $\theta_{\mathbf{z}}$ of a learning algorithm, it is the estimation or sample risk.
  • The approximation risk measures how well the model approximates the best hypothesis in $\mathcal{G}$.
  • If $\ell$ depends on $\theta$ directly, as a regularised loss ${\bigl(y-f_\theta(x)\bigr)}^2+\lambda\|\theta\|^2$, there is no Bayes risk; only the excess risk remains.
1.2 · Examples

Examples of risk functions

Three examples accompany us through the course:

  1. Regression (Example 1.3). In supervised learning, $\mathsf{Z}=D\times\mathsf{Y}$, the hypotheses are measurable predictors $g$, $h_\theta=f_\theta$, and $\ell(z;g)=\ell\bigl(y,g(x)\bigr)$ with a loss $\ell(y,y')$ of the prediction $y'$ of $y$, $\ell(y,y)=0$. Regression: $\mathsf{Y}=\mathbb{R}$,
    $$\ell(x,y;\theta)=\bigl(y-f_\theta(x)\bigr)^2;$$
    the risk is the prediction error, the Bayes risk of $L^2(\mu)$ is the Bayes error of Lecture 2, and the risk terms are the errors of Theorem 3.1 of Lecture 2. Our standard example.
  2. Pattern recognition: $\mathsf{Y}=\{\pm1\}$, the classification error → Examples 1.4, 1.6
  3. Density estimation: the cross entropy → Example 1.8
1.2 · Examples

Example 1.4: pattern recognition

A class ${C\subseteq D}$ of inputs, say images of dogs, and $\mathsf{Z}=D\times\{\pm1\}$:

$$f_C(x)=\begin{cases}+1, & \text{if } x\in C,\\ -1, & \text{if } x\notin C,\end{cases}\qquad\qquad \ell(x,y;\theta)=\begin{cases}0, & \text{if } y=f_\theta(x),\\ 1, & \text{if } y\neq f_\theta(x).\end{cases}$$
  • Target: the indicator function $f_C$; learning system: ${f_\theta=f_{C_\theta}}$, with sets ${C_\theta\subseteq D}$.
  • The risk is the classification error, the probability of a wrong label:
$$\mathcal{L}(\theta)=\int_{\mathsf{Z}}\ell(x,y;\theta)\,\nu(\mathrm{d}x,\mathrm{d}y)=\nu\bigl\{(x,y)\in \mathsf{Z}:y\neq f_\theta(x)\bigr\}.$$
1.2 · Examples

The Bayes classifier

In Example 1.4, the risk of an arbitrary classifier $f_C$ with measurable $C\subseteq D$ (other values $g(x)$ always err), where $K_\pm(x):=K(x,\{\pm1\})$:

$$\textstyle\mathcal{L}(f_C)=\int_{D\setminus C}K_+(x)\,\mu(\mathrm{d}x)+\int_C K_-(x)\,\mu(\mathrm{d}x).$$

Theorem 1.5. The Bayes risk $\mathcal{L}^\ast=\inf_C\mathcal{L}(f_C)$ is attained by the classifier $f_{C^\ast}$ with $C^\ast:=\{x:K_+(x)\geq K_-(x)\}$:

$$\textstyle\mathcal{L}^\ast=\int_D\min\bigl\{K_+(x),K_-(x)\bigr\}\,\mu(\mathrm{d}x),\qquad f_{C^\ast}(x)=\begin{cases}+1, & \text{if } K_+(x)\geq K_-(x),\\ -1, & \text{if } K_+(x)<K_-(x).\end{cases}$$

Proof. $x$ contributes $K_+(x)$ if $x\notin C$ and $K_-(x)$ if $x\in C$; $C^\ast$ picks the smaller.□

  • The Bayes classifier $f_{C^\ast}$ predicts the more likely label; it is unique up to the ties $K_+(x)=K_-(x)$.
  • $\mathcal{L}^\ast>0$ unless the labels are determined by the inputs almost surely.
1.2 · Examples

Example 1.6: threshold classifiers

  • $D=[0,1]$, $\mu$ uniform, $C_t:=[t,1]$, $f_t:=f_{C_t}$, $\mathcal{L}(t):=\mathcal{L}(f_t)\geq\mathcal{L}^\ast$. For strictly increasing $K_+$, $C^\ast=[t^\ast,1]$ with $K_+(t^\ast)=\frac12$.
  • Top: the label probabilities $K_\pm(x)$. They cross at $t^\ast$, not at $\tfrac12$.
  • Bottom: the probability of a wrong label. Its area $\mathcal{L}(t)$ is the Bayes risk (red) plus the excess risk (blue).
or drag $t$ along the axis
1.2 · Examples

Entropy and KL divergence

Definition 1.7. Let $\nu$ and $\nu'$ be probability measures on $(\mathsf{Z},\mathcal{Z})$ with positive densities $p$ and $p'$ with respect to a reference measure $\lambda$. Whenever the integrals exist,

  1. the entropy of $\nu$ is $H(\nu):=-\int_{\mathsf{Z}} p\ln p\,\mathrm{d}\lambda$,
  2. the cross entropy of $\nu$ and $\nu'$ is $H(\nu,\nu'):=-\int_{\mathsf{Z}} p\ln p'\,\mathrm{d}\lambda$,
  3. the KL divergence of $\nu'$ from $\nu$ is $D_{\mathrm{KL}}(\nu\,\|\,\nu'):=\int_{\mathsf{Z}} p\ln\frac{p}{p'}\,\mathrm{d}\lambda$.
  • For finite $H(\nu)$: $D_{\mathrm{KL}}(\nu\,\|\,\nu')=H(\nu,\nu')-H(\nu)$.
  • $D_{\mathrm{KL}}(\nu\,\|\,\nu')\geq0$, with equality only for $\nu'=\nu$.
  • Not a metric: neither symmetric nor satisfying the triangle inequality.
1.2 · Examples

Example 1.8: density estimation

Data and model distributions with positive densities with respect to $\lambda$:

$$\nu(\mathrm{d}z)=p(z)\,\lambda(\mathrm{d}z),\qquad \nu_\theta(\mathrm{d}z)=p(z;\theta)\,\lambda(\mathrm{d}z),\quad \theta\in\Theta.$$

The deviation of $\nu_\theta$ from $\nu$ is measured by the KL divergence:

$$D_{\mathrm{KL}}(\nu\,\|\,\nu_\theta)=H(\nu,\nu_\theta)-H(\nu).$$
  • $H(\nu)$ does not depend on $\theta$: minimising $D_{\mathrm{KL}}$ means minimising the cross entropy.
  • This suggests a loss whose risk is the cross entropy.
1.2 · Examples

Example 1.8: loss as surprise

$$\ell(z;\theta)=\ln\frac{1}{p(z;\theta)},\qquad \mathcal{L}(\theta)=\int_{\mathsf{Z}}\ln\frac{1}{p(z;\theta)}\,\nu(\mathrm{d}z)=H(\nu,\nu_\theta)$$
  • The loss is the surprise at $z$ under $p(\cdot\,;\theta)$; the risk is the mean surprise or cross entropy.
  • Empirical risk minimisation is maximum likelihood.
  • Model: $\nu_\theta=\mathcal{N}(a,s^2)$.
1.2 · Examples

Example 1.8: risk terms

$D_{\mathrm{KL}}\geq0$, with equality only for $\nu_\theta=\nu$: the entropy $H(\nu)$ is the least cross entropy, the Bayes risk for the class of positive densities:

$$\mathcal{L}(\theta)\;=\;\underbrace{H(\nu)}_{\text{entropy}}\;+\;\underbrace{\inf_\Theta D_{\mathrm{KL}}(\nu\,\|\,\nu_\theta)}_{\text{approximation risk}}\;+\;\underbrace{\mathcal{L}(\theta)-\inf_\Theta\mathcal{L}}_{\text{excess risk}}$$
  • The approximation risk vanishes if $\nu$ belongs to the model.
  • For the maximum likelihood estimate, the excess risk is the estimation risk; the loss is a surprise, not an error.
  • This needs a finite $H(\nu)$, which, unlike the Bayes error, can be negative if $\lambda$ is not a counting measure.
1.2 · Examples

Risk terms in the three examples

exampleclass $\mathcal{G}$loss $\ell(z;h)$minimiser over $\mathcal{G}$Bayes risk $\mathcal{L}^\ast$approximation risk
regression
(Ex. 1.3)
$g\in L^2(\mu)$${\bigl(y-g(x)\bigr)}^2$regression function $f^\ast$Bayes error $\mathcal{L}(f^\ast)$$\inf_\Theta{\|f^\ast-f_\theta\|}^2_{L^2(\mu)}$
pattern recognition
(Ex. 1.4, 1.6)
classifiers $f_C$$1$ if $y\neq g(x)$, else $0$Bayes classifier $f_{C^\ast}$$\int_D\min\{K_+,K_-\}\,\mathrm{d}\mu$$\inf_\Theta\mathcal{L}-\mathcal{L}^\ast$
density estimation
(Ex. 1.8)
positive densities $q$$\ln\frac{1}{q(z)}$density $p$ of $\nu$entropy $H(\nu)$$\inf_\Theta D_{\mathrm{KL}}(\nu\,\|\,\nu_\theta)$
  • The model elements $h_\theta=f_\theta$, $f_{C_\theta}$, $p(\cdot\,;\theta)$ lie in $\mathcal{G}$.
  • In all three, the excess risk $\mathcal{L}(\theta)-\inf_\Theta\mathcal{L}$ completes the decomposition $\mathcal{L}(\theta)=\mathcal{L}^\ast+(\inf_\Theta\mathcal{L}-\mathcal{L}^\ast)+(\mathcal{L}(\theta)-\inf_\Theta\mathcal{L})$.
Lecture 3

Summary

  • A loss ${\ell(z;\theta)}$ defines a risk, the expected loss (Definition 1.1); the data distribution $\nu$ is unknown to the learner.
  • The Bayes risk of a class $\mathcal{G}$ (Definition 1.2) gives Bayes + approximation + excess risk; for $\theta_{\mathbf{z}}$, the excess risk is the estimation or sample risk.
  • Regression: for the squared error, the risk terms are the errors of Lecture 2.
  • Pattern recognition: the classification error; the Bayes classifier predicts the more likely label (Theorem 1.5).
  • Density estimation: the loss is a surprise, the risk the cross entropy = entropy + KL divergence; the Bayes risk is the entropy.

Next: learning algorithms, consistency and the empirical risk minimisation principle (Lecture 4).