Machine Learning II · Prof. Dr. Nihat Ay

Lecture 1
Approximation and generalisation

From learning a function to predicting new data.

Introduction

Where we stand

Machine Learning I was about representing functions by neural networks and fitting them to data. Machine Learning II asks when a fitted function also predicts new data well.

  1. What did we do in the previous course “Machine Learning I”?
  2. Universal approximation of two-layer neural networks
  3. The setting of supervised learning
  4. Contents of “Machine Learning II”
Section 1 of 4
1

What did we do in the previous course “Machine Learning I”?

1 · What did we do in the previous course “Machine Learning I”?

A network represents a function

$$f_\theta(x)=\sum_{j=1}^{3} v_j\,\sigma\bigl(\langle w_j,x\rangle-\vartheta_j\bigr)-\vartheta_0,\qquad \sigma(r)=\frac{1}{1+e^{-r}}$$
edge colour: blue positive, red negative weight
edge thickness: absolute value of the weight
neuron shading: threshold, coloured alike
  • With fixed parameter values $\theta$, the collection of all weights and thresholds, the network computes a function ${f_\theta:\mathbb{R}^2\to\mathbb{R}}$.
  • Other parameter values give other functions: each click below chooses a new $\theta$, all weights and thresholds at once.
1 · What did we do in the previous course “Machine Learning I”?

The class of representable functions

$$d(f,f_{\theta^\ast}) = \min_{\theta}\, d(f,f_\theta)$$
  • $\mathcal{F}=\{f_\theta:\theta\in\Theta\}$ is the set of all functions that the network represents with varying parameter values $\theta$.
  • The ultimate goal of training is a function $f_{\theta^\ast}$ in $\mathcal{F}$ that approximates the target function $f$ optimally.

Drag $f_\theta$ on $\mathcal{F}$, then minimise $d(f,f_\theta)$ by gradient descent (GD) or stochastic gradient descent (SGD).

Section 2 of 4
2

Universal approximation of two-layer neural networks

2 · Universal approximation of two-layer neural networks

Boolean functions: finitely many neurons suffice

$x_1$$x_2$$f(x_1,x_2)$

Each output +1 gets its own hidden neuron.

Click an output to switch it between −1 and +1.

  • Neurons with two states, $-1$ and $+1$: a hidden neuron fires exactly for one input pattern with output $+1$.
  • The output neuron fires if one of the hidden neurons fires.
  • So every Boolean function of $n$ inputs is represented with at most $2^n$ hidden neurons.
2 · Universal approximation of two-layer neural networks

More hidden neurons, larger function classes

$$\mathcal{F}_l=\{\text{functions represented with } l \text{ hidden neurons}\},\qquad l=1,2,3,\dots$$
two-layer networks with $l=1,2,3,\dots$ hidden neurons and the sets of functions that they represent
  • A hidden neuron with output weight $v_j=0$ does not contribute. So every function with $l$ hidden neurons can also be represented with $l+1$ hidden neurons, and the classes are nested: ${\mathcal{F}_1\subseteq\mathcal{F}_2\subseteq\mathcal{F}_3\subseteq\cdots}$.
2 · Universal approximation of two-layer neural networks

Continuous functions: more and more neurons

$$\mathcal{F}_1\subseteq\mathcal{F}_2\subseteq\mathcal{F}_3\subseteq\cdots,\qquad \mathcal{F}_l=\{\text{functions represented with } l \text{ hidden neurons}\}$$
  • Each hidden neuron contributes one smoothed step; with more neurons, the steps follow the target more closely.
  • Cybenko: with a sigmoid activation function, the closure of ${\bigcup_{l}\mathcal{F}_l}$ in the supremum norm is $C({[0,1]}^n)$, all continuous functions on ${[0,1]}^n$.
Section 3 of 4
3

The setting of supervised learning

3 · The setting of supervised learning

Generator, supervisor, learning machine

  • Generator (G): data points $x_1,x_2,\dots$ in an input space $D$, drawn independently according to the same distribution $\mu$ (i.i.d.).
  • Supervisor (S): assigns the output ${y_i=f(x_i)}$ with a function ${f:D\to\mathbb{R}}$.
  • Learning machine (LM): produces outputs ${f_\theta(x)}$ with functions from a class ${\mathcal{F}=\{f_\theta:\theta\in\Theta\}}$.
3 · The setting of supervised learning

Learning from a training set

At time $m$, the learner knows neither $\mu$ nor $f$, only the sample sequence, the training set

$$\mathbf{z}=(z_1,\dots,z_m),\qquad z_1=(x_1,y_1),\dots,z_m=(x_m,y_m).$$
  • From $\mathbf{z}$, the learner has to find a parameter value $\theta_{\mathbf{z}}$ such that $f_{\theta_{\mathbf{z}}}$ predicts the supervisor on future inputs: ideally, ${f_{\theta_{\mathbf{z}}}(x)\approx f(x)}$.
  • The assignment ${\mathbf{z}\mapsto\theta_{\mathbf{z}}}$ is called a learning algorithm.
  • How can the learner do this? We first take the view of an external observer who knows $\mu$ and $f$.
3 · The setting of supervised learning

Prediction error and approximation error

The external observer measures the distance of functions by the $L^2$-distance with respect to $\mu$,

$$d(f,g)={\Bigl(\int_D{\bigl(f(x)-g(x)\bigr)}^2\,\mu(dx)\Bigr)}^{1/2},$$

and the prediction error of the learning system is the squared distance to the target:

$$\mathcal{L}(\theta):=d^2(f,f_\theta)=\int_D{\bigl(\underbrace{f(x)}_{\mathllap{\text{supervisor}\kern-1.3em}}-\underbrace{f_\theta(x)}_{\mathclap{\text{predictor}}}\bigr)}^2\,\underbrace{\mu(dx)}_{\mathrlap{\kern-1.3em\text{data distribution}}}.$$

Minimising the prediction error gives an optimal parameter value $\theta^\ast$ (assuming that the minimum is attained), and the minimum is the approximation error:

$$\mathcal{L}(\theta^\ast)=\min_{\theta\in\Theta}\mathcal{L}(\theta)=d^2(f,f_{\theta^\ast}).$$
  • No learning algorithm achieves a prediction error below the approximation error: ${\mathcal{L}(\theta_{\mathbf{z}})\geq\mathcal{L}(\theta^\ast)}$ for every sample sequence $\mathbf{z}$.
  • But the learning system can neither evaluate nor minimise $\mathcal{L}$: it knows neither $\mu$ nor $f$.
3 · The setting of supervised learning

Empirical error and sample error

The learner replaces $\mathcal{L}$ by the empirical prediction error on the samples and minimises it:

$$\mathcal{L}_{\mathbf{z}}(\theta):=\frac1m\sum_{i=1}^m{\bigl(\underbrace{f(x_i)}_{y_i}-f_\theta(x_i)\bigr)}^2,\qquad \mathcal{L}_{\mathbf{z}}(\theta_{\mathbf{z}})=\min_{\theta\in\Theta}\mathcal{L}_{\mathbf{z}}(\theta).$$
  • This is empirical risk minimisation. For fixed $\theta$, ${\mathcal{L}_{\mathbf{z}}(\theta)\approx\mathcal{L}(\theta)}$ for large $m$ (law of large numbers). But $f_{\theta_{\mathbf{z}}}$ is adapted to the samples, so its empirical error can be much smaller than its prediction error, see overfitting. Whether $f_{\theta_{\mathbf{z}}}$ generalises is the central question.
  • The quality of $\theta_{\mathbf{z}}$ is measured by the sample error (estimation error) ${\mathcal{L}(\theta_{\mathbf{z}})-\mathcal{L}(\theta^\ast)}$.
  • The errors are the squares of the distances. For a linear class $\mathcal{F}$, $f_{\theta^\ast}$ is the orthogonal projection of $f$ (right angle), and Pythagoras gives the sample error ${d^2(f_{\theta^\ast},f_{\theta_{\mathbf{z}}})}$.
$$\text{prediction error}=\text{approximation error}+\text{sample error}$$
3 · The setting of supervised learning

An example: fitting a line

  • Samples ${z_i=(x_i,y_i)}$ with ${y_i=f(x_i)}$; the sliders move the line $f_\theta$, red: the residuals.
  • The learner chooses $\theta_{\mathbf{z}}$ with ${\mathcal{L}_{\mathbf{z}}(\theta_{\mathbf{z}})=\min_{\theta}\mathcal{L}_{\mathbf{z}}(\theta)}$, where ${\mathcal{L}_{\mathbf{z}}(\theta)=\frac17\sum_{i=1}^{7}{\bigl(y_i-f_\theta(x_i)\bigr)}^2}$.
  • Dashed: the minimiser $f_{\theta^\ast}$ of the prediction error $\mathcal{L}$, with $\mu$ uniform on ${[0,1]}$.
3 · The setting of supervised learning

Underfitting and overfitting

  • $\mathcal{F}$: polynomials of the chosen degree; $\mu$ uniform on ${[0,1]}$.
  • Class not complex enough: the approximation is bad, underfitting.
  • Class too complex: the generalisation is bad, overfitting.

Good generalisation needs complexity control, the main subject of “Machine Learning II”.

Section 4 of 4
4

Contents of “Machine Learning II”

4 · Contents of “Machine Learning II”

Complexity control

  1. Statistical learning theory of Vapnik and Chervonenkis (Lectures 2–8): when does empirical risk minimisation generalise? For a hierarchy ${\mathcal{F}^{(1)}\subseteq\mathcal{F}^{(2)}\subseteq\mathcal{F}^{(3)}\subseteq\cdots}$, structural risk minimisation (Lecture 6) chooses the class with the best trade-off between fit and complexity: the minimal empirical prediction error within the class plus a penalty growing with its VC dimension.
    • With growing class index $k$, the empirical error decreases and the complexity term grows.
    • The chosen class (dashed) minimises their sum, a bound on the prediction error (thick curve).
4 · Contents of “Machine Learning II”

Complexity control, continued

  1. Support vector machines, a prominent application of VC theory (Lecture 7): they separate two classes of data points by a hyperplane with a maximal margin, which controls the complexity of the class of separating functions.
    circled: the support vectors on the margin
  2. Learning from samples as an inverse problem (Lectures 9–12): finding $g$ with $g(x_i)=y_i$ is typically ill-posed. Regularisation penalises complexity, for instance Tikhonov regularisation,
    $$\frac1m\sum_{i=1}^m{\bigl(g(x_i)-y_i\bigr)}^2+\lambda\,{\|g\|}_K^2\ \to\ \min_{g\in\mathcal{H}_K},$$
    where $\lambda>0$ balances the fit to the data against the complexity of $g$, measured by the squared norm ${\|g\|}_K^2$ of the reproducing kernel Hilbert space $\mathcal{H}_K$.
4 · Contents of “Machine Learning II”

Validation and deep learning

  1. Practical methods such as cross-validation: the samples are divided into a training set and a validation set, and the class of functions is chosen so that training with this class performs best on the validation set; cross-validation repeats this for several divisions and averages. A separate test set then estimates the prediction error.
  2. New developments on generalisation within the field of deep learning: deep networks often have more parameters than samples, fit the training data exactly, and still generalise well (double descent, implicit regularisation, neural tangent kernel).
Lecture 1

Summary

  • A neural network with parameter values $\theta$ represents a function $f_\theta$; all of them form the class $\mathcal{F}$.
  • Two-layer networks are universal: finitely many neurons for Boolean functions, infinitely many in the limit for continuous ones.
  • Supervised learning: a generator draws ${x\sim\mu}$, a supervisor gives ${y=f(x)}$, and the learner chooses $\theta_{\mathbf{z}}$ from the samples $\mathbf{z}$.
  • Prediction error = approximation error + sample error; the learner can only minimise the empirical error $\mathcal{L}_{\mathbf{z}}$.
  • Too simple a class underfits, too complex a class overfits: generalisation needs complexity control.

Next: random outputs and the Bayes error (Lecture 2).