Machine Learning I was about representing functions by neural networks and fitting them to data. Machine Learning II asks when a fitted function also predicts new data well.
What did we do in the previous course “Machine Learning I”?
Universal approximation of two-layer neural networks
The setting of supervised learning
Contents of “Machine Learning II”
Section 1 of 4
1
What did we do in the previous course “Machine Learning I”?
1 · What did we do in the previous course “Machine Learning I”?
edge colour: blue positive, red negative weight edge thickness: absolute value of the weight neuron shading: threshold, coloured alike
With fixed parameter values $\theta$, the collection of all weights and thresholds, the network computes a function ${f_\theta:\mathbb{R}^2\to\mathbb{R}}$.
Other parameter values give other functions: each click below chooses a new $\theta$, all weights and thresholds at once.
1 · What did we do in the previous course “Machine Learning I”?
$\mathcal{F}=\{f_\theta:\theta\in\Theta\}$ is the set of all functions that the network represents with varying parameter values $\theta$.
The ultimate goal of training is a function $f_{\theta^\ast}$ in $\mathcal{F}$ that approximates the target function $f$ optimally.
Drag $f_\theta$ on $\mathcal{F}$, then minimise $d(f,f_\theta)$ by gradient descent (GD) or stochastic gradient descent (SGD).
Section 2 of 4
2
Universal approximation of two-layer neural networks
2 · Universal approximation of two-layer neural networks
Boolean functions: finitely many neurons suffice
$x_1$
$x_2$
$f(x_1,x_2)$
Each output +1 gets its own hidden neuron.
Click an output to switch it between −1 and +1.
Neurons with two states, $-1$ and $+1$: a hidden neuron fires exactly for one input pattern with output $+1$.
The output neuron fires if one of the hidden neurons fires.
So every Boolean function of $n$ inputs is represented with at most $2^n$ hidden neurons.
2 · Universal approximation of two-layer neural networks
More hidden neurons, larger function classes
$$\mathcal{F}_l=\{\text{functions represented with } l \text{ hidden neurons}\},\qquad l=1,2,3,\dots$$
two-layer networks with $l=1,2,3,\dots$ hidden neurons and the sets of functions that they represent
A hidden neuron with output weight $v_j=0$ does not contribute. So every function with $l$ hidden neurons can also be represented with $l+1$ hidden neurons, and the classes are nested: ${\mathcal{F}_1\subseteq\mathcal{F}_2\subseteq\mathcal{F}_3\subseteq\cdots}$.
2 · Universal approximation of two-layer neural networks
Continuous functions: more and more neurons
$$\mathcal{F}_1\subseteq\mathcal{F}_2\subseteq\mathcal{F}_3\subseteq\cdots,\qquad \mathcal{F}_l=\{\text{functions represented with } l \text{ hidden neurons}\}$$
Each hidden neuron contributes one smoothed step; with more neurons, the steps follow the target more closely.
Cybenko: with a sigmoid activation function, the closure of ${\bigcup_{l}\mathcal{F}_l}$ in the supremum norm is $C({[0,1]}^n)$, all continuous functions on ${[0,1]}^n$.
Section 3 of 4
3
The setting of supervised learning
3 · The setting of supervised learning
Generator, supervisor, learning machine
Generator (G): data points $x_1,x_2,\dots$ in an input space $D$, drawn independently according to the same distribution $\mu$ (i.i.d.).
Supervisor (S): assigns the output ${y_i=f(x_i)}$ with a function ${f:D\to\mathbb{R}}$.
Learning machine (LM): produces outputs ${f_\theta(x)}$ with functions from a class ${\mathcal{F}=\{f_\theta:\theta\in\Theta\}}$.
3 · The setting of supervised learning
Learning from a training set
At time $m$, the learner knows neither $\mu$ nor $f$, only the sample sequence, the training set
From $\mathbf{z}$, the learner has to find a parameter value $\theta_{\mathbf{z}}$ such that $f_{\theta_{\mathbf{z}}}$ predicts the supervisor on future inputs: ideally, ${f_{\theta_{\mathbf{z}}}(x)\approx f(x)}$.
The assignment ${\mathbf{z}\mapsto\theta_{\mathbf{z}}}$ is called a learning algorithm.
How can the learner do this? We first take the view of an external observer who knows $\mu$ and $f$.
3 · The setting of supervised learning
Prediction error and approximation error
The external observer measures the distance of functions by the $L^2$-distance with respect to $\mu$,
Minimising the prediction error gives an optimal parameter value $\theta^\ast$ (assuming that the minimum is attained), and the minimum is the approximation error:
No learning algorithm achieves a prediction error below the approximation error: ${\mathcal{L}(\theta_{\mathbf{z}})\geq\mathcal{L}(\theta^\ast)}$ for every sample sequence $\mathbf{z}$.
But the learning system can neither evaluate nor minimise $\mathcal{L}$: it knows neither $\mu$ nor $f$.
3 · The setting of supervised learning
Empirical error and sample error
The learner replaces $\mathcal{L}$ by the empirical prediction error on the samples and minimises it:
This is empirical risk minimisation. For fixed $\theta$, ${\mathcal{L}_{\mathbf{z}}(\theta)\approx\mathcal{L}(\theta)}$ for large $m$ (law of large numbers). But $f_{\theta_{\mathbf{z}}}$ is adapted to the samples, so its empirical error can be much smaller than its prediction error, see overfitting. Whether $f_{\theta_{\mathbf{z}}}$ generalises is the central question.
The quality of $\theta_{\mathbf{z}}$ is measured by the sample error (estimation error) ${\mathcal{L}(\theta_{\mathbf{z}})-\mathcal{L}(\theta^\ast)}$.
The errors are the squares of the distances. For a linear class $\mathcal{F}$, $f_{\theta^\ast}$ is the orthogonal projection of $f$ (right angle), and Pythagoras gives the sample error ${d^2(f_{\theta^\ast},f_{\theta_{\mathbf{z}}})}$.
Samples ${z_i=(x_i,y_i)}$ with ${y_i=f(x_i)}$; the sliders move the line $f_\theta$, red: the residuals.
The learner chooses $\theta_{\mathbf{z}}$ with ${\mathcal{L}_{\mathbf{z}}(\theta_{\mathbf{z}})=\min_{\theta}\mathcal{L}_{\mathbf{z}}(\theta)}$, where ${\mathcal{L}_{\mathbf{z}}(\theta)=\frac17\sum_{i=1}^{7}{\bigl(y_i-f_\theta(x_i)\bigr)}^2}$.
Dashed: the minimiser $f_{\theta^\ast}$ of the prediction error $\mathcal{L}$, with $\mu$ uniform on ${[0,1]}$.
3 · The setting of supervised learning
Underfitting and overfitting
$\mathcal{F}$: polynomials of the chosen degree; $\mu$ uniform on ${[0,1]}$.
Class not complex enough: the approximation is bad, underfitting.
Class too complex: the generalisation is bad, overfitting.
Good generalisation needs complexity control, the main subject of “Machine Learning II”.
Section 4 of 4
4
Contents of “Machine Learning II”
4 · Contents of “Machine Learning II”
Complexity control
Statistical learning theory of Vapnik and Chervonenkis (Lectures 2–8): when does empirical risk minimisation generalise? For a hierarchy ${\mathcal{F}^{(1)}\subseteq\mathcal{F}^{(2)}\subseteq\mathcal{F}^{(3)}\subseteq\cdots}$,structural risk minimisation (Lecture 6) chooses the class with the best trade-off between fit and complexity: the minimal empirical prediction error within the class plus a penalty growing with its VC dimension.
With growing class index $k$, the empirical error decreases and the complexity term grows.
The chosen class (dashed) minimises their sum, a bound on the prediction error (thick curve).
4 · Contents of “Machine Learning II”
Complexity control, continued
Support vector machines, a prominent application of VC theory (Lecture 7): they separate two classes of data points by a hyperplane with a maximal margin, which controls the complexity of the class of separating functions.
circled: the support vectors on the margin
Learning from samples as an inverse problem (Lectures 9–12): finding $g$ with $g(x_i)=y_i$ is typically ill-posed. Regularisation penalises complexity, for instance Tikhonov regularisation,
where $\lambda>0$ balances the fit to the data against the complexity of $g$, measured by the squared norm ${\|g\|}_K^2$ of the reproducing kernel Hilbert space $\mathcal{H}_K$.
4 · Contents of “Machine Learning II”
Validation and deep learning
Practical methods such as cross-validation: the samples are divided into a training set and a validation set, and the class of functions is chosen so that training with this class performs best on the validation set; cross-validation repeats this for several divisions and averages. A separate test set then estimates the prediction error.
New developments on generalisation within the field of deep learning: deep networks often have more parameters than samples, fit the training data exactly, and still generalise well (double descent, implicit regularisation, neural tangent kernel).
Lecture 1
Summary
A neural network with parameter values $\theta$ represents a function $f_\theta$; all of them form the class $\mathcal{F}$.
Two-layer networks are universal: finitely many neurons for Boolean functions, infinitely many in the limit for continuous ones.
Supervised learning: a generator draws ${x\sim\mu}$, a supervisor gives ${y=f(x)}$, and the learner chooses $\theta_{\mathbf{z}}$ from the samples $\mathbf{z}$.
Prediction error = approximation error + sample error; the learner can only minimise the empirical error $\mathcal{L}_{\mathbf{z}}$.
Too simple a class underfits, too complex a class overfits: generalisation needs complexity control.
Next: random outputs and the Bayes error (Lecture 2).