What ML Brings To The Vision Party

Every modern computer vision system has ML inside it. Sometimes obviously (a CNN is a stack of learnable filters), sometimes subtly (a softmax head is a multinomial logistic regression; a feature normalisation step is implicit feature engineering). The ML toolbox is the universal language of the modern field — when you see "logistic regression" inside a detection head, it's the exact same logistic regression you'd write for a binary tabular classifier.

The course assumes you've seen most of this; this unit is the rapid 90-minute review.

The Three Task Families

Supervised — given $(X,y)$ pairs, learn $f:X\to y$. *Classification* (discrete $y$) and *regression* (continuous $y$) are the two main flavours.

Unsupervised — given only $X$, find structure. Clustering, density estimation, dimensionality reduction.

Reinforcement — agent takes actions, gets rewards. Not the focus of this course.

The Splits Nobody Should Mess Up

Train fits weights. Val tunes hyperparameters (number of layers, learning rate, regularisation strength, augmentation policy). Test reports final performance, exactly once. If you tune on test, test becomes a second val and you have no unbiased estimate of generalisation. This rule is sacred — exam graders will dock marks for "we tuned the model based on the test set's performance".

A related warning: the curse of dimensionality. In high dimension, distances concentrate (nearest and farthest become close to equal), volumes blow up, and methods like kNN or kernel density estimation need exponential data to fill the space.

Logistic Regression — The Canonical Classifier

For binary classification, the simplest model that respects "output should be a probability" is logistic regression: $\hat{y}=\sigma (w^{\top }x+b)$ where $\sigma (z)=1/(1+e^{-z})$ squashes any real number to $(0,1)$.

Train with binary cross-entropy: $L=-y\log \hat{y}-(1-y)\log (1-\hat{y})$. Not MSE. Two reasons: (a) MSE on top of sigmoid is non-convex — local minima everywhere. (b) Gradients vanish at saturation; CE has the clean $(\hat{y}-y)x$ form because the sigmoid derivative cancels.

The decision boundary is linear: $w^{\top }x+b=0$. So logistic regression can only separate classes that are linearly separable in feature space. (Apply a non-linear feature map first — that's what kernel methods and neural networks do.)

For $K>2$ classes, generalise to softmax: $P(y=k\mid x)=e^{w_k^{\top }x}/{\sum }_j{e}^{w_j^{\top }x}$, loss $-{\sum }_k{y}_k\log {\hat{y}}_k$. This is what sits on top of every CNN and ViT — the final "MLP head + softmax" is multinomial logistic regression.

Neural Networks — Just Layered Logistic Regression

A neuron is $y=\varphi (w^{\top }x+b)$. Stack into layers. Activations: sigmoid (saturates both ends, vanishing gradient), tanh (zero-centred but still saturates), ReLU $\max (0,x)$ (fast, sparse, gradient = 1 in positive half, can "die" if always negative), Leaky ReLU and GELU fix the dying problem.

Backpropagation is just the chain rule walked backwards through the network. Modern frameworks (PyTorch, TF) build a computation graph at forward time and run autodiff at backward time. You will be asked to compute backprop by hand on a 2-layer MLP — practise this once, it's mostly bookkeeping.

Initialisation matters. Don't init to zero — all neurons in a layer would be identical and stay identical forever (symmetry). Use Xavier/Glorot ($w\sim \mathcal{N}(0,1/n_{\text{in}})$) for sigmoid/tanh and Kaiming/He ($w\sim \mathcal{N}(0,2/n_{\text{in}})$) for ReLU. The factor of 2 in Kaiming compensates for the half-zeroed activations.

Six Tricks That Speed Up NN Training

(1) Batch Normalisation — normalise activations to zero-mean unit-variance per mini-batch, then learnable scale and shift. Faster training, lets you use higher learning rates, less init-sensitive, slight regularisation. (2) Higher LR (BN enables this). (3) Dropout — randomly zero $p$ fraction of activations during training (~0.5 in FC, ~0.1–0.3 in CNN); at inference, turn off. Acts as ensemble + regulariser. (4) Shuffle data between epochs. (5) Less L2 when BN is present. (6) LR decay / scheduling — start higher, decay over epochs.

Ensembles — Trade Variance For Bias Or Vice Versa

Bagging = Bootstrap Aggregating. Sample with replacement, train independent models, average predictions. Reduces VARIANCE. Random Forest is the canonical example.

Boosting = Sequential. Each model focuses on the previous model's errors via reweighting. Reduces BIAS. AdaBoost is textbook; Viola-Jones face detection (2001) used AdaBoost over Haar features to make real-time face detection practical.

Mix-up: confusing bagging and boosting is a guaranteed mark loss.

Density Estimation And GMM

To model $p(x)$ from data: histograms are bin-dependent and break in high $d$. KDE puts a Gaussian on every point. GMM is the parametric workhorse: $p(x)={\sum }_k{\pi }_k\mathcal{N}(x\mid {\mu }_k,{\Sigma }_k)$.

Train with EM. *E-step*: compute responsibilities ${\gamma }_{ik}={\pi }_k\mathcal{N}(x_i\mid {\mu }_k,{\Sigma }_k)/{\sum }_j{\pi }_j\mathcal{N}(x_i\mid {\mu }_j,{\Sigma }_j)$. *M-step*: re-estimate ${\mu }_k,{\Sigma }_k,{\pi }_k$ by weighted MLE. Iterate. The likelihood is guaranteed to increase monotonically, but you'll only reach a local maximum — init matters.

RNNs — Sequences

$h_t=f_{\theta }(h_{t-1},x_t)$. Same $\theta$ shared across all time steps. Train with BPTT (back-propagation through time).

The problem: $\partial L/\partial h_0\sim \prod W$, which either vanishes or explodes after many time steps. LSTM introduces a *cell state* that's updated additively: $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$. The three gates (forget, input, output) are sigmoids; ${\tilde{c}}_t$ is a tanh proposal. The additive path is the constant error carousel — gradient can flow back across many timesteps without vanishing.

GRU is the simpler variant: one update gate (merges forget + input) and one reset gate. No separate cell state. Often as good as LSTM, fewer parameters.

For exploding gradients (a different problem the gating doesn't fix), use gradient clipping.

RNN I/O shapes: 1-to-1 (image classification), 1-to-many (image captioning: CNN encoder → RNN decoder), many-to-1 (video classification, sentiment), many-to-many synced (per-frame labels), many-to-many shifted (translation, video captioning).

Metrics — The Bit Students Always Lose Marks On

Confusion matrix: TP, FP, TN, FN. From these:

Precision = $TP/(TP+FP)$ — of what you retrieved, how much was right. Monitor when false positives are costly (spam filter flagging real mail; criminal-conviction wrongness).

Recall = TPR = $TP/(TP+FN)$ — of what was right, how much did you retrieve. Monitor when false negatives are costly (cancer screening missing a tumour; safety inspections missing a fault).

F1 = $2PR/(P+R)$ — harmonic mean. Punishes one-sided extremes — you can't be 100% precise by retrieving nothing.

Specificity (TNR) = $TN/(TN+FP)$. FPR = $1-$ TNR.

Average Precision = area under the PR curve for one class. mAP = mean over classes (object detection) or queries (retrieval).

ROC vs PR. ROC plots TPR vs FPR. PR plots Precision vs Recall. For balanced data they tell similar stories. For imbalanced data with rare positives, ROC looks deceptively flattering (the abundant TNs inflate the AUC); PR is preferred — precision directly tracks how often you're right on the rare class.

kNN And Regression

kNN. Lazy. No training. To predict for $x$, find its $k$ nearest training points and vote. Small $k$ → high variance (noisy boundary, overfits). Large $k$ → high bias (smooths over real structure, underfits). Use $k$ odd to break ties. Distances: L2, L1, cosine, Mahalanobis. Curse of dimensionality bites hard.

Linear regression. $y={\beta }_0+{\beta }_{1}x+ϵ$. Least squares minimises SSE. Closed-form: $\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y$ (normal equations). Polynomial regression is still linear regression — linear in the coefficients $\beta$, just with polynomial features. Truly non-linear models (logistic, NN) need iterative methods.

L1 vs L2 regularisation. L1 (Lasso) $\lambda \sum |w|$ → exact zeros → feature selection. L2 (Ridge) $\lambda \sum w^2$ → smooth shrinkage, no exact zeros.

PCA / SVD — The Dimensionality Toolkit

PCA finds the orthogonal directions of maximum variance. Algorithm: centre the data; compute covariance $\Sigma$; eigendecompose $\Sigma =V\Lambda V^{\top }$; keep the top-$k$ eigenvectors; project. ${\lambda }_i$ = variance along $v_i$. The top-$k$ projection captures the most variance among all $k$-dimensional projections.

SVD decomposes any matrix: $X=U\Sigma V^{\top }$. The right singular vectors $V$ are PCA components (eigenvectors of $X^{\top }X$). Numerically stable (no need to form $X^{\top }X$, which squares the condition number). Best rank-$r$ approximation in Frobenius norm is $X_r=U_{:r}{\Sigma }_{:r}{V}_{:r}^{\top }$ (Eckart–Young).

LoRA = Low-Rank Adaptation: write the weight update as $\Delta W=AB$ where $A\in {\mathbb{R}}^{d\times r},B\in {\mathbb{R}}^{r\times d},r\ll d$. Train only $A,B$ — saves vast amounts of memory and compute when fine-tuning huge LLMs. Spiritually the same trick as SVD/PCA: low-rank structure compresses information.

Eigenfaces apply PCA to face images: top eigenvectors look like ghostly average faces; represent any face as a combination of them, classify by nearest neighbour in the low-dim space.

Clustering — Unsupervised Grouping

k-means. Init $K$ centres; assign each point to nearest centre; recompute centres as mean of assigned; iterate. Four issues to memorise: HARD assignments, SPHERICAL bias, INIT sensitivity, OUTLIER sensitivity. Fixes: k-means++ (smart init that spreads centres), GMM (soft assignments + covariance shapes), k-medoids (use the actual data point closest to mean, robust to outliers).

Hierarchical clustering. Agglomerative (bottom-up): each point starts as a cluster, merge closest pairs. Divisive (top-down): one cluster, recursively split. Output: a dendrogram. Cut at different heights → different numbers of clusters. Linkage: single (closest pair — can chain), complete (farthest pair — compact), average, Ward (variance-minimising).

What You Walk In Carrying

Train/Val/Test discipline. Logistic regression: sigmoid + CE, SGD update $w\leftarrow w-\eta (\hat{y}-y)x$, why not MSE, why no zero init. NN basics: layers, activations, backprop, BatchNorm, dropout, Kaiming init. Six speed-up tricks. Bagging vs Boosting (RF vs AdaBoost / Viola-Jones). GMM and EM (E-step responsibilities, M-step weighted MLE, monotone improvement). RNN: $h_t=f(h_{t-1},x_t)$; LSTM 3 gates + additive cell state; GRU 2 gates; BPTT + clipping. Metrics: confusion matrix, P/R/F1, AP/mAP, ROC vs PR (PR for imbalanced). kNN: lazy, k odd, variance/bias trade-off, curse of dim. Linear regression: normal equations $(X^{\top }X{)}^{-1}{X}^{\top }y$; L1 vs L2. PCA: centre → eigendecompose → top-$k$; SVD; Eckart–Young; LoRA. Clustering: k-means + 4 issues + k-means++ fix; GMM soft; hierarchical + linkage.