Ugg turn pile-of-numbers into wise-rock

Machine Learning (ML) — Ugg feed rock a pile of past hunts, rock learn the pattern, rock become a function that guess the future. Inside every seeing-machine (computer vision) sit these same wise-rocks: a softmax head IS logistic regression, a deep filter-stack IS a brain-net.

Three kind of learning-hunt. Supervised — Ugg show question $X$ and answer $Y$; two flavor, classification (answer is a bucket: deer or wolf) and regression (answer is a number: how heavy). Unsupervised — only $X$, rock find own groups. Reinforcement — rock poke world for reward; not this cave.

Three piles, no cheat with test

Split hunts three ways. Train pile — fit the weights. Validation pile — tune the knobs (rock size, learn-rate, regularization strength). Test pile — measure final power ONE time. Ugg warn: no tune on test-pile! Peek at test to pick knobs and it becomes another val — no honest guess of new-mammoth power left. Curse of dimensionality: too many numbers per footprint (high $d$) and all distances look the same, need a mountain of data — kills the nearest-neighbor magic.

Sigmoid-rock: the yes/no hunter

Logistic regression — $\hat{y}=\sigma (w^{\top }x+b)$. The sigmoid $\sigma (z)=1/(1+e^{-z})$ squash any number into $(0,1)$, a "how-sure." Train with binary cross-entropy $L=-y\log \hat{y}-(1-y)\log (1-\hat{y})$ — punish confident-and-wrong hard. Ugg warn: no use MSE (square-error) here! Under sigmoid it is bumpy (non-convex) and the gradient dies at the flat ends.

Learn step is clean: $w\leftarrow w-\eta (\hat{y}-y)x$ — nudge weight by error times input, times step-size $\eta$. The nasty sigmoid-slope cancels; THAT is why cross-entropy is friend. The decision boundary is a straight line $w^{\top }x+b=0$ — sigmoid-rock only split what a spear-line splits. Many buckets? Softmax $P(y=k|x)=e^{w_k^{\top }x}/{\sum }_j{e}^{w_j^{\top }x}$ — share belief across $K$ classes, add to one.

Stack the rocks: brain-net

One neuron: $y=\varphi (w^{\top }x+b)$ — weigh, add, bend through activation $\varphi$. Stack into layers = Multilayer Perceptron (MLP). Activations: sigmoid and tanh bend nice but go flat at ends (vanishing gradient — slope die, learning stall). ReLU $\max (0,x)$ — fast, sparse, but a neuron can "die"; Leaky-ReLU and GELU keep it alive.

Backpropagation — chain rule walked backward, each step multiply the local slope, push flows from loss back to every weight. Ugg warn: no start weights at zero! All-zero weights make every neuron in a layer identical, they get identical push forever (symmetry), whole layer collapse to one. Fix with random start: Xavier $\mathcal{N}(0,1/n_{in})$ for sigmoid/tanh, Kaiming (He) $\mathcal{N}(0,2/n_{in})$ for ReLU (the 2 pays back the half of neurons ReLU zeroes out).

Six fire-tricks to train faster

(1) Batch Normalization — squeeze each mini-batch of activations to zero-mean unit-variance, then learn scale $\gamma$ and shift $\beta$.(2) higher learn-rate (BN allow it).(3) Dropout — randomly silence fraction $p$ of neurons each step; like many rocks voting; turn OFF at hunt-time.(4) shuffle data each pass.(5) less L2 when BN present.(6) decay learn-rate over time. Ugg warn: at hunt-time BN uses the stored running mean/var, NOT the batch — mix that up and machine break in the field.

Many-rock tribe: bagging vs boosting

Bagging (bootstrap aggregating) — draw many samples with replacement, train rocks apart, average → cut variance; Random Forest is the classic. Boosting — rocks in a line, each new one focus on the last one's mistakes → cut bias; AdaBoost is textbook, and Viola-Jones face-finder (2001) used it. Ugg warn: no mix up! Bagging = parallel = kill variance. Boosting = sequential = kill bias.

Guess the berry-cloud: GMM + EM

Gaussian Mixture Model (GMM): $p(x)={\sum }_k{\pi }_k\mathcal{N}(x\mid {\mu }_k,{\Sigma }_k)$ — a weighted sum of bell-hills, weights ${\pi }_k$ add to one. Train with EM (Expectation-Maximization). E-step: for each point compute responsibility ${\gamma }_{ik}$ — how much hill $k$ owns point $i$. M-step: re-place each hill's center ${\mu }_k$ and spread ${\Sigma }_k$ by weighted average. Repeat — likelihood always climbs, but only to a local hill-top, so start matters.

Sequence-rock: RNN and LSTM

RNN: $h_t=f_{\theta }(h_{t-1},x_t)$ — same weights reused each step, carry memory $h$; train with BPTT (backprop through time). Trouble: gradient $\sim \prod W$ — vanish if small, explode if big. LSTM fix vanish with an additive cell-state highway $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$ (three gates: forget, input, output) — this constant error carousel carries memory far back. GRU — two gates, simpler, often as good. For explode: gradient clipping (cap the size).

Count hits and misses: metrics

From TP, FP, TN, FN: Precision $P=TP/(TP+FP)$ (of what Ugg grabbed, how much right), Recall $R=TP/(TP+FN)$ (of all real, how much caught), F1 $=2PR/(P+R)$ (harmonic mean). *Ugg warn: F1 is NOT $(P+R)/2$!* Watch precision when a false-alarm costs (spam filter eats a real letter); watch recall when a miss costs (screening misses a real tumor). AP = area under the precision-recall curve, mAP = mean over classes. Ugg warn: on lopsided data ROC look too pretty (many easy TN) — use the PR curve; and plain accuracy is a lie (always-say-no scores 99%).

Lazy hunter, straight line, squeeze

kNN — no training, just store points, guess by vote of the $k$ nearest. Small $k$ jumpy (variance), big $k$ too smooth (bias); keep $k$ odd; curse of dimension bites hard. Linear regression $y={\beta }_0+{\beta }_{1}x+ϵ$, least squares, closed form normal equations $\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y$ — fails when features collinear (can't invert), then use gradient descent. Polynomial regression is still "linear" (linear in $\beta$). L1 (Lasso) $\lambda \sum |w|$ pushes weights to exact zero → pick features; L2 (Ridge) $\lambda \sum w^2$ shrinks smooth, no zero.

Squash the footprint: PCA and SVD

PCA (Principal Component Analysis) — find straight directions of biggest spread: center the data, make covariance $\Sigma$, eigen-crack $\Sigma =V\Lambda V^{\top }$, keep the top-$k$ eigenvectors, project; ${\lambda }_i$ = variance along $v_i$. Ugg warn: center first! or the top direction just points at the mean. SVD $X=U\Sigma V^{\top }$ — the $V$ ARE the PCA directions, ${\sigma }_i=\sqrt{{\lambda }_i}$; more stable (no squaring), and top-$r$ = best rank-$r$ copy (Eckart–Young). LoRA writes a weight change as thin $\Delta W=AB$ with $r\ll d$ — cheap fine-tune of giant rocks, same low-rank spirit. Eigenfaces = PCA on faces.

Group without answers: clustering

k-means — pick $K$ centers, assign each point to nearest, move center to the mean, repeat. Four woes: HARD assignment, likes SPHERE clusters, START-sensitive, OUTLIER-sensitive. Fixes: k-means++ (smart spread start), GMM (soft assignment + shapes), k-medoids (outlier-tough). Hierarchical clustering — merge closest pairs bottom-up (agglomerative) or split top-down (divisive) into a dendrogram tree; cut at a height to choose the cluster count. Linkage: single (chains thin), complete, average, Ward.

Ugg remember

• Sigmoid + cross-entropy, step $w\leftarrow w-\eta (\hat{y}-y)x$; never MSE, never zero-init.
• Bagging kills variance (Random Forest); Boosting kills bias (AdaBoost, Viola-Jones).
• EM: E-step responsibilities, M-step weighted re-fit; likelihood only climbs, to a local top.
• $P=TP/(TP+FP)$, $R=TP/(TP+FN)$, $F1=2PR/(P+R)$; PR curve for lopsided data; precision when a false-alarm hurts, recall when a miss hurts.
• PCA = eigenvectors of covariance (center first!); SVD = the stable same trick; k-means = hard clusters, seed with k-means++.