No teacher? Picture teach itself!

Long time, Ugg need smart human to point and grunt "this mammoth, this rock" — slow, cost many shiny stones. Then clever caveman ask: what if picture teach ITSELF?

This called self-supervised learning (SSL) — network learn from shape hiding inside the data, no human label. Ugg build a puzzle the picture can answer alone: "these two crop-pieces come from same picture, yes?" Solve many such puzzle, features grow sharp and useful. Every modern beast — CLIP, DINO, MAE — grow up this way. No label. Just structure.

Four tribe of SSL: (1) old-school — jigsaw, colour-in, autoencoder. (2) contrastive — SimCLR, MoCo, BYOL, SwAV. (3) language-image contrastive — CLIP. (4) generative — masked autoencoder (MAE). This lesson about tribe 2 and 3. Trivia: the Gelato Bet — elder Efros bet a no-label model would catch a human-label model; won ~2020. Exam grunt "Gelato Bet" = SSL caught supervised.

Pull same together, push different apart

Contrastive learning — one rule for whole tribe. Take one picture, make two shaken copies (crop it, shake its colour). Two copies = positive pair (friends, same picture!). Every OTHER picture in the batch = negative (stranger!). Train so friends stand close, strangers stand far. That IS it. Only thing that change between method: what shake, where strangers come from, how loss shaped.

Measure "close" with cosine similarity — angle between two arrow-stones: $\text{sim}(u,v)=\frac{u^{\top }v}{\Vert u\Vert \Vert v\Vert }$. Point same way → 1; point apart → small. Only direction matter, not length.

SimCLR — simplest fire

SimCLR (Chen 2020). Four stone-parts: (1) augmentation shake picture into the two views. **(2) encoder $f_{\theta }$** — big rock-brain (ResNet) — squeeze picture into feature $h$. **(3) projection head $g_{\varphi }$** — small 2-layer brain — map $h$ to space $z$. (4) NT-Xent loss, on $z$, NOT on $h$. Ugg warn big: after training, THROW AWAY $g_{\varphi }$, keep only $h$ for the hunt — head there at training, gone at downstream. Why? The loss is brutal, rubbing out colour and position; the head soak that up so brain $f$ keep the rich stuff. Worth 5–10% downstream.

Aggressive augmentation the headline — shake HARD. Random crop + colour-jitter the two strong shakes; both must be strong, together beat either alone. Crop teach "same thing near or far," jitter teach "same thing any colour" — so network cannot cheat on an easy hint.

The loss, NT-Xent (= InfoNCE). Batch of $N$ picture → $2N$ views. For friend pair $(z_i,z_j)$:

$L_{i,j}=-\log \frac{\exp (\text{sim}(z_i,z_j)/\tau )}{{\sum }_{k\ne i}\exp (\text{sim}(z_i,z_k)/\tau )}$

Top = friend closeness. Bottom = sum over ALL other $2N-1$ views (friend AND strangers). $\tau$ (temperature, ~0.07–0.5) sharpen the pick. Exam gold: this is EXACTLY softmax cross-entropy — logits = similarity, true-label = which one is the friend. Also bigger batch much better (more strangers in the bottom — batch 256 → ~64%, batch 8192 → ~76%). But 8192 need a giant TPU pod; small cave cannot. Next hunter fix this.

MoCo — keep strangers in a cave-queue

MoCo (He 2020) split number-of-strangers AWAY from batch size. Memory queue — a long FIFO line of $K=65,536$ old features; strangers come from this line, not the batch. Many strangers, small batch! Momentum encoder — a slow twin, EMA of the real one: $\xi \leftarrow m\xi +(1-m)\theta ,m\approx 0.999$. Twin crawl only 0.1% per step, so the whole queue look like it came from one steady brain — this is consistency. Ugg warn: twin NOT trained by backprop — only real encoder $\theta$ learn; twin $\xi$ just slow-copy it. Each step, enqueue new keys, dequeue oldest.

BYOL — throw away ALL strangers

BYOL (Grill 2020), "Bootstrap Your Own Latent." Heretic question: what if NO negative at all? Tribe cried "collapse — same arrow for every picture!" BYOL show: no, if built right. Online branch = $f_{\theta }\to g_{\theta }\to q_{\theta }$ (encoder, projector, extra predictor). Target branch = $f_{\xi }\to g_{\xi }$ (no predictor), $\xi$ = EMA of $\theta$. Loss push online guess onto target: $2-2\cos (\text{online},\text{target})$.

Three magic stones, all needed: (1) predictor — small brain on the ONLINE side only (break symmetry). (2) stop-gradient on target — no backprop through $\xi$. (3) EMA update of $\xi$ from $\theta$. Why no collapse? Predictor break symmetry (only online can perfect-match, so cannot cheat to a constant); EMA keep target a bit behind — chase moving prey, cannot lock up. Ugg warn: remove stop-gradient and it DOES collapse to one constant arrow, loss zero, features useless.

SwAV — sort into cluster-piles

SwAV (Caron 2020), "Swapping Assignments between Views." Learn $K\approx 3000$ prototype arrows (a codebook). For each view: normalise feature $z$, then get a soft pile-pick $q$ with the Sinkhorn-Knopp algorithm — it force every pile to be used about equally across the batch, so no collapse to one pile. The swap: from view $v$, guess the OTHER view's pile $q_{v^{\prime }}$. Cheap — no pairwise compares. Ugg warn: remove Sinkhorn, all views fall in one pile — dead.

CLIP — words join the hunt

CLIP (Radford 2021) use paired text as teacher instead of picture-only. Why word beat fixed class-label? Label no scale (past ~10k class, impossible to label); label weak-describe ("white shirt" vs "blue shirt" — own class each? madness); label no compose ("laptop on top of table" is two things plus a relation, no one class fits). Word have none these limits, and internet give billions picture-word pairs free. (Grandfather DeViSE, 2013, tried this picture-text joint space 8 year early, too small.)

Two encoders, one shared space: picture-brain $f_I$, word-brain $f_T$, both L2-normalise. Build an $n\times n$ table of picture-vs-word closeness; friends sit on the diagonal (picture $i$ ↔ text $i$). Loss is symmetric: cross-entropy down the rows (picture→word) PLUS down the columns (word→picture), averaged: $L=\frac{1}{2}(L_{i\to t}+L_{t\to i})$. Ugg warn: loss go BOTH ways, not one way!

Secret sauce is NOT a clever brain — it the DATA. WIT (WebImageText): 400 million picture-word pairs, 500k searches, up to 20k pairs each. Exam line: why CLIP work? → scale of natural-language supervision, 400M pairs. Zero-shot magic — classify a brand-new task, no training:(1) write each class as "a photo of a {class}", embed as text;(2) embed the image;(3) cosine image-vs-each-text;(4) pick argmax. CLIP hit ~76% top-1 on ImageNet with zero labels of that task — matching a fully-trained ResNet-50. Ugg warn: this ZERO-shot, not few-shot.

CLIP weak spot: not fine-grain (miss exact colour, breed); not compositional — "person on horse" vs "horse on person" score the same, because CLIP match the WHOLE picture, not the relation. Winoground (2022) show CLIP near chance when only word-order swaps. Yet CLIP's text-brain live inside Stable Diffusion — every such picture made with CLIP inside.

Ugg remember

• SSL = supervision from data structure, no human label. Four tribe: old-school, contrastive, language-image, generative.
• Contrastive: two views of one picture = positive (pull close), all else = negative (push far). NT-Xent = softmax cross-entropy; its bottom includes the positive too.
• SimCLR = aug + $f$ + projection $g$ (THROW $g$, keep $f$) + NT-Xent, big batch. MoCo = queue + momentum encoder (EMA, not backprop). BYOL = predictor + stop-grad + EMA, NO negatives. SwAV = prototypes + Sinkhorn swap.
• CLIP = two encoders, shared space, SYMMETRIC InfoNCE, diagonal = positives; power from 400M-pair WIT data, not architecture.
• Zero-shot ≠ few-shot: "a photo of a {class}" + cosine + argmax → ~76% ImageNet, no fine-tune.