Old hunt need enemy. New hunt need no enemy.

Long-ago hunt was contrastive learning — Ugg pull same-thing rocks close, push different-thing rocks far. The far ones Ugg call negatives — enemy rocks. But enemy-hunt cost much: need MANY enemy rock (big pile, big memory-cave), get fussy about picture-twists, and features come out okay, not great.

Then three wise-caves (year 2021) ask: why Ugg need enemy rock at all? One by one they show — NO NEED. Three new fires: DINO, MAE, JEPA. This the second great tribe of teach-self learning (self-supervised learning — learn from picture alone, no label-stone), beside the contrastive tribe.

DINO — cub copy big-brother, no name-tag

DINO means "self-DIstillation with NO labels." Take one picture-chew brain (a ViT, vision transformer). Make TWO copy:

• Student $f_{{\theta }_s}$ — the cub. Learns by backprop (push-back-the-error). This one changes.
• Teacher $f_{{\theta }_t}$ — big-brother. Same shape, different weight-stones, NO backprop.

Both spit a big fan of chance-numbers over $K=65,536$ slots (a learned codebook), squashed by softmax (turn numbers into chances that add to one):

The $\tau$ is a heat-knob (temperature). Cub must match teacher's fan for the same picture (twisted different ways):

This cross-entropy — how far cub-belief sits from teacher-belief. Error pushes ONLY through cub. Teacher no learn from error — it just hands soft answer-hints.

How teacher grow? Teacher is a slow smooth echo of cub — EMA (exponential moving average):

Early: $\lambda \approx 0.996$, teacher follows cub quick-ish. Late: $\lambda \to 1$, teacher freeze like ice. So cub chases a slightly-late, smoothed copy of self, and shaky mistakes average away. (Borrowed from MoCo/BYOL.)

Multi-crop — DINO's signature. Ugg cut one picture many ways, two sizes: 2 big "global" views (>half the picture, $224\times 224$) go to BOTH teacher and cub; 6–10 small "local" views (<half, $96\times 96$) go to CUB ONLY. Every (teacher-global, cub-other) pair makes one cross-entropy; sum them. Teacher sees only big, cub sees big AND small — so cub must guess the whole-scene fan from one little scrap: "this grass-patch belongs to that wide meadow." That is the local-to-global magic.

Ugg fear flat-brain sickness — collapse

Danger! Lazy cub-and-teacher find a cheat: spit the SAME constant fan for EVERY picture. Loss goes low, but all knowing is DEAD. This collapse. Two sick flavors: single-dim domination (softmax always spikes the SAME slot) and uniform output (every slot equal $1/K$, flat like still water). DINO block BOTH with two fire-tricks.

Trick 1 — Centering (blocks single-dim spike). Teacher subtracts a running-average bias $c$ from its logits before softmax:

The bias-stone stops any one slot always winning. Slide-word: "centering done through bias term added to logits."

Trick 2 — Sharpening (blocks flat water). Teacher uses VERY low heat ${\tau }_t\approx 0.04$ — much colder than cub's ${\tau }_s\approx 0.1$. Cold heat spikes the fan near one-hot, so a confident teacher always gives a real hint. Slide-word: "sharpening done using low value of temperature."

Both needed — opposing forces. Centering alone → all flat (uniform). Sharpening alone → all spike one slot. TOGETHER → steady. Remember: centering spreads, sharpening peaks.

What DINO catch: 78.3% top-1 on ImageNet using just k-NN on frozen features (no extra training); 76.1% with linear probe; trained on 2×8-GPU, 3 days. The marvel: DINO's [CLS] token attention draws the object shape by itself — segmentation-like maps with NO segmentation teaching. DINOv2 scaled this to 142M hand-picked pictures. DINOv2 + registers add spare register tokens (scratch-stones, no place-meaning) so real patch-stones stop being scratch-pad — cleaner attention.

MAE — the cover-up game, vision's BERT

MAE (masked autoencoder) is vision's BERT. BERT hides ~15% of word-stones and guesses them back. Do the same at 15% for pictures? BAD — brain cheats by copying texture from a neighbor patch, learns nothing. Pictures too repeat-y; a missing patch is easy to guess from neighbor color.

MAE fix — hide 75%! Slide: "BERT mask 15%, MAE remove 75% of image tokens" and "make pretraining difficult, otherwise model will shortcut." Rebuild 75% from only 25% visible — no cheap copy-neighbor trick left, so brain must learn whole-shape.

Asymmetric encoder-decoder (cheap-compute trick): cut picture into $N$ patches; hide 75%, keep 25%; a deep encoder chews ONLY the 25% visible (never sees hidden); put learnable mask tokens back at hidden spots, each with its place-mark (positional embedding); glue visible-chewed + mask tokens; a small light decoder chews the full set and paints back the hidden PIXELS. Encoder sees a quarter → cost drops ~4×. Decoder is little, does the rebuild, then gets THROWN AWAY — only encoder kept for real hunt.

Pixel MSE comparing true patch $x_i$ to painted patch ${\hat{x}}_i$, only on the hidden set $\mathcal{M}$ (visible patches given free — no loss there). Peak at 75% mask — remember the number.

JEPA — no paint rock, guess the rock-spirit

DINO leans on hand-picked picture-twists. MAE leans on pixel-paint — but pixel detail (texture, light, noise) is mostly USELESS for meaning. LeCun asks: why paint pixel at all? Better guess the representation — the shape-spirit — of the missing part.

JEPA — Joint-Embedding Predictive Architecture. I-JEPA has three parts: a context encoder $f_{\theta }$ (sees one chunk, makes features); a target encoder $f_{\xi }$ (sees other chunks, grows by EMA of $\theta$, NO gradient); and a predictor $g_{\varphi }$ (from context features + where targets sit, guesses target features).

L2 distance in FEATURE space, with stop-gradient ($\text{sg}$) on the target side. Target grows by EMA, not error, which blocks the lazy-constant cheat. MAE vs JEPA: MAE guesses PIXELS, JEPA guesses REPRESENTATIONS; MAE wastes brain on texture/color, JEPA wastes on NOTHING but meaning; MAE's costly part is the pixel decoder, JEPA's is the target encoder. Variants: I-JEPA (image, 2023), V-JEPA (video, 2024), VL-JEPA (vision-language, 2025).

Ugg warn: no do this!

• DINO uses NO enemy rocks (no negatives). Calling DINO "contrastive" — WRONG.
• DINO's EMA teacher is NOT MoCo's momentum key-encoder — same-looking math, different job.
• MAE hides 75%, not 15%. The 15% is BERT.
• MAE keeps only the ENCODER; the decoder is thrown away.
• DINO anti-collapse is TWO tricks (centering AND sharpening), not one.
• Teacher is colder: ${\tau }_t\approx 0.04<{\tau }_s\approx 0.1$. Never flip it.
• JEPA is NOT just "MAE in latent-cave" — separate EMA target-brain plus stop-grad.

Ugg remember

• No enemy rock needed. DINO, MAE, JEPA all learn with NO negatives — second SSL tribe after contrastive.
• DINO = cub copies EMA-teacher's soft fan (cross-entropy, error only through cub); multi-crop gives local-to-global; [CLS] attention draws objects free.
• Two tricks or DINO dies: centering spreads (blocks single-dim), sharpening ${\tau }_t\approx 0.04$ peaks (blocks uniform). Need BOTH.
• MAE = vision's BERT, hide 75%. Deep encoder on 25% visible, light throwaway decoder paints hidden pixels, keep encoder only.
• JEPA = guess the shape-spirit, not the pixel. Context + EMA target encoder + predictor, L2 in feature space with stop-grad.