Ugg chop picture into rocks, feed to word-brain

For nine winters — since the AlexNet hunt of 2012 — every best eye-machine was a CNN (Convolutional Neural Network): a machine that slide a little stamp-rock over picture, seeing only close-neighbor pixels. Tribe swear eye MUST work this way. Then Google shaman show new trick: a plain Transformer — same word-brain that turn English grunt into French grunt — beat CNN at seeing! One catch: first feed it a MOUNTAIN of picture-food.

The magic scroll called "An Image is Worth 16×16 Words." That IS the recipe: chop picture into $16\times 16$ squares, call each one a word-token, feed to word-brain. Machine named ViT (Vision Transformer). After 2021, ViT push CNN off the throne.

Why Ugg no feed every pixel

Why not make every pixel a token? A $224\times 224$ picture have $50,176$ pixel. Word-brain self-attention — every token stare at every other — cost grow like $\mathcal{O}(N^2)$: that is $\approx 2.5$ BILLION look-looks per layer. Fire run out! So Ugg squish the line by chopping into patch-rocks.

How Ugg make the rocks

Patch tokenisation — chop picture into non-overlap $P\times P$ squares.

For $224\times 224$ with $P=16$: $N=14^2=196$ rocks. Flatten each rock to $P^{2}C$ numbers, then squish to $D$ numbers with learn-matrix $E$:

Ugg WARN: patch-making is NOT "just flatten"! It a learned linear projection — same as one Conv2d, kernel $P$, stride $P$. For ViT-B, $P^{2}C=16^2\cdot 3=768=D$, so $E$ almost square.

[CLS] token — glue one special learn-rock at the front: $[\text{CLS},{\text{patch}}_1,\dots ,{\text{patch}}_N]$. Line now $N+1=197$ long. Ugg WARN: never forget the $+1$! At the end, push the final CLS-rock through one linear layer for the class-guess. Could instead global-average-pool all rocks — within 1%. ViT keep CLS because BERT did.

Position embedding — brain no know which rock sit where, so add place-marker. Choices: none (drop ~3%), 1D learned (number rocks in reading-order, learn one vector each — THIS is ViT), 2D learned (tiny gain). Remember: ViT use 1D markers even though picture is 2D. Brain figure out the 2D by itself.

The word-brain block

ViT use Pre-Norm: clean numbers with LayerNorm BEFORE each step, add old numbers back after.

MSA = multi-head self-attention (rocks look at rocks). MLP = think-step: ${\text{Linear}}_1(D\to 4D)\to \text{GELU}\to {\text{Linear}}_2(4D\to D)$. The $4\times$ fat-middle is standard.

ViT herd — burn the Base into memory: ViT-Base has $L=12$, $D=768$, MLP $=3072$, $H=12$, ~86M. ViT-Large ~307M, ViT-Huge ~632M. Always MLP $=4D$, per-head $d_k=D/H\approx 64$. "ViT-B/16" = Base with $16\times 16$ rocks.

Count the stones (exam love this)

Per block: attention $W_Q,W_K,W_V,W_O$ each $D\times D$ → $4D^2$; MLP up+down → $8D^2$. **Per block $=12D^2$.**

Match slide's $85,524,480$. LN, CLS, markers too small to count.

The four quiz stones

• **Q1: bigger picture $224\to 336$** (patch stay 16). Rocks $196\to 441$, sequence UP ~2.25×. Params ~SAME. But markers now wrong size — bilinear-interpolate them, no new params.
• **Q2: smaller patch $32\to 16$** (picture stay). Rocks $49\to 196$, sequence UP exactly 4×. Params shrink tiny ($E$ smaller); attention compute ~4× more, memory ~16× more.
• **Q3: more layers $8\to 12$.** Sequence SAME. Params UP linear — each block $+12D^2$, so $+28$M.
• **Q4: more heads $1\to 8$** ($D$ fixed). Ugg WARN, trickiest! Sequence SAME, params SAME. Heads only SPLIT the $D$ space ($d_k=D/H$); projections stay $D\to D$. Heads no multiply stones! Most tribe grunt this wrong.

Why ViT so hungry

Inductive bias — a machine's built-in belief. CNN born with two: locality (nearby matter most) and translation equivariance (move object, still same object). Small food → belief HELP; big food → belief LIMIT. ViT born with almost none — every rock stare at every rock from layer 1. Small food (ImageNet): ViT overfit, ResNet win. HUGE food (JFT-300M, 300 million picture): ViT win by far, and transfer better everywhere.

Do ViT see like CNN?

Raghu shaman use CKA (Centered Kernel Alignment) — a sameness-ruler for machine-thoughts. CNN thoughts climb a slow ladder (texture → part → whole object); ViT thoughts stay UNIFORM — even layer 1 look across the whole picture. So attention distance is both small AND large in every ViT layer: global eyes right away, but no multi-scale ladder, making plain ViT weak at dense job (segment, detect). Bonus: the CLS-rock learn to stare at the true object (DINO grow this into free segment-maps), and 1D markers end up self-similar in 2D — sameness high near, fade radial.

Swin — patch the two ViT holes

ViT two weakness: (1) $\mathcal{O}(N^2)$ too costly at big picture; (2) no ladder of scales. Swin (Shifted Window) Transformer fix both.

Idea 1 — Windowed attention. Chop rocks into $M\times M$ windows ($M=7$), attention only INSIDE each window. Cost drop $\mathcal{O}(N^2)\to \mathcal{O}(M^{2}N)$ — **linear in $N$**. On $56\times 56$ grid: 9.8M → 154k look-looks, ~64× cheaper. Ugg WARN: "linear" true only when window $M$ stay fixed.

Idea 2 — Shifted windows. Pure windows never talk across walls. Fix: every OTHER block, slide the grid by $(M/2,M/2)$. Block $\ell$ = W-MSA (aligned), block $\ell +1$ = SW-MSA (shifted) — now rocks from old-separate windows share one, so news travel across walls. Ragged edge windows fixed by cyclic shift + masked attention (wrap rocks around, mask the wrap). The cyclic shift IS how the shift get done — not a separate thing.

Idea 3 — Patch merging. ViT keep same rock-count forever; Swin shrink like CNN. Take a $2\times 2$ block, glue features, project $4C\to 2C$ — half the space, double the channel. Four stages: $56^2$ at $C\to 28^2$ at $2C\to 14^2$ at $4C\to 7^2$ at $8C$ — a real feature-pyramid, so Swin drop into detection and segment machines.

Ugg remember

• **ViT = chop picture into $16\times 16$ patch-tokens, add [CLS] + 1D learned markers, run $L$ Pre-Norm Transformer blocks, read the CLS-rock for the guess.**
• **Per block $=12D^2$** ($4D^2$ attention $+8D^2$ MLP). ViT-B: $L=12,D=768$, MLP $3072$, $H=12$, ~86M. Sequence $=N+1=HW/P^2+1$.
• Quiz Q4: more heads → params UNCHANGED. Heads split $D$, no multiply. The trap tribe always fall in.
• ViT hungry: small food ResNet win, JFT-300M big food ViT win — ViT own no locality bias to lean on.
• **Swin = window attention ($\mathcal{O}(N)$) + shifted windows (cross-talk) + patch merging (4-stage pyramid).** ViT to pretrain, Swin to detect and segment.