Old whisper break when sentence too long

Long ago, tribe turn English into French with two magic boxes. First box read all English, squeeze it into ONE shiny stone. Second box hold that stone, grunt French words one by one. Ugg call this the Seq2Seq bottleneck — whole sentence crushed into single fixed vector. But one small stone cannot hold big hunt-story! "Cat that sat on mat" and "cat sat on mat" cannot both fit same stone. Long sentence, box forget, bad translate. This the sickness. Attention the cure.

Three cave-paintings that change everything

Ugg carve three names on wall, remember the chain:

• Bahdanau 2015 ("Jointly Learning to Align and Translate") — first stick attention onto RNN as add-on.
• Xu 2015 ("Show, Attend and Tell") — same attention, point at pictures, make captions.
• Vaswani 2017 ("Attention Is All You Need") — throw RNN in fire! Whole box now only attention. This the Transformer. Every modern beast — ViT, GPT, PaliGemma — child of this paper.

Three hunt-tasks Ugg must name: image captioning (picture in, words out), sentiment classification (words in, one label out), machine translation (words in, words out).

How box read and write

Encoder RNN read source, make hidden stones $h_1,\dots ,h_N$. Old way keep only last stone $h_N$. Decoder RNN start from it, spit one token, feed own token back, spit next — one at a time. This called autoregressive, a language model $p(y_t\mid y_{<t},x)$.

Attention — decoder look back at ALL stones

Bahdanau say: no crush everything in one stone! Every step, decoder LOOK BACK over all encoder stones, grab what it need. Four steps by the fire:

1. Alignment score — how good is stone $i$ for this step? $$e_{t,i}=v^{\top }\tanh (W_{\text{dec}}{h}_{t-1}^{\text{dec}}+W_{\text{enc}}{h}_i^{\text{enc}})$$ This a tiny brain (additive MLP) that sniff decoder stone and encoder stone together, grunt one number.

2. Softmax squash numbers into weights that sum to one: $${\alpha }_{t,i}=\frac{\exp (e_{t,i})}{{\sum }_j\exp (e_{t,j})}$$ These the attention weights.

3. Context vector — mix all encoder stones by their weight: $$c_t={\sum }_i{\alpha }_{t,i}{h}_i^{\text{enc}}$$ Big weight where it matter, small where not.

4. Combine $c_t$ with decoder stone, guess next word.

Magic: paint the weight map ${\alpha }_{t,i}$ and it glow near-diagonal — French "cat" light up on English "cat" stone. Nobody teach this alignment! It just emerge. Guaranteed fire-story for exam.

Soft vs hard. Soft attention = smooth weighted mix, leave footprint (differentiable). Hard attention = pick ONE stone by dice-roll, no smooth footprint, must train with REINFORCE. Soft win, become standard.

Fire-side training tricks

Teacher forcing — feed decoder the TRUE last word in training. Fast, but breed exposure bias (at real hunt, decoder only ever see its own words). Student forcing — feed decoder its OWN last guess. Harder train, but honest. Ugg warn: at inference, ALWAYS student forcing — no true word exist yet!

Greedy = grab biggest token each step, quick but short-sighted. Beam search = keep top-$B$ trails alive (say $B=4$), grow all, keep best $B$ again, at end pick best whole trail. More fire-wood, better words.

Softmax temperature: $p_i=\exp (z_i/T)/{\sum }_j\exp (z_j/T)$. $T=1$ normal, big $T$ flat (wild), small $T$ peaky (sure).

The Transformer — throw RNN in fire

RNN slow: step 5 must wait step 4. Serial. If attention already look anywhere, why not let EVERY token look at EVERY token, all at once, no recurrence? That the Transformer. Old paper: $N=6$ encoder blocks + $N=6$ decoder blocks.

Self-attention the engine. Take input $X$, make three roles with learned rock-matrices — Query (what I hunt?), Key (what I offer?), Value (what I carry?): $Q=X{W}_Q$, $K=X{W}_K$, $V=X{W}_V$. Then scaled dot-product attention: $$\text{Attn}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V$QK^\top$=everyqueryrubeverykey=pairwiselikeness($n\times n$).Softmaxmakeeachqueryaweightoverkeys.Times$V$ = each token grab weighted mix of values.

**Why divide by $\sqrt{d_k}$?** Ugg gold! Sum of $d_k$ unit-variance products has variance $d_k$, so std $\sqrt{d_k}$. For $d_k=64$, std $=8$ — too big, softmax SATURATE, one logit eat all, gradient die. Divide by $\sqrt{d_k}$ pull variance back to 1. Ugg warn: divide by $\sqrt{d_k}$, NOT $d_k$!

Multi-head attention: run $h$ small heads, each $d_k=d/h$, concat, project with $W_O$. Each head sniff different bond — one grammar, one who-is-who, one local. Ugg WARN loud: more heads NO make more stones! $W_Q,W_K,W_V$ together always cost $3d^2$ no matter $h$. Heads split the room, not add room.

Three kinds of look, and order-stones

Three flavours: encoder self-attention (Q,K,V all from input, no mask, look both ways); decoder masked self-attention (Q,K,V from output-so-far, causal mask stop peek at future); cross-attention (Q from decoder, K AND V from encoder — this Bahdanau in Q-K-V clothes!).

Causal mask = upper-triangle full of $-\infty$, so after softmax those become 0 — token $i$ only see tokens $\le i$. Ugg warn: forget this and decoder cheat, copy future, fail at real hunt. Same-thing names: look-ahead, autoregressive, left-to-right. Another warn: cross-attention take BOTH K and V from encoder — not K from decoder!

Positional encoding. Self-attention is permutation-equivariant — shuffle tokens, output shuffle same way. It cannot tell "cat sat" from "sat cat"! Fix: add sinusoidal positional encoding to input embeddings: $$\text{PE}(\text{pos},2i)=\sin (\text{pos}/10000^{2i/d}),\text{PE}(\text{pos},2i+1)=\cos (\text{pos}/10000^{2i/d})$$Eachspotgetownsine-cosinefootprintatslowerandslowerfrequency.Whysine?$\text{PE}(\text{pos}+k)$isaLINEARfunctionof$\text{PE}(\text{pos})$, so brain read relative distance easy. Ugg warn: add PE to the input BEFORE block 1, not after! Sinusoid stretch to longer sentence; learned PE does not.

The block, and eye-that-look-wrong

One encoder block: $z^{\prime }=\text{MSA}(\text{LN}(z))+z$; $z=\text{MLP}(\text{LN}(z^{\prime }))+z^{\prime }$. That "$+z$" mean add back the old stone (residual/skip connection) so gradient flow. Encoder block = 2 sub-layers (attention + MLP); decoder block = 3 (masked-attention + cross + MLP). MLP swell $d\to 4d\to d$ — the 4× rule. Old 2017 paper use Post-Norm $\text{LN}(x+\text{sublayer})$; modern use Pre-Norm $x+\text{sublayer}(\text{LN}(x))$ for stable deep stack. Ugg warn: original was Post-Norm, NOT Pre!

Show, Attend and Tell: picture → CNN → grid of spot-features, LSTM attend over spots for each word, heatmap show where model "look". But careful — heatmap show WHERE eye point, not that eye SEE right. Red-light trick fool it.

Why Transformer beat everyone

• Parallel — all token bonds at once, GPU happy, way faster than RNN.
• Short path — every token touch every other in ONE layer, no long-distance forget (cost: $O(n^2)$ pairs, the bill Flash Attention pay later).
• Universal — same beast eat text, image (ViT), sound, video, even DNA — just change tokenise.

Ugg remember

• One-stone bottleneck the disease; attention (look back at ALL encoder stones, weighted by softmax) the cure. Chain: Bahdanau 2015 → Show-Attend-Tell 2015 → Vaswani 2017 Transformer.
• Scaled dot-product $\text{softmax}(Q{K}^{\top }/\sqrt{d_k})V$ — divide by $\sqrt{d_k}$ (NOT $d_k$) to stop softmax saturate.
• Multi-head cost SAME stones as single-head ($3d^2$); heads split $d$, not multiply. Encoder block 2 sub-layers, decoder block 3, MLP is $4\times$.
• Three looks: encoder self, decoder masked-self (causal mask stop future-peek), cross (Q from decoder, K+V from encoder = Bahdanau).
• Self-attention blind to order → add sinusoidal PE to the input; sinusoid stretch to long, learned PE not. At inference, always student-forcing.