Ugg teach rock-eye to talk with word-mouth

Long ago, two tribe never share fire. Vision tribe eat picture, grunt name. Language tribe eat word, grunt word. Clever cave-folk (CLIP, BLIP, Flamingo, LLaVA) make ONE brain do both — a VLM (Vision-Language Model). Ugg walk you three brain: PaliGemma → Qwen2-VL → Gemma 4, each fix hole in one before.

The big split

Modality gap — deep split between picture and word. Word is countable rock: each word a number in $\{0,\dots ,|V|-1\}$, spirit fetched off a shelf, $e=W_{\text{emb}}[t]$, only ~32k word on shelf. Picture is endless river: $x\in {\mathbb{R}}^{H\times W\times C}$, dots 0–255, no shelf big enough — must run encoder, a grinding-stone that chew picture into spirit-numbers. Every VLM want one thing:

Push matching picture and word into SAME spirit-place. Rest is just carving-work.

Three rock make a VLM

Stack: Vision Encoder (chew picture) → Connector (fix shape) → LLM (big think-brain). Carve this spell on the wall:

Glue picture-token and word-token into ONE line, feed to normal brain. Ugg warn: NO special "cross-attention" beast here! Plain self-attention over the glued line do all mixing of eye and mouth — that is the whole trick.

PaliGemma — simplest brain that hunt good

Three rock: SigLIP-So400m eye-grinder (400M, frozen), Connector one flat rock (linear, fresh random), Gemma-2B brain (already trained). All together < 3B. Picture make 256 / 1024 / 4096 token for size 224 / 448 / 896.

SigLIP (Sigmoid CLIP): same shape as CLIP, one change in pain-count. Batch of $N$ picture-word pair — match on diagonal big, off-diagonal small. CLIP use softmax over whole row: big batch, all tribe must shout together each step, slow and wobbly. SigLIP use sigmoid — every pair its own yes/no fight. Score, then pain:

$\tau$ learn-able heat, $b$ learn-able tilt. Each pair alone → SigLIP grow to any batch size, no wobble. Ugg warn: no say "CLIP and SigLIP same loss, just different squash" — SigLIP truly pair-by-pair yes/no.

Inside, ViT (Vision Transformer) cut picture into square patch, patch-size $P=14$, so $N=(H/P)(W/P)$. For $224^2$: $N=16^2=256$ patch. Each patch flatten to spirit-number plus position-mark, $z_i=p_{i}E+e_i^{\text{pos}}$. **Carve on hand: $N=256$ patch, encoder width $D_{\text{enc}}=1152$.**

Flat-rock Connector stretch 1152 → 2048 (what Gemma want):

Big test-point: this flat rock is the ONLY fresh-random piece of PaliGemma. All else start already-trained; flat rock just line up two old spirit-place.

Prefix-LM mask: line is [ picture | prompt | answer ]. Picture+prompt = context → see BOTH way (every token see every token). Answer = thing to make → see only LEFT, one at a time (causal). Ugg warn: pain counted ONLY on answer token — picture and prompt are free-gift, brain no need guess them.

Three cook-stage:(1) unimodal — SigLIP and Gemma each learn alone, no mix;(2) multimodal — freeze SigLIP, train flat rock + LLM on big picture-word pile (224 → 256 token);(3) task transfer — un-freeze all, bigger picture, learn special hunt.

Everything-is-word trick: EVERY answer is word, even where-is answer. Detect dog → brain grunt four location token $⟨\text{loc0142}⟩⟨\text{loc0210}⟩⟨\text{loc0821}⟩⟨\text{loc0867}⟩$ then "dog"; the four mean $[y_{\text{min}},x_{\text{min}},y_{\text{max}},x_{\text{max}}]$. Shelf grow 1024 loc token + 128 segment codeword. Ugg warn: PaliGemma NEVER spit float number for box — only loc-token.

Qwen2-VL — fix two hole

Hole 1, resolution bottleneck: SigLIP squash to fixed size; 3mm spot on $2048^2$ X-ray shrink to nothing, gone forever, no get back. Hole 2, aspect distortion: squash wide 16:9 into square 1:1 bend everything, receipt look ~40% short. Fix both with dynamic resolution + M-RoPE.

Dynamic resolution: no squash. Cut patch at true size, $N_h=H/P$, $N_w=W/P$, $N=N_h{N}_w$ — change per picture (e.g. $448\times 224$ → $32\cdot 16=512$ token). Clamp by budget $N_{\text{min}}\le N\le N_{\text{max}}$: more token slower but sharper, hunter choose. Ugg warn: Qwen2-VL NOT stuck at 224 — native size, only clamp.

Why old 1D-RoPE break on picture: number patch by raster $m=\text{row}\cdot W+\text{col}$. Patch $(2,W-1)$ and $(3,0)$ sit far apart but $m$ differ by 1 — look neighbor, LIE. Patch $(2,0)$ and $(3,0)$ truly touch but $m$ differ by $W$ — look far, LIE. Flat one-line number no fit 2D ground.

2D-RoPE: split head-dim in HALF — spin $[0:d/2]$ by row, $[d/2:d]$ by col — now score feel true gap $(\Delta r,\Delta c)$, work any size. M-RoPE (Multimodal RoPE): moving-picture add time, split head-dim in THIRD; token $(t,r,c)$ spin $[0:d/3]$ by time, $[d/3:2d/3]$ by row, $[2d/3:d]$ by col. Edge: still picture → $m_t=0$; word token → $m_t=m_r=m_c=$ idx. Now score feel $(\Delta t,\Delta r,\Delta c)$ — brain reason how thing move frame to frame. Ugg warn: M-RoPE split head-dim in thirds, NOT three rope stacked. Qwen2-VL also swap flat rock for 2-layer MLP, use ViT 600M, Qwen2 7B/72B brain; the 72B match GPT-4o.

Gemma 4 — weave, no stitch

Qwen2-VL still train eye and mouth in separate place, then stitch — connector is a narrow squeeze-bridge between two spirit-place made for different goal (flat rock only spin+stretch). Gemma 4 go native multimodal: train together from scratch, picture-patch and word-token share the SAME weight-rock from early layer, no connector at all. Lecture say: stitched → woven. Ugg warn: Gemma 4 have NO connector, not even MLP.

Chain to remember: PaliGemma (linear, 1D-RoPE, fixed) → Qwen2-VL (MLP, M-RoPE, dynamic) → Gemma 4 (woven, native).

When brain must MOVE — VLA

VLM describe cup fine but no lift cup. VLA (Vision-Language-Action) — robot driven by brain. Big idea: action is just another token. Chop each smooth move into $B=256$ bin; 7 arm-joint × 256 = 1792 new token (OpenVLA reuse the 256 least-used word-token, so shelf no grow).

$+0.5$ land in bin middle; worst slop is half-bin $(a_{\text{max}}-a_{\text{min}})/(2B)$ (1-meter arm, 256 bin → ~2mm, fine for grab, not for cut-flesh). OpenVLA: two eye-grinder DinoV2 (good for where-hand-go) + SigLIP (good for follow-word), MLP glue them, LLaMA-2 7B brain, action de-tokeniser at end, 970k robot-walk trained. Train from ready VLM with joint pain $L=L_{\text{LM}}+\lambda L_{\text{action}}$ — keep word-skill AND learn to move. Full hunt-arc: perceive → align → reason → generate → act.

Ugg remember

• Three rock: Vision Encoder → Connector → LLM. Glue picture+word token, plain self-attention do the mixing — no cross-attention beast.
• Connector = the ONLY fresh-random piece in stitched VLM; it lonely stretch 1152 → 2048 to line up two already-trained spirit-place.
• SigLIP = pair-by-pair sigmoid yes/no (scale to any batch); CLIP = softmax over whole row. Not the same thing.
• Prefix-LM: picture+prompt see both-way, answer see left-only, pain on answer token only. Box come out as loc-token, never float.
• **M-RoPE = $(t,r,c)$ triple, head-dim split in thirds** (still-pic $t=0$, word all=idx). VLA = VLM + action de-tokeniser, 7-DOF × 256 bin. Gemma 4 = woven, no connector.