Time come alive on cave wall

Ugg spend thirteen moons staring at ONE frozen rock-picture. But world no hold still! Man walking not same as man standing. Time — this the missing footprint-direction, and every eye-spirit Ugg learn before is blind to it.

A video is many rock-pictures stacked in a row, flip fast, picture move. But careful — video not JUST a pile of pictures, same like a sentence not just a pile of letters. Video understanding ask: how teach eye-spirit to see MOTION, ORDER, and hunter-intent? Two hunts, learn separate: (1) the TASKS and picture-piles, (2) the ARCHITECTURES that kill them.

Picture-piles (datasets) by campfire age

Old-old: KTH (2003, six action, plain wall) and Weizmann (2005, action as shadow-shape). Tiny toy hunts, before big-brain time.

Middle: HMDB-51 (51 action) and UCF-101 (101 action) — clips from far-tribe YouTube. Something-Something (174 action) is sneaky — it test fine hand-play like "put something on top of something," so spirit must understand rock-physics, not just remember the room.

Big: Kinetics (Carreira & Zisserman, 2017) — hundreds-of-thousand clips, 400/600/700 action. When tribe ask "is there an ImageNet for video?" Ugg grunt: YES, Kinetics. Nearly every modern video-spirit pretrain (warm-up) on it.

Beyond-name: AVA — Atomic Visual Actions. Not just name the action — also draw a box WHERE and mark WHEN.

Six hunts (tasks) to know

• Action Classification — one name for whole clip. "This dancing?"
• Temporal Action Localization — WHEN in a long messy video? "Dance: 12.4s–18.7s."
• Spatio-temporal Action Localization — WHEN and WHERE, box plus time. AVA is king.
• Video Captioning — say one sentence about the clip.
• Text-to-Video Retrieval — hunter say words, spirit find matching clip. Backward of captioning.
• Video Situation Recognition — who is doer, who is done-to, what tool.

Ugg warn — the scene-cheat! An eye-spirit can score 80% on Kinetics by looking ONLY at the cave-room (kitchen → guess cooking), never look at the hunter. That why Something-Something is harder — its backgrounds boring on purpose.

Architecture tribes

Tribe 1 — Frame-by-frame 2D CNN. Naive baseline. Run a flat eye-rock (ResNet) on each picture alone, average the votes. Work okay — many action clear from one frame. BUT blind to motion: walk-forward and walk-backward look the SAME. Bad.

Tribe 2 — 3D Convolution (Conv3D). A flat stamp-rock slide $K_H\times K_W$ over one picture. A fat CUBE-stamp slide $K_T\times K_H\times K_W$ over the picture-stack — catch neighbors in space AND time. Remember the shapes:

That is Batch, Channels, Time, Height, Width — the Time $T$ is the NEW thing over the flat eye. C3D was the first big one.

Tribe 3 — I3D (Inflated 3D ConvNet). Clever! Train a cube-spirit from nothing is hard — video-piles small. But flat 2D eye-rocks already fat and wise from ImageNet. Inflation trick: take a flat $K\times K$ stamp, copy it $K_T$ times along time to make a cube, THEN divide by $K_T$:

Why divide? On a BORING video (same picture every frame), the $K_T$ copies add up $K_T$-times too loud; divide by $K_T$ cancel it, so the cube-stamp make the SAME loudness as the old flat stamp. Warm-start smart, not random. Then fine-tune on Kinetics. I3D + Kinetics = the classic video recipe.

Tribe 4 — Two-Stream. Split WHAT from HOW. Two eye-rocks side by side. Spatial stream: one RGB frame → appearance. Temporal stream: a stack of optical flow → motion.

Optical flow — a dense field of little arrows $(u,v)$ saying how each speck moved between frame $t$ and frame $t+1$. The motion-signal the flat eye miss. Ugg warn: temporal input has $2L$ channels ($u$ AND $v$, times $L$ flows), NOT $L$! Then late fusion — average both votes.

Tribe 5 — CNN + RNN (LRCN). Eye-rock squeeze each frame to a feature, then feed the sequence to an LSTM memory-snake that reason over time. Best for variable-length outputs, like captions.

Tribe 6 — SlowFast. Two scout-paths. Slow path: few frames (4) but MANY channels → rich "what animal" meaning. Fast path: many frames (32) but FEW channels → motion. Lateral connections let Fast whisper motion into Slow at each stage. Motion is low-dimension, so Fast need few channels — that why Fast is ~8× cheaper even seeing 8× more frames.

Tribe 7 — Video Transformers. ViViT cut the video into little jelly-cubes $t\times 16\times 16$ (tubelet embedding), each cube become a talking-stone (token), then run a transformer. TimeSformer test WHERE the stones talk. Joint space-time — every stone talk every stone — cost $\mathcal{O}((TN{)}^2)$, too much! Divided space-time WON: each stone first talk its same-spot twins across TIME, then talk same-frame neighbors across SPACE, alternating. Near-linear cost. Chant it: "divided — temporal then spatial."

CNN tribe (I3D, SlowFast): local eye, strong built-in bias, better on small data. Transformer tribe (ViViT, TimeSformer): global talking-stones, less bias, better on huge data.

The frame-count quiz

Cube-eater want 64 frames; your clips run 48–240. Shorter clip: pad it (loop, repeat last frame, or zero) or interpolate up to 64. Longer clip: subsample (every 4th frame), sliding window (many 64-frame chunks, average the softmaxes), or multi-crop average. Ugg warn: no say "just resize" — that is SPACE. The quiz ask about the TIME axis!

Ugg remember

• Time is the missing dimension. Video not just a picture-pile; motion and order carry the meaning.
• Kinetics = ImageNet for video (pretrain here). AVA = box + time localization.
• **I3D = inflate the flat stamp into a cube, divide by $K_T$.** No mix up: C3D from scratch, SlowFast = two paths at different fps.
• **Two-Stream temporal stream = $2L$ channels** ($u,v$ per flow), joined by late fusion.
• TimeSformer winner = divided attention (temporal then spatial). CNN win on small data, Transformer win on big data.