Computer Vision

Detection asks two questions per object: 'where' (a bounding box) and 'what' (a class). The history of the field is a story of moving the expensive parts of that pipeline — region proposals, per-region computation, ranking — onto the GPU and into a single shared backbone.

Detection labels regions; dense prediction labels pixels. The architectural challenge is restoring spatial resolution lost during downsampling — usually with transposed convolutions, skip connections, or dilated convolutions.

Predicting K joint locations as a 17×2 vector throws away spatial structure. Almost every modern pose method predicts a heatmap per joint instead and reads the joint location off the argmax — same idea as dense prediction in segmentation, with parabola-fit interpolation for sub-pixel accuracy.

Point clouds break CNNs because CNNs assume a regular grid. The two big ideas are (1) voxelize and use 3D conv (memory-heavy, low resolution), or (2) build symmetric, permutation-invariant operators that act directly on the unordered point set.

NeRF encodes a scene as a neural network — query (x, y, z, view) and get density + colour. 3DGS keeps the rendering equation but replaces the neural query with a sorted bag of explicit 3D Gaussians, getting real-time rendering at NeRF-comparable quality.

Attention is a soft, learnable dictionary lookup. The decoder asks a question (Q); each position in the source provides a key (K); the value (V) at the matching position is returned. Multi-head attention runs many such lookups in parallel, each learning a different relationship.

Feeding 50 K raw pixels into self-attention is infeasible (O(n²)). ViT solves this by treating non-overlapping 16×16 patches as 'words' — for a 224² image that's 14×14 = 196 tokens, well within Transformer reach.

Self-supervised learning invents the supervision signal: 'this random crop and that random crop of the same image should look similar; everything else in the batch should look different.' That's the entirety of SimCLR. Variants then attack the dependency on huge batches.

DINO trains a student to match an EMA teacher's output distribution. MAE masks 75% of patches and reconstructs pixels. JEPA predicts target REPRESENTATIONS instead of pixels. All three avoid negatives entirely.

Original Transformer (2017) → modern LLM (2024+) is roughly seven independent upgrades. Each fixes a specific failure mode of the original — training stability, attention sharpness, position generalisation, or inference efficiency.

Modern VLMs stitch a pretrained vision encoder, a small connector, and a pretrained LLM together — three pillars. The connector is the only randomly initialised piece. Then mask attention so the image+prompt are 'free context' (bidirectional) and only the answer is generated causally.

Video ≠ images × T. The thing the model needs to capture is the TEMPORAL pattern — 'sit down' and 'stand up' are the same image set in reverse order. The architecture choices are about how to consume the time dimension efficiently.