Box no tell Ugg what body do

Box round person say "person here." Box no say person dance, fight, lift mug, fall. The doing live in the BODY — where arm, leg, head sit. So Ugg do pose estimation: look at picture, guess where all body joints and limbs are — then read the dance, capture motion, animate cave-avatar. First big fire sold to tribe: Kinect (Xbox, 2010) read body from depth-picture with random forests.

Four ways Ugg draw a body

Small detail to big:

Skeleton (keypoints) — fixed list of $K$ dots, each a spot $(x,y)$ (or $(x,y,c)$ with confidence-grunt). COCO use 17 dots (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles). Output $K\times 2$ numbers. Simplest, most common.

DensePose — not just dots: paint EVERY body-pixel with where it live on one standard body-skin (wrap flat map round a 3D body). Good for try-clothes magic.

Body mesh (SMPL) — full 3D shell. SMPL = template of $6890$ stone-dots (vertices) + two knobs: shape $\beta \in {\mathbb{R}}^{10}$ (tall/short, wide/thin) and pose $\theta \in {\mathbb{R}}^{72}$ ($24$ joints $\times$ $3$ twist-numbers). Give $(\beta ,\theta )$, it build the exact posed body — same every time, smooth to nudge.

Foundation model (Sapiens) — Meta 2024. One giant eye-brain (ViT backbone) fed 300M+ people-pictures, then small heads read pose, cut-out, depth, and surface-slope all at once.

Why "just guess the numbers" break

Lazy way: picture $\to$ CNN $\to$ one 2048 feature-stick $\to$ line-layer $\to$ $K\times 2=34$ numbers, train L2 on $(x,y)$. Ugg warn: this BREAK, four reasons:

1. L2 in pixel-land brutal — 5-pixel miss on tiny picture punished same as on huge picture. Bumpy loss-ground. 2. No spatial thinking — CNN squash whole picture to one stick first; the 2D shape smashed, but pose IS a place-problem. 3. No maybe — one point out; arm could be front OR back, so model average the two $\to$ nonsense middle. 4. No hide-handle — joint behind rock forces a made-up lie.

Fix: do not guess numbers — draw heatmaps.

The heatmap fire

For each joint draw a heatmap $H_k$ where $H_k(i,j)$ = chance joint $k$ sit at pixel $(i,j)$. Truth-map = soft Gaussian glow round the real spot:

$G_k(i,j)=\exp \left(-\frac{(i-x_k{)}^2+(j-y_k{)}^2}{2{\sigma }^2}\right)$

Hot at true joint, fade as you walk away; $\sigma$ = glow width. Loss match Ugg map to glow pixel by pixel:

$L_{\text{pose}}={\sum }_k\Vert H_k-G_k{\Vert }_2^2$

Read joint = brightest pixel: ${\hat{p}}_k=\arg {\max }_{i,j}{H}_k(i,j)$. Argmax land on whole pixels only — sharpen with parabola-fit on top pixel + 2 neighbours (peak at $x^{\ast }=-b/2a$) or smooth softargmax for sub-pixel. Now Ugg keep the 2D shape, show many-peak maybe, get gradient everywhere, and hidden joint just glow dim (low confidence). This flip — numbers to dense map — made pose work.

Ugg sharpen map over many fires (CPM)

Convolutional Pose Machine (CPM) refine belief-maps over $T$ stages. Stage 1: picture $\to$ CNN $\to$ map. Stage $t$: (picture-features + old map) $\to$ CNN $\to$ better map. Each later stage see wider (bigger receptive field), so it use far-apart logic: "head here $\to$ neck just below $\to$ shoulders likely there." Feed each stage the truth-glow too — intermediate supervision — so gradient no vanish down the deep chain. $T\approx 6$ typical.

Ugg score the hunt (PCK)

PCK (Percentage of Correct Keypoints) — joint "correct" if close to truth:

$\text{correct}⟺\Vert {\hat{p}}_i-p_i\Vert \le \alpha \cdot d_{\text{ref}}$

NORMALISE by body size, else 5-pixel miss on giant person vs far-tiny person mean different thing. PCKh@0.5: threshold $=0.5\times$ head-bone length (MPII). PCK@0.2: $0.2\times$ torso width (FLIC). Higher better, near 100% is target. Head-bone chosen because torso squash short when person sit; head stay steady.

Many people, two hunt-styles

Troubles: unknown how many (1 or 50), people tangle and hide each other, and runtime should not blow up with the crowd.

OpenPose (bottom-up) — picture $\to$ CNN $\to$ two branches. Branch 1: $K$ keypoint heatmaps, ALL joints of ALL people at once (a soup: 7 shoulders, 6 elbows, 9 wrists...). Branch 2: Part Affinity Fields (PAF) — per limb-type a 2D arrow-field; along the limb each pixel hold a unit arrow pointing down the limb, zero elsewhere. To join joint $A$ to $B$, walk the line and add up arrow-agree:

$\text{Score}(A,B)={\int }_0^1{L}_c(p(u))\cdot {\hat{v}}_{AB}du,p(u)=(1-u)A+uB$

Big sum = real limb links them. Then Hungarian matching pair joints into clean skeletons. Channels out $=K+2L$ (K maps + 2 per limb): 18 joints + 19 limbs $\to$ $18+38=56$. Runtime near-flat with crowd — joint-finding is fixed cost, only matching grow.

Mask R-CNN keypoints (top-down) — first find each person box (Faster R-CNN), then run single-person pose on each crop via a keypoint-head giving a $28\times 28$ heatmap per joint. Runtime grow with people, $O(P)$.

Trade-off: top-down = sharper per person but brittle (miss box = miss pose), $O(P)$ $\to$ good for few-people precision (golf swing on a launch monitor). Bottom-up = looser per person, robust to misses, near-flat runtime $\to$ good for crowd real-time (30 fps on phone).

Flat picture to fat 3D (SMPL + HMR)

SMPL turn $(\beta ,\theta )$ into a mesh by linear blend skinning — each stone-dot pulled by nearby joints with learned weights:

$M(\beta ,\theta )=W(T_P(\beta ,\theta ),J(\beta ),\theta ,\mathcal{W})$

Human Mesh Recovery (HMR) — CNN (ResNet-50) read one flat photo $\to$ guess $(\beta ,\theta ,\text{camera})$ $\to$ SMPL build mesh $\to$ throw 3D joints back to flat and match truth dots:

$L_{2D}={\sum }_k\Vert \Pi (\text{SMPL}(\beta ,\theta {)}_k)-p_k{\Vert }^2$

Extra adversary-judge grunt "real human pose, or impossible?" keep the body believable.

Ugg warn: no do this!

• OpenPose channels $=K+2L$, NOT $2K$. 18 joints + 19 limbs = 56, not 36.
• Always divide PCK by body size (head-bone or torso), never raw pixels.
• $\theta =72$ is pose, $\beta =10$ is shape — no swap them.
• Top-down NOT always sharper — in crowd with hide, bottom-up win.
• PAF score is an INTEGRAL (average many points on the line), not one dot-product.
• Argmax on a noisy map can lie — use a confidence threshold.
• HMR: SMPL live in 3D metres, camera project to 2D pixels — no mix the frames.

Ugg remember

• Box no tell the doing; pose = joints + limbs. Four draws, small to big: skeleton (K dots) $\to$ DensePose $\to$ SMPL 3D mesh $\to$ Sapiens foundation.
• Guess-numbers BREAK (bad loss, no space-think, no maybe, no hide-handle). Fix = heatmaps: Gaussian glow target, pixel-MSE loss, argmax read.
• CPM refine belief-maps over stages, wider view each stage, intermediate supervision stop the vanishing gradient.
• OpenPose bottom-up: heatmaps + PAF line-integral + Hungarian, $K+2L$ channels, flat runtime. Top-down detect-then-pose: sharper, brittle, $O(P)$.
• SMPL: $\beta =10$, $\theta =72$, $6890$ vertices via linear blend skinning. HMR regress them from one photo with a 2D reprojection loss.