Ugg paint every rock, not just box round mammoth

Old hunt-way draws a box round mammoth — but box no tell which dots are mammoth, which are grass behind. Dense prediction instead gives a name or number to EVERY pixel (tiny colored dot). Paint the whole wall, dot by dot.

Five ways, easy to hard. Semantic segmentation — every dot gets its kind-name; all mammoth-dots one color, no telling Mammoth-1 from Mammoth-2. Instance segmentation — kind-name AND which-one number, only for countable beasts. Panoptic segmentation — name EVERYTHING, plus which-one number for countable beasts; most complete. Depth estimation — every dot gets a how-far distance, not a name. Multi-task dense prediction — one shared brain-rock, many hands: depth + surface-direction + names at once.

Urge to group: Ugg eye see GROUPS, not lone dots. Big trap — is LOOK-same the same as MEAN-same? No! Brown and white mammoth look different but mean same; black cat on black rock look same but mean different. The fight is teach machine MEAN-grouping, not LOOK-grouping.

Things vs Stuff (exam rock): Things are countable beasts with shape — person, cart, mammoth — count Cat-1, Cat-2. Stuff is shapeless spread known by texture — sky, river, mud — no count. Ugg warn: NO make "sky-1, sky-2"! Old pre-fire-brain ways, just name-drop: region-grow, edge-find, motion-group, classify-by-property. Fire-brain took over.

FCN — first fire-brain painter

FCN (Fully Convolutional Network) (Long, 2015). Old classify-brain ends in a fully-connected layer that needs fixed picture-size and spits ONE name. FCN swaps it for a **$1\times 1$ convolution** — a one-dot stamp that spits a whole MAP of names, one per spot. Path: picture into squeeze-blocks (small feature-map), then $1\times 1$ stamp, then upsample (blow back to big), then name every dot. Four rocks: squeeze-part (encoder) catches MEANING; blow-up-part (decoder) returns to dot-space; the middle is small so edges come fuzzy; takes ANY picture size (no FC to lock it). FCN-32s blows up straight from the deepest layer (coarse); FCN-16s and FCN-8s add skip lines from earlier bigger maps, summed in, for sharper edges.

Transposed conv — learnable blow-up

Transposed convolution is learnable blow-up. Normal stamp: big in, small out (stride 2 = jump 2 in input per 1 out). Transposed flips it: stride 2 = jump 2 in the OUTPUT per 1 input, so output GROWS. Each input-dot's value weights a filter-copy pressed at the output; overlapping presses add. In matrix words, normal is $y=Wx$ and transposed is $y=W^{\top }x$ — same rocks, flipped direction. Ugg warn: NO call it "deconvolution"! That means undo-a-convolution to recover the old signal; transposed conv undoes nothing, just learns a good blow-up.

Near-vs-far fight — why U-Net born

Global context (whole beast) is needed to KNOW "this is mammoth." Local context (fine dots) is needed to know WHERE whisker ends and grass begins. Plain FCN squeezes so hard the middle kills local detail — blurry edge. U-Net (Ronneberger, 2015): when blowing up, concatenate (stick side-by-side) the matching same-size encoder map before the next stamp. Now the decoder holds BOTH deep-meaning and saved sharp-local (kept before the squeeze killed it). Shape like a "U." Edges sharp again. Born for cell-pictures; now the noise-cleaner inside Stable Diffusion. Ugg shortcut: U-Net skip = CONCAT (stick beside); ResNet skip = ADD (pile on top) — no mix up!

Dilated conv and 1x1 conv

Dilated (atrous) convolution — put gaps of (rate − 1) between stamp-teeth. See wider without more rocks and without shrinking. DeepLab stacks these; ASPP runs parallel branches at rates 6, 12, 18 for a many-size view. Ugg warn: too-big gaps make gridding artifacts (checkerboard, near dots never compared) — fix with co-prime rates {1, 2, 5}, not {2, 4, 8}. The **$1\times 1$ convolution** is a tiny brain at each spot alone — mix channels only, no neighbor. Three jobs: shrink channels (1024→256), nonlinear channel-mix with ReLU, and per-dot classifier (K channels = K names per dot).

Mask R-CNN and RoI Align

Mask R-CNN (He, 2017), go-to for instance painting: Faster R-CNN plus a THIRD head — a little FCN spitting a $28\times 28$ yes/no mask per box per class, trained with per-dot binary cross-entropy. One mask PER class (80 for COCO) so the mask-head need not also pick the name — cleaner. Loss adds three:

name-loss plus box-loss plus mask-loss. RoI Align — small change, huge win. Old RoI Pool ROUNDS box corners to whole pixels — deadly for masks. RoI Align uses bilinear blend at exact fraction spots, 4 sample points per bin, NO rounding — masks snap to real edges. PointRend goes further: edges hard, middles easy, so predict a coarse mask, find unsure dots (probability near 0.5, likely a boundary), run a point-brain MLP with high-res feature only THERE, repeat.

Depth — how far each dot

Monocular depth — guess far-ness from ONE picture. Geometry alone cannot (small-near looks like big-far), but the brain learns priors — usual sizes, scene shape, perspective. MiDaS — no guess metres; guess relative depth ("bench closer than tree"). Easier, and relative signals are everywhere (stereo, 3D films, web video). Loss is scale-and-shift-invariant:

first slide-and-stretch the guess by best $(s,t)$ to fit the target, THEN measure leftover error — so one model eats many mismatched datasets. Ugg warn: MiDaS depth is RELATIVE — NO read as metres! ZoeDepth: MiDaS relative-pretrain, then fine-tune separate metric heads (KITTI outdoor, NYU indoor) — guess relative (generalizes), convert to metres, zero-shot. Turn a depth-map into a 3D point cloud with the pinhole model: each dot $(u,v,d)$ becomes

a full 3D scene from one flat photo. Depth, normals, and segmentation can share one encoder with task-heads (Eigen & Fergus, 2015) — seed of foundation vision models.

Losses and metrics

Cross-entropy per dot is the default, but trap: with 95% sky the brain says "sky everywhere," low loss, useless. Balanced CE reweights by rareness. Dice loss chases overlap directly, good when foreground is tiny (medical):

The star loss is Focal Loss (RetinaNet, 2017):

The crush-factor $(1-p_t{)}^{\gamma }$ goes near 0 when the brain is already right ($p_t=0.9\to 0.01$, 100x smaller) but stays near 1 when it is wrong — so the gradient is ruled by what the brain gets WRONG. Set $\gamma =0$ and plain CE comes back.

Ugg warn: pixel accuracy is a trap — 95% background scores 0.95 while being garbage. Use mIoU, the mean overlap-over-union per class:

each class counts equal, so a lazy predictor scores only ~0.5. And Dice:

same ranking as IoU, kinder to small beasts. Ugg warn: Dice's bottom is the SUM $|A|+|B|$, NOT the union — the only difference from IoU.

Ugg remember

• Dense prediction names EVERY dot. Things are countable (have instances), Stuff shapeless (none), Panoptic does both.
• FCN swaps FC for a $1\times 1$ conv; encoder = meaning, decoder = transposed conv blow-up ($y=W^{\top }x$, NOT deconvolution); the bottleneck gives blurry edges.
• U-Net CONCAT skips restore sharp edges (ResNet skips ADD). Mask R-CNN = Faster R-CNN + $28\times 28$ per-class mask head + RoI Align (bilinear, no rounding); PointRend refines unsure boundary dots.
• MiDaS = relative depth (scale-and-shift-invariant loss); ZoeDepth adds metric heads. Back-project $(u,v,d)$ to 3D with the pinhole model.
• Focal Loss factor $(1-p_t{)}^{\gamma }$ crushes easy dots so hard ones dominate; use mIoU not pixel accuracy; Dice bottom is SUM not union.