Ugg teach flat rock how to see

Long ago Ugg try teach big FLAT net to see. Picture $224\times 224$, three color, all dot wired to 1000 answer = about $150$ million shiny stone for ONE layer. Too heavy to carry! And flat net dumb: cat in top-left corner and cat in bottom-right corner share NO evidence — two different beast to it. Bad hunt.

Why stamp-rock beat flat net

Convolution — Ugg take small stamp-rock (say $3\times 3$), slide over WHOLE picture, press-press, make new picture (feature map). Same stamp every spot. This called weight sharing — one stamp, all places. Two magic fall out:

Few stone — stamp no care how big picture, so cut stone about $1000\times$ vs flat net. And translation equivariance: $f(T_v(x))=T_v(f(x))$. Mean: Ugg move mammoth in picture, its mark move the SAME way in feature map.

Stack many stamp-layer. Low layer see edge. Next see texture. Next see part. Top see whole beast. Simple stack grow into deep hunter.

Numbers tribe must carve in stone

Conv params — how much stone one layer eat:

$F\cdot (C_{\text{in}}\cdot K^2+1)$

$F$ = stamp count, $K\times K$ = stamp size, $C_{\text{in}}$ = color-in, the $+1$ = bias. Ugg WARN: this NOT depend on picture width or height — only stamp and color!

Output size — how big new picture come out:

$\lfloor (W+2P-K)/S\rfloor +1$

$P$ = padding (edge-stuffing), $S$ = stride (jump size). Use FLOOR rock — round DOWN, no round up.

Same padding (output size = input size, odd stamp): $P=(K-1)/2$. Stamp 3 → pad 1, stamp 5 → pad 2, stamp 7 → pad 3. Valid padding ($P=0$) shrink picture by $K-1$.

Receptive field — the patch of first picture that touch one output dot. Stack $L$ stamp of $3\times 3$ at stride 1: $\text{RF}=2L+1$. Every stride-2 squish DOUBLE the reach. Deep tribe see whole mammoth with tiny stamp — no giant stamp needed.

Two small rock beat one big rock

Two $3\times 3$ stack = same reach as one $5\times 5$ (both reach 5). But stone: $2\cdot 9\cdot C^2=18C^2$ small-way vs $25C^2$ big-way — save ~28% AND get free extra bend (ReLU) in the middle. Three $3\times 3$ = $7\times 7$ reach, $27C^2$ vs $49C^2$. This why VGG tribe use ONLY $3\times 3$.

Tiny 1×1 rock — four job

1×1 convolution — stamp size one. No mix space, only mix color. Four job:(1) bottleneck — drop color cheap BEFORE costly $3\times 3$/$5\times 5$.(2) inject bend (little per-dot brain).(3) mix color across channel.(4) cheap — only $C_{\text{in}}\cdot C_{\text{out}}$ stone. Ugg WARN: in Inception, 1×1 go BEFORE the big stamp, not after!

Squish-rock — no stone at all

Pooling shrink picture. Max-pool keep the biggest — robust to small wiggle, favorite for naming. Avg-pool take the middle. Global Average Pooling (GAP) squash whole $H\times W$ down to one dot, replace the giant flat tail (ResNet do this; VGG carry heavy flat tail). Ugg WARN: pool have ZERO learn-stone — it only shrink work (FLOPs), no weights! Backward: max send gradient only to the biggest spot; avg spread equal ($÷k^2$). Other WARN: GAP throw away WHERE — fine for naming, deadly for pixel-map (segmentation) that need spatial.

Spread-out stamp — see wide, stay cheap

Dilated (atrous) convolution — poke $d-1$ hole between stamp teeth. Effective stamp = $K+(K-1)(d-1)$. So $3\times 3$ with $d=2$ reach like $5\times 5$ but use only 9 stone. Good for segmentation (DeepLab: keep resolution, grow reach) and cave-sound (WaveNet). Ugg WARN: big $d$ make GRIDDING — neighbor dot sample different pixel, leave gaps. Fix: stack dilation like $\{1,2,5\}$, not $\{2,4,8\}$.

Split the stamp for phone-tribe

Depthwise-separable convolution — cut normal stamp in two: depthwise ($K\times K$ on each color ALONE) then pointwise ($1\times 1$ to mix color). Stone = $K^2{C}_{\text{in}}+C_{\text{in}}{C}_{\text{out}}$, ratio $\approx 1/C_{\text{out}}+1/K^2$ — about $8$–$9\times$ cheaper at $K=3$. This the MobileNet magic. Ugg WARN: it is depthwise + POINTWISE, NOT depthwise + $3\times 3$!

Wash the numbers clean

Batch Normalisation — wash each color across the $(N,H,W)$ axes:

$\hat{x}=(x-{\mu }_B)/\sqrt{{\sigma }_B^2+ϵ};y=\gamma \hat{x}+\beta$

Mean: subtract batch-middle, divide by spread, then learn scale $\gamma$ and shift $\beta$ — one pair PER COLOR (length $C$), NOT per spatial spot (common trap!). Effect: train fast, allow big step, less fussy on start-weights. Ugg WARN: at hunt-time (inference) use the RUNNING middle/spread from training, NOT this batch! And batch-of-1 no work — use GroupNorm or LayerNorm (LayerNorm also feed the ViT tribe).

Move-same vs no-care

Equivariance: $f(T_v(x))=T_v(f(x))$ — output MOVE when input move; feature maps do this. Invariance: $f(T_v(x))=f(x)$ — output NO change when input move; after GAP + flat + softmax, the final answer do this. NOT the same thing! Ugg WARN: CNN handle shift fine, but NOT turn — kernels are LEARNED, not turn-symmetric. Fix with rotation-augment or G-CNN.

The great hunt lineage

• LeNet (LeCun, 1989/98) — first stamp-net to work. ~60k stone, read hand digit.
• AlexNet (2012) — start the fire! 60M stone, 8 layer, 2 GPU. Bring ReLU (fast, but WARN: dead-ReLU if pre-activation always negative — use Leaky/ELU/GELU), dropout ($p=0.5$), augment. Won ILSVRC-2012 big.
• VGG (2014) — all $3\times 3$, deep. VGG-19: 143.7M stone (mostly flat tail), top-1 74.2%.
• Inception / GoogLeNet (2014) — parallel branch of many stamp size, 1×1 bottleneck BEFORE costly stamp. v3: 27.2M, 77.3%.
• ResNet (2015) — the big one. $y=F(x)+x$ skip. Fix deep-net ROT (plain net past ~20 layer get WORSE — optimize fail, not overfit). Gradient $\partial y/\partial x=\partial F/\partial x+I$ — the $+I$ mean gradient never die. ResNet-50: 25.6M, 76.1% — the workhorse. WARN: skip ADD to $F(x)$, not replace it!
• DenseNet (2016) — $x_l=H_l([x_0,\dots ,x_{l-1}])$ — glue ALL past outputs (concatenate, NOT add).
• SENet (2017) — channel attention: GAP → little brain → sigmoid → scale each color. +1% top-1, near-free.
• MobileNet (2017) — depthwise-separable all over. 4.2M, ~70%, for phone. (v2 add inverted residual — that is v2, NOT v1!)
• EfficientNet (2019) — compound scaling: grow depth, width, resolution TOGETHER by one dial: $d={\alpha }^{\varphi },w={\beta }^{\varphi },r={\gamma }^{\varphi }$ with $\alpha {\beta }^2{\gamma }^2\approx 2$. B0: 5.3M, 77.7%.

Walk backward through the layers

Quick backward-grunts: conv backward = conv with FLIPPED stamp (for $\partial L/\partial x$), and input-cross-correlate-gradient for $\partial L/\partial W$. Max-pool send gradient to the argmax spot ONLY. Avg-pool spread it $÷k^2$. ReLU is a gate — pass gradient if forward was $>0$, else 0. Skip send gradient through $+I$ → no vanish.

Moving picture and cave sound

Video: C3D use 3D stamp $K_t\times K_h\times K_w$, trained from scratch, costly. I3D INFLATE a 2D ImageNet stamp along time and divide by $K_t$ (so a still-picture-repeated gives the SAME mark → free ImageNet pretrain). Ugg WARN: I3D inflate, C3D from scratch — no mix up! SlowFast = slow path (low fps, many channel = meaning) + fast path (high fps, few channel = motion). Sound: 1D CNN on raw wave (WaveNet dilated), or 2D CNN on spectrogram picture (frequency has spatial structure).

Ugg remember

• **Conv stone $=F(C_{\text{in}}{K}^2+1)$, NO care about $H,W$** — weight sharing is the whole magic. Pool has ZERO stone.
• **Output $=\lfloor (W+2P-K)/S\rfloor +1$**, same-pad $P=(K-1)/2$, reach of $L$ stacked $3\times 3$ is $2L+1$. Two $3\times 3$ beat one $5\times 5$ (fewer stone + extra bend).
• **ResNet skip $y=F(x)+x$** gives gradient $\partial F/\partial x+I$ → no vanish → deep net trains. ADD, not replace.
• **Depthwise-separable ~$9\times$ cheaper**; 1×1 for bottleneck; dilated widens reach cheap; BN washes per channel and uses RUNNING stats at inference.
• Equivariance (feature map) ≠ invariance (after pool), and CNN sees shift but NOT rotation.