Intuition

A convolution layer is a stack of learned filters slid across an image — a CNN is a hierarchy of these, with each layer composing simpler features (edges → texture → parts → objects). The reason CNNs work in vision: WEIGHT SHARING (one filter, all spatial locations) gives translation equivariance (a shifted input produces a shifted output) and slashes parameters by ~1000× vs a dense network. The decade of CNN architecture research (2012–2020) was about how to go deeper, wider, or more efficient without breaking trainability.

Explanation

Convolution layer parameters. A conv layer with $F$ filters, kernel $K\times K$, input channels $C_{\text{in}}$: parameters = $F\cdot (C_{\text{in}}\cdot K^2+1)$ (the +1 is bias). **Independent of $H,W$** — same filter applied at every position. Compare to a fully-connected layer between $H\times W\times C_{\text{in}}$ and $F$ outputs: $H\cdot W\cdot C_{\text{in}}\cdot F$ parameters — typically 1000× more. This is the *parameter efficiency* of CNNs.

Output spatial size. $\lfloor (W+2P-K)/S\rfloor +1$. Same padding with odd $K$ uses $P=(K-1)/2$ → output size equals input. Valid padding ($P=0$) shrinks by $K-1$. Two stacked $3\times 3$ convs (stride 1) have the same RECEPTIVE FIELD as one $5\times 5$ conv but use FEWER parameters ($2\cdot 9\cdot C^2=18C^2$ vs $25C^2$) AND add an extra non-linearity. Three $3\times 3$ = $7\times 7$ RF, $27C^2$ vs $49C^2$.

Receptive field. Set of input pixels that influence a given output. For $L$ stacked $3\times 3$ convs at stride 1: RF = $2L+1$. Doubles at every stride-2 step. Architectures balance RF (need large to see objects), spatial resolution (need fine for dense prediction), and compute.

1×1 convolution — four uses. (1) Channel reduction (bottleneck) — cheap way to drop $C$ before an expensive $3\times 3$ or $5\times 5$. (2) Non-linearity injection — per-pixel MLP between conv layers. (3) Cross-channel mixing without spatial mixing. (4) Cheap — only $C_{\text{in}}\cdot C_{\text{out}}$ params. Used in: Inception (before expensive convs), ResNet bottleneck, MobileNet pointwise, SENet, feature-pyramid laterals.

Pooling. Max-pool routes argmax → ROBUST to translation, dominant in classification. Avg-pool equal-weight average; Global Average Pooling (GAP) before classifier replaces giant FC layers (ResNet, EfficientNet). No parameters — pools only reduce FLOPs, not weights. Backward: max-pool routes gradient to argmax only; avg-pool distributes equally ($÷k^2$).

Dilated (atrous) convolution. Insert $d-1$ zeros between kernel taps. Effective kernel size = $K+(K-1)(d-1)$. A $3\times 3$ with dilation 2 has a $5\times 5$ RF using 9 parameters. Use: semantic segmentation (DeepLab — keep resolution while enlarging RF), WaveNet (1D causal). Drawback: gridding artefact at large dilation rates.

Standard vs depthwise-separable convolution. Standard $K^2\cdot C_{\text{in}}\cdot C_{\text{out}}$ params. Depthwise-separable = depthwise ($K\times K$ per input channel) + pointwise ($1\times 1$). Params = $K^2\cdot C_{\text{in}}+C_{\text{in}}\cdot C_{\text{out}}$. Ratio ≈ $1/C_{\text{out}}+1/K^2$ — about $8$–$9\times$ cheaper for $K=3,C=256$. Used in MobileNet, Xception.

BatchNorm in CNNs. Normalise per channel across the $(N,H,W)$ axes; learnable $\gamma ,\beta$ per channel (length $C$). At inference: use running mean/var, NOT batch stats. Effect: faster training, higher LR, less init-sensitive, slight regularisation. Variants: LayerNorm (across $C,H,W$ per sample — used in ViT), GroupNorm (groups of channels — works at batch=1), InstanceNorm (per-sample per-channel — style transfer).

Equivariance vs invariance. Convolutions are TRANSLATION EQUIVARIANT: $f(T_v(x))=T_v(f(x))$ — shift input, output shifts the same way. After global pooling + FC, the network becomes translation INVARIANT: $f(T_v(x))=f(x)$. CNNs are NOT rotation equivariant — kernels are learned, not symmetric. Fix: data augmentation, or G-CNNs (group-equivariant CNNs).

LeNet-5 (LeCun, 1989/1998). First CNN to win at scale — handwritten digit recognition. Two conv-pool blocks then FC. ~60 k parameters. Trained on USPS / MNIST. The architecture template every later CNN follows: conv → pool → conv → pool → flatten → FC → softmax.

AlexNet (Krizhevsky, Sutskever, Hinton, 2012). Won ILSVRC-2012 by a huge margin (top-5 16% vs 26% next best) and started the deep-learning revolution. 60 M params, 8 layers (5 conv + 3 FC), trained on 2 GPUs. Three innovations: ReLU (much faster than tanh/sigmoid), dropout (in the FC layers, $p=0.5$), data augmentation (random crops + horizontal flips + colour jitter). Top-1 = 56.5%.

VGG (Simonyan & Zisserman, 2014). Stack lots of $3\times 3$ convs. VGG-16: 138 M params. VGG-19: 143.7 M params, 19.6 GFLOPs, top-1 74.2%. Heavy: most parameters live in the FC layers. Lesson: depth + tiny filters > shallow + huge filters.

Inception / GoogLeNet (Szegedy et al., 2014). Inception module = parallel branches at multiple kernel sizes ($1\times 1$, $3\times 3$, $5\times 5$, $3\times 3$ max-pool), concatenated. **$1\times 1$ bottleneck** placed BEFORE $3\times 3$/$5\times 5$ to cut channels — slashes parameters. Inception-v3: 27.2 M params, 5.71 GFLOPs, top-1 77.3%.

ResNet (He et al., 2015) — residual connections. Beyond ~20 layers, plain CNNs DEGRADE (training loss goes up — not just overfitting, optimisation failure). Fix: $y=F(x)+x$. The skip connection lets the network learn the IDENTITY by default (set $F(x)=0$) and only learn corrections. **Gradient: $\partial y/\partial x=\partial F/\partial x+I$** — the '+I' term ensures gradient never vanishes through the skip. Bottleneck block: $1\times 1\to 3\times 3\to 1\times 1$ with skip. ResNet-50: 25.6 M params, 4.09 GFLOPs, top-1 76.1%. The workhorse backbone.

DenseNet (Huang et al., 2016). $x_l=H_l([x_0,x_1,\dots ,x_{l-1}])$ — each layer takes ALL previous layer outputs as input via CONCATENATION (not addition like ResNet). Strong gradient flow, feature reuse. DenseNet-161: 28.7 M params, 7.73 GFLOPs, top-1 77.1%.

SENet (Squeeze-Excitation, Hu et al., 2017). Channel attention module added to existing backbones: GAP across spatial → 2-layer MLP → sigmoid scale → multiply with original channels. Recalibrates which channels matter. +1% top-1 on ResNet, almost free in compute.

MobileNet (Howard et al., 2017). Depthwise-separable convs throughout → $\sim 9\times$ fewer params/FLOPs than standard convs. MobileNet-v1: 4.2 M params, ~70% top-1. v2 adds inverted residuals and linear bottlenecks. Designed for phones / edge devices.

EfficientNet (Tan & Le, 2019). Compound scaling: scale depth $d$, width $w$, and input resolution $r$ together by a single coefficient $\varphi$, with $d={\alpha }^{\varphi },w={\beta }^{\varphi },r={\gamma }^{\varphi }$ where $\alpha \cdot {\beta }^2\cdot {\gamma }^2\approx 2$ (so FLOPs scale as $2^{\varphi }$). EfficientNet-B0: 5.3 M params, 0.39 GFLOPs, top-1 77.7% — the best efficiency on the chart for years.

Backprop through conv layer. Conv backward = conv with FLIPPED kernel for $\partial L/\partial x$; cross-correlation with input for $\partial L/\partial W$. MaxPool backward routes to argmax (others = 0). AvgPool backward distributes equally ($÷k^2$). ReLU backward is a gate (pass if forward $>0$, else 0). Skip backward through $y=F(x)+x$ adds $I$ → no vanishing.

CNN for video. C3D uses 3D convs $K_t\times K_h\times K_w$ over $(T,H,W)$ — expensive ($\times K_t$ factor over 2D), trained from scratch. I3D inflates 2D ImageNet kernels along time and divides by $K_t$ — borrows ImageNet pretraining for free. SlowFast has slow (low-fps, high-channel = semantics) and fast (high-fps, low-channel = motion) pathways with lateral connections. For long videos: sliding window + aggregation.

CNN for audio. 1D CNN on raw waveform — WaveNet uses DILATED 1D convs for long context. 2D CNN on spectrogram (STFT or mel) — treat audio as image; works because frequencies have spatial structure.

Definitions

• Convolution layer — Stack of learnable filters slid across input. Weight-shared across space → translation equivariance. Params $F(C_{\text{in}}{K}^2+1)$, independent of $H,W$.
• Receptive field — Set of input pixels that influence one output unit. Grows with depth (and stride). $L$ stacked $3\times 3$ stride 1 → $\text{RF}=2L+1$.
• Pooling (max / avg / GAP) — Downsample with NO parameters. Max routes argmax (translation-robust). Avg distributes equally. GAP collapses $H\times W\to 1\times 1$, replaces giant FC.
• Same / Valid padding — Same: $P=(K-1)/2$ (odd $K$) → output size equals input. Valid: $P=0$ → output shrinks by $K-1$.
• 1×1 convolution — Per-pixel MLP across channels. Four uses: channel reduction (bottleneck), non-linearity injection, cross-channel mixing, cheap. Used in Inception, ResNet, MobileNet, SENet.
• Dilated (atrous) convolution — Inserts $d-1$ zeros between kernel taps → effective kernel $K+(K-1)(d-1)$. Enlarges RF without parameters or stride. Used in DeepLab, WaveNet. Gridding artefact at large $d$.
• Depthwise-separable convolution — Depthwise ($K\times K$ per channel) + pointwise ($1\times 1$). $\sim 8$–$9\times$ cheaper than standard conv at $K=3$. Foundation of MobileNet, Xception.
• Batch Normalisation (CNN) — Normalise per channel across $(N,H,W)$; learnable $\gamma ,\beta$ per channel. Inference uses running mean/var (not batch). Faster training, less init-sensitive.
• Translation equivariance vs invariance — $f(T_v(x))=T_v(f(x))$ vs $f(T_v(x))=f(x)$. Conv layers are equivariant; after global pool + FC, network is invariant. CNNs are NOT rotation equivariant.
• LeNet — (LeCun, 1989/1998) First successful CNN. ~60k params. Two conv-pool blocks then FC. Template every later CNN follows.
• AlexNet — (Krizhevsky et al., 2012) 60M params, ReLU + Dropout + augmentation, trained on 2 GPUs. Won ILSVRC-2012 by 10% top-5 — started the deep-learning era.
• VGG — (Simonyan & Zisserman, 2014) Only $3\times 3$ convs, deep + simple. VGG-19: 143.7M params, mostly in FC layers. Top-1 74.2%.
• Inception / GoogLeNet — (Szegedy et al., 2014) Parallel branches at multiple kernel sizes with 1×1 bottlenecks placed BEFORE expensive 3×3/5×5. Inception-v3: 27.2M params, top-1 77.3%.
• ResNet — (He et al., 2015) $y=F(x)+x$ residual connection. Gradient $\partial F/\partial x+I$ ⇒ no vanishing. Enables 100+ layers. ResNet-50: 25.6M params, 76.1% top-1 — the workhorse backbone.
• DenseNet — (Huang et al., 2016) $x_l=H_l([x_0,\dots ,x_{l-1}])$ — concatenation (not addition). Strong gradient flow, feature reuse.
• SENet (Squeeze-Excitation) — (Hu et al., 2017) Channel attention: GAP → 2-layer MLP → sigmoid scale → multiply. +1% top-1 at near-zero cost.
• MobileNet — (Howard et al., 2017) Depthwise-separable convs throughout. ~4.2M params, ~70% top-1. Designed for phones / edge.
• EfficientNet — (Tan & Le, 2019) Compound scaling: $d={\alpha }^{\varphi },w={\beta }^{\varphi },r={\gamma }^{\varphi }$, $\alpha {\beta }^2{\gamma }^2\approx 2$. B0: 5.3M params, top-1 77.7%.
• C3D / I3D / SlowFast — Video CNN family. C3D: 3D conv from scratch. I3D: inflate 2D pretrained weights along time. SlowFast: slow (semantics, low fps high channel) + fast (motion, high fps low channel) pathways.

Formulas

• $$\text{Conv output size:}\lfloor (W+2P-K)/S\rfloor +1$$
• $$\text{Conv params:}F\cdot (C_{\text{in}}\cdot K^2+1)\text{(+1 bias)}$$
• $$\text{Same pad (odd K):}P=(K-1)/2$$
• $$\text{Receptive field of }L\text{ stacked 3×3 stride-1 convs:}\text{RF}=2L+1$$
• $$\text{Dilated conv effective kernel:}{K}^{\prime }=K+(K-1)(d-1)$$
• $$\text{Depthwise-separable params:}{K}^2{C}_{\text{in}}+C_{\text{in}}{C}_{\text{out}}(\approx 1/C_{\text{out}}+1/K^2\text{ of standard})$$
• $$\text{BatchNorm:}\hat{x}=(x-{\mu }_B)/\sqrt{{\sigma }_B^2+ϵ};y=\gamma \hat{x}+\beta$$
• $$\text{Residual:}y=F(x)+x;\partial y/\partial x=\partial F/\partial x+I$$
• $$\text{Dense:}{x}_l=H_l([x_0,x_1,\dots ,x_{l-1}])(\text{concatenate})$$
• $$\text{Compound scaling (EfficientNet):}d={\alpha }^{\varphi },w={\beta }^{\varphi },r={\gamma }^{\varphi };\alpha {\beta }^2{\gamma }^2\approx 2$$

Derivations

**Two stacked $3\times 3$ = one $5\times 5$ RF, fewer params.** RF after one $3\times 3$ = 3; after two stacked = $3+(3-1)=5$. Params: two $3\times 3$ at $C$ channels = $2\cdot 9\cdot C^2=18C^2$. One $5\times 5$ = $25C^2$. Saving = $7C^2$ (~28%) AND you get an extra non-linearity between the two. Same logic gives three $3\times 3$ = $7\times 7$ RF, $27C^2$ vs $49C^2$.

Depthwise-separable param ratio. Standard $K^2{C}_{\text{in}}{C}_{\text{out}}$; depthwise-separable $K^2{C}_{\text{in}}+C_{\text{in}}{C}_{\text{out}}$. Ratio = $(K^2{C}_{\text{in}}+C_{\text{in}}{C}_{\text{out}})/(K^2{C}_{\text{in}}{C}_{\text{out}})=1/C_{\text{out}}+1/K^2$. For $K=3,C_{\text{out}}=256$: $1/256+1/9\approx 0.115$ → ~$8.7\times$ cheaper.

Residual gradient no longer vanishes. $y=F(x)+x\Rightarrow \partial y/\partial x=\partial F/\partial x+I$. Even if $\Vert \partial F/\partial x\Vert$ is tiny, the identity term guarantees the gradient norm is at least 1 through the skip. Through $L$ stacked residual blocks the gradient through the skip path is exactly $I$, not $\prod \partial F/\partial x$. This is the formal reason 100-layer networks train at all.

Inflation in I3D. Take a 2D filter $W\in {\mathbb{R}}^{K\times K}$ pretrained on ImageNet. Inflate to 3D as $W_{3D}(t,h,w)=W(h,w)/K_t$ for all $t$. Why divide by $K_t$? A 'boring video' (still image repeated $K_t$ times) should produce the SAME activation as the 2D filter on the image. Without the division, the 3D filter outputs $K_t$× the original.

1×1 conv parameter count. $1\cdot 1\cdot C_{\text{in}}\cdot C_{\text{out}}+C_{\text{out}}$. For $C_{\text{in}}=C_{\text{out}}=256$: $\approx 65k$ parameters, vs a $3\times 3$ at the same widths which is 9× more. That's the bottleneck saving.

Examples

• **Param count for $3\times 3\times 64\to 128$ conv with bias.** $128\cdot (64\cdot 9+1)=128\cdot 577=73,856$.
• Output size. Input $224\times 224$, conv $K=7$, $S=2$, $P=3$: $\lfloor (224+6-7)/2\rfloor +1=\lfloor 223/2\rfloor +1=111+1=112$. So $112\times 112$ — exactly the standard first-layer output of ResNet-50.
• **ViT-B/16 patch count at $224^2$.** $14\times 14=196$ patches; +1 CLS = 197 tokens. (CNN-adjacent but examined.)
• **Receptive field of 5 stacked $3\times 3$ stride-1 convs.** $2\cdot 5+1=11$ pixels. Add a stride-2 pool in the middle and the effective RF on the input doubles.
• MobileNet param saving. Standard $3\times 3\times 256\to 256$: $9\cdot 256\cdot 256=589,824$. Depthwise-separable: $9\cdot 256+256\cdot 256=2,304+65,536=67,840$. Saving $\approx 8.7\times$. ✓
• Two 3×3 vs one 5×5. Two $3\times 3$ with $C=256$: $2\cdot 9\cdot 256^2=1,179,648$. One $5\times 5$: $25\cdot 256^2=1,638,400$. Two-3×3 wins by 28% AND adds an extra ReLU.

Diagrams

• Receptive field growth as a function of layer depth — log-scale plot.
• Inception module: 4 parallel branches with 1×1 bottlenecks before 3×3/5×5, concatenated output.
• ResNet bottleneck block: $1\times 1\to 3\times 3\to 1\times 1$ with skip-add to the input.
• DenseNet dense block: each layer concatenates all previous layer outputs.
• SE block: GAP → FC → ReLU → FC → sigmoid → multiply with input channels.
• MobileNet depthwise-separable: depthwise (per-channel) + pointwise (1×1).
• EfficientNet compound-scaling curve: top-1 accuracy vs FLOPs, EfficientNet on the Pareto front.
• Equivariance vs invariance: shifted input + same shift in feature map (equivariant) vs same scalar output after pooling+FC (invariant).
• I3D inflation: 2D filter $K\times K$ replicated $K_t$ times along time, divided by $K_t$.

Edge cases

• 'Dying ReLU'. Once a neuron's pre-activation is always negative, gradient is 0 forever → neuron stays dead. Fix: Leaky ReLU, ELU, GELU, or higher Kaiming init.
• BN at inference using batch stats is a deployment bug — use running mean/var.
• BN with batch size 1 is meaningless (batch variance undefined). Use GroupNorm or LayerNorm.
• **Dilated conv with high $d$ produces gridding artefacts** — neighbouring outputs sample disjoint input pixels. Mitigation: stack dilations $\{1,2,5\}$ rather than $\{2,4,8\}$ (HDC pattern).
• CNN not rotation equivariant. Kernels are learned; nothing constrains them to be rotation-symmetric. Data augmentation or G-CNNs needed.
• Very deep plain CNN (>20 layers) degrades. Training accuracy drops because gradient/optimisation fails — not overfitting. Residual connections fix.
• Depthwise-separable convs are LESS expressive than standard convs at the same width — they trade expressivity for parameter efficiency. May need to widen to compensate.
• Global Avg Pooling loses spatial structure — useful for classification, fatal for dense prediction (segmentation needs to keep spatial).

Common mistakes

• **Conv params depend on $H,W$.** WRONG — they depend on $F,C_{\text{in}},K$. Weight sharing across spatial locations.
• Pooling has parameters. WRONG — max and avg pool have no learnable params.
• 'BatchNorm normalises across all axes equally'. Wrong — in CNN, BN normalises across $(N,H,W)$ per channel; $\gamma ,\beta$ are PER CHANNEL (length $C$), not per spatial position.
• Inception 1×1 convs are placed AFTER the expensive 3×3/5×5. Wrong — placed BEFORE to reduce channels first (bottleneck).
• ResNet replaces all skip connections with identity. No — the skip is added to a residual function. $y=F(x)+x$, not $y=x$.
• Depthwise-separable = depthwise + 3×3 conv. No — depthwise + POINTWISE (1×1).
• 'CNNs are rotation equivariant'. No — only translation. Rotation needs data aug or G-CNNs.
• Translation EQUIvariance and INvariance are the same. No — equivariance shifts the output; invariance doesn't. CNN feature maps are equivariant; the post-pool classification is invariant.
• 'I3D trains a 3D CNN from scratch'. No — that's C3D. I3D INFLATES 2D pretrained weights.
• MobileNet uses inverted residuals from v1. No — that's v2. v1 is straight depthwise-separable.

Shortcuts

• Conv params: $F(C_{\text{in}}{K}^2+1)$, independent of $H,W$.
• Output size: $\lfloor (W+2P-K)/S\rfloor +1$.
• Same pad odd K: $P=(K-1)/2$.
• Receptive field: $L$ stacked $3\times 3$ stride 1 → $2L+1$.
• **Two $3\times 3$ < one $5\times 5$:** fewer params + extra non-linearity.
• 1×1 conv: bottleneck, mixer, cheap, plus a non-linearity.
• Depthwise-separable: ~$8$–$9\times$ cheaper at $K=3$.
• ResNet skip: $y=F(x)+x$; gradient $\partial F/\partial x+I$ ⇒ no vanishing.
• DenseNet: concatenate (not add).
• EfficientNet: compound-scale $d,w,r$ together by one coefficient.
• Inflation (I3D): copy 2D filter $K_t$ times, divide by $K_t$.
• Translation EQUIvariance (feature maps) ≠ INvariance (after pool + FC).

Proofs / Algorithms

Translation equivariance of convolution. Define $T_v(I)(x,y)=I(x-v_x,y-v_y)$. For a convolution $\varphi =I⊛K$: $\varphi (T_v(I))(x,y)={\sum }_{i,j}K(i,j)I((x-v_x)-i,(y-v_y)-j)=\varphi (I)(x-v_x,y-v_y)=T_v(\varphi (I))(x,y)$. Hence $\varphi \circ T_v=T_v\circ \varphi$ — equivariance.

Residual gradient retains scale. $\partial y/\partial x=\partial F/\partial x+I$. Through $L$ stacked residuals, the contribution along the skip path is $I$ at every block, hence the product of identities along the skip path is still $I$. Therefore even if $\Vert \partial F/\partial x\Vert$ is tiny, total gradient norm $\ge 1$.

Depthwise-separable param savings. Standard $K^2{C}_{\text{in}}{C}_{\text{out}}$; depthwise-separable $K^2{C}_{\text{in}}+C_{\text{in}}{C}_{\text{out}}$. Ratio $=1/C_{\text{out}}+1/K^2$. At $K=3,C_{\text{out}}\to \infty$: ratio $\to 1/9\approx 11\%$, i.e., $9\times$ cheaper in the limit.

Compound scaling balances depth, width, resolution. For an isolated CNN with $d$ layers, width $w$, resolution $r$: FLOPs $\propto d\cdot w^2\cdot r^2$. Compound scaling with $d={\alpha }^{\varphi },w={\beta }^{\varphi },r={\gamma }^{\varphi }$ gives FLOPs $\propto {\alpha }^{\varphi }{\beta }^{2\varphi }{\gamma }^{2\varphi }=(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }$. EfficientNet fixes $\alpha {\beta }^2{\gamma }^2\approx 2$ so doubling $\varphi$ doubles FLOPs.