Many picture in, brand-new picture out

Ugg take 100 picture of cave-room with phone-rock, all sides. Now Ugg want computer show the room from a spot where Ugg never stood. That is novel view synthesis — make picture from new eye-spot not in the pile.

Hard for long time. Then 2020 bring NeRF (Neural Radiance Field): a tiny brain-net $f_{\theta }(x,y,z,\theta ,\varphi )\to (\text{RGB},\sigma )$ — give it a point and a look-direction, it grunt back colour and thickness. To paint one pixel, shoot spear-ray from eye through pixel, poke many points on ray, ask brain-net each, add colour weighted by thickness (volumetric rendering). Pretty, but SLOW: hours to train, paint only ~1 picture per second. No good for VR goggle.

Then 2023 bring 3D Gaussian Splatting (3DGS): same goal, fit picture to scene — but throw brain-net away, use millions of fuzzy blobs, paint real-time.

Two kind of scene-rock, plus fuzzy third

• Explicit — count shape straight: point cloud, mesh (corners+faces), voxel grid.
• Implicit — hide shape in a function. Signed Distance Function (SDF) $f(x,y,z)$ grunt distance to nearest skin (skin = where it zero). NeRF also implicit.
• 3DGS = new sixth slot: explicit like points, but each point a FUZZY blob (Gaussian), not hard dot.

3DGS no learn — it just fit one cave

Big idea: 3DGS have NO brain-net, NO learning. It is per-scene optimisation. Brain-net weights $[b,w_1,w_2,\dots ]$ train on big pile so they work on NEW picture — generalise. 3DGS params $[\mu ,\Sigma ,\alpha ,\text{SH}]$ fit ONE cave's picture; work nowhere else. New cave → whole new blobs. Ugg warn: no say 3DGS "learn weights that work everywhere" — WRONG.

COLMAP tell where Ugg stood

Before 3DGS, run picture through COLMAP — old Structure-from-Motion machine. It grunt back: (1) camera poses — 6-DOF spot+tilt of each picture; (2) sparse point cloud — few 3D dots from matched footprints. Poses say where picture taken; sparse cloud seed one blob per dot. Ugg warn: 3DGS NOT end-to-end — pose-finding done by old COLMAP first, before 3DGS begin!

Three big rocks hold the cave

Three Pillars:(1) Scene modelling — what each blob is.(2) Image formation — how blob make flat picture.(3) Optimisation — how blob fit picture.

Pillar one — what one fuzzy egg is

Each blob a fuzzy 3D egg, solid middle, see-through edge:

Front number $1/((2\pi {)}^{3/2}\sqrt{|\Sigma |})$ DROPPED — opacity control the magnitude instead. Four groups:

• **Mean $\mu$** — where egg sit. 3 number.
• **Covariance $\Sigma$ — egg shape+tilt. 3×3 symmetric = 6 free number, but MUST be positive semi-definite (PSD)** or egg flip inside-out; raw SGD break it. Fix — split it: $\Sigma =RS{S}^{\top }{R}^{\top }$, PSD by build. $S$ = scale, 3 number, stored $s=\exp (s_{\text{raw}})$ (always positive); $R$ = rotation as a quaternion, 4 number. Total 7. Quaternion not Euler because Euler get gimbal lock and jumps.
• **Opacity $\alpha$** — 1 number, sigmoid into $[0,1]$. 1 solid, 0 ghost.
• Colour by spherical harmonics — skin shine different per side (chrome ball white here, blue there); plain RGB no catch it. Use spherical harmonics (SH), wavy basis on the ball of look-directions. Degree 3: $(L+1{)}^2=16$ per colour ×3 colour = 48.

Add up: $3+7+1+48=59$ number per egg. Cave use 1M–4M eggs → 60M–240M number, still paint real-time.

Pillar two — squish eggs onto flat rock

Three step: (1) Sort by depth front-to-back. (2) Project each 3D egg to 2D — mean by pinhole, $\Sigma$ squished by the Jacobian of the projection. (3) Alpha-composite front-to-back. Ugg warn: no skip the sort — wrong order make wrong transmittance and wrong colour!

Rendering equation, memorise:

$c_i$ = egg colour (from SH at look-direction). ${\alpha }_i$ = effective opacity — stored $\alpha$ times flat-egg brightness at pixel (middle strong, edge weak). $\prod (1-{\alpha }_j)$ = transmittance — chance light slip past all front eggs unbeaten. Drop under $10^{-4}$ → stop.

All step differentiable: 3DGS built a differentiable tile rasteriser in CUDA on GPU, so pixel-gradient flow back to every $\mu ,\Sigma ,\alpha ,\text{SH}$. THIS let gradient-descent work.

Pillar three — grow eggs, kill eggs

COLMAP give thousands; cave need millions. Adaptive Density Control (ADC), every few hundred step, three move: (1) Clone small egg in under-built spot (big position-gradient = one egg not enough) — copy it, let gradient pull copies apart. (2) Split big egg in over-built spot into two smaller (scale $/1.6$) inside old egg. (3) Prune ghost eggs ($\alpha <0.005$) — cost work, give nothing.

Loss:

$L_1$ = pixel-by-pixel difference (sharp); $L_{D\text{-SSIM}}$ use SSIM for local brightness+contrast+structure. Full hunt: init from COLMAP → pick random picture → rasterise → loss → backprop → Adam step → sometimes ADC → repeat ~30,000 times.

Ugg count how good

Hold out every 8th picture to test. PSNR $=10{\log }_{10}(R^2/\text{MSE})$, $R=255$ for 8-bit; higher better, NO ceiling (Ugg warn: PSNR not bounded!), ~30 dB good. SSIM in $[-1,1]$, higher better, local structure. LPIPS — cut into patches, run through old pretrained net (AlexNet/VGG), measure deep-feature distance; lower better, match human eye. Arrows to memorise: PSNR ↑, SSIM ↑, LPIPS ↓. Test-grounds: Tanks & Temples, Deep Blending, Mip-NeRF 360, DL3DV-10K.

3DGS beat NeRF how

NeRF: implicit MLP, has brain-net, train hours–days, ~1 fps, hard to edit, small store ~MB. 3DGS: explicit eggs, no brain-net, train tens of minutes, >100 fps, easy edit one egg, big store ~GB. Trade: 3DGS pay memory, buy fast render+train and editability.

Ugg remember

• 3DGS = per-scene optimisation, NO brain-net; the params ARE the eggs, they no generalise across caves.
• Each egg = 59 numbers: $3(\mu )+7(\Sigma =RS{S}^{\top }{R}^{\top })+1(\alpha )+48(\text{SH deg 3})$. Never 58, never raw 6-number $\Sigma$.
• Render = sort by depth → project 3D→2D → alpha-composite; $C=\sum c_i{\alpha }_i\prod (1-{\alpha }_j)$; whole thing differentiable in CUDA.
• ADC: clone small, split big (scale/1.6), prune ghost. COLMAP runs first — 3DGS not end-to-end.
• Metrics PSNR↑ SSIM↑ LPIPS↓ (PSNR unbounded); quaternion is 4 number (Euler is 3 but has gimbal lock).