Ugg find scatter-dots floating in the air

Self-cart drive at night, eye-camera blind. On the roof sit a magic light-stick — LiDAR — that shoots light-pulses, counts how long each takes to bounce back, and drops a dot where it hit something hard. Each dot carries three numbers $(x,y,z)$. A million dots, scattered, no order, no grid. This pile is a point cloud — the beast we must tame.

Why old eye-rock no work here

Ugg's old picture-brain is a CNN — it slides a stamp-rock over a neat grid where every dot knows its neighbours: left, right, up, down. Point cloud smashes three things a CNN needs — chant these, root of the whole unit:

• Unstructured — no grid, dots just float. Ugg cannot grab a $3\times 3\times 3$ box of neighbours, there is no box.
• Irregularly distributed — some places thick with dots (mammoth hide), some places empty (sky, holes).
• Unordered — same beast can be written $\{p_1,\dots ,p_N\}$ or shuffled any of $N!$ ways. Brain MUST grunt the same answer every time.

Extra pain: no giant pile of labelled 3D data (no ImageNet for dots), so method must learn from little.

Five ways to hunt a 3D beast

Keep this list of five in the skull — it the spine of everything:

• View-based — walk around beast, draw many flat pictures, run a flat CNN (Multi-view CNN).
• Volumetric — chop space into little cubes, run a 3D CNN (VoxNet).
• Point-based — touch the raw dots straight (PointNet, PointNet++).
• Graph-based — dots are friends, tie near-friends with strings (DGCNN).
• Mesh-based — use the triangle-skin of the beast (MeshCNN).

(Sixth way, fuzzy blobs — 3D Gaussian Splatting — is next-fire's tale.) Three jobs the tribe wants: classification (one name for whole cloud → $K$ scores), part segmentation (name each dot's body-part → $N\times m$ matrix), semantic segmentation (name each dot's scene-thing → $N\times m$). Same net-body, only the tail-head change.

VoxNet — the big dumb hammer

First idea simple: if flat CNN love a grid, build a fat 3D grid. VoxNet drop cubes over the dots. Each cube — a voxel — is full or empty; this map is an occupancy grid. Push the block through two 3D-conv layers, then flat layers, out come scores. It work — but die when grid grow. Four rocks in the shoe, chief ask these MOST:

1. Memory blow up cube-fast — grid side $N$ costs $\mathcal{O}(N^3)$ cubes. $1024^3$ ≈ a billion, $256^3$ ≈ 16 million. Too fat for the brain-stone. 2. Most cubes empty — beast-skin is thin, inside and air are zero. 99 in 100 cubes hold nothing. 3. Blurry eyes — most VoxNet only afford $32^3$, tiny-pebble resolution. 4. Staircase skin — snapping smooth $(x,y,z)$ to blocky cubes makes quantisation artefacts, smooth curve turns jaggy.

Ugg say: VoxNet only win when data is thick and truly solid — like a healer's body-scan (CT). For thin skin-clouds, reach for PointNet.

PointNet — the clever fire

Naive try: mush all dots into one long string $[x_1,y_1,z_1,\dots ]$ and feed a plain brain-net (MLP). BAD. Shuffle dots → new string → new answer. This is permutation variance: the brain learns the ORDER, not the shape.

PointNet (Qi and tribe, 2017) cure it with symmetric functions — functions that no care about order. Ugg know two already: $\max$ and sum. PointNet write the whole-cloud feature as

where $h$ is a shared MLP (same little brain pressed on each dot alone), $\max$ takes the biggest across all dots, and $\gamma$ is another MLP. Because $\max$ no care about order, the whole thing no care about order — permutation invariance by construction. Chant the three-step recipe: shared MLP on each dot → max across dots → MLP.

Big magic word: Universal Approximation Theorem — any smooth order-blind function can be copied as close as Ugg want by this shape, given fat enough $h$. So PointNet is not just A order-blind net, it is THE one.

Ugg warn: use max, NOT sum, NOT average! Max keeps only the loudest dot per feature. The surviving dots are critical points — they trace the beast's outline. Drop other dots, answer no change; but a sneaky enemy who removes the critical dots blinds the net.

Full body: $N\times 3$ dots → tiny T-Net guesses a $3\times 3$ turn-matrix to straighten the beast → shared MLP lifts each dot to 64 → another T-Net straightens features → shared MLP lifts to 1024 → max-pool → one 1024-long global feature. Naming head: MLP → $K$ scores. Part-segment head: glue the global feature back onto EVERY dot's own feature (each dot now knows itself AND the whole beast), then shared MLP: ${\hat{y}}_i=\text{MLP}([\varphi (x_i);g])$ with $g={\max }_j\varphi (x_j)$.

PointNet++ — see close AND far

PointNet's wound: each dot walks alone until the one big max — no local neighbourhood, no $3\times 3$ eye. It sees whole shape but misses small texture (the bend where chair-leg meets seat). PointNet++ picks spread anchor-dots with Farthest Point Sampling (FPS):

next anchor is the dot FARTHEST from all anchors picked so far — fills the biggest empty hole, spreads even. Then gather neighbours in a ball query (all dots inside radius $r$), run PointNet inside each little group, stack again and again. Early layers see close, late layers see far — like a CNN grows its eye.

DGCNN — dots make a friend-web

DGCNN say: a point cloud IS a graph. Each dot a node; tie each dot to its $k$ nearest dots. The move is EdgeConv: for dot $i$ and near-friend $j$, glue $[x_i,x_j-x_i]$ — the dot itself PLUS the arrow-offset to the friend — push through MLP, then squash the $k$ edges symmetrically:

The offset $x_j-x_i$ gives the close-eye, like image-conv says "this pixel vs neighbour"; symmetric squash means friend-order no matter. Cleverest grunt — the dynamic graph: rebuild the near-friend web in FEATURE space, not position space, after each layer. Early layers tie dots that sit close; late layers tie dots that MEAN the same (all wing-tip dots group, even on opposite wings). Ugg warn: feature-space is the whole "Dynamic" — miss it and you got the wrong net. Snags: all-pairs distance is $O(N^2)$ memory; fixed $k$ chokes on wild density; no shrinking across layers.

MeshCNN — when beast has triangle-skin

Sometimes data is a mesh — corner-points tied into triangles — richer than dots because it holds the surface topology. MeshCNN's grunt: use edges as the conv unit (NOT corners, NOT faces!). Why? Every edge in a triangle-mesh sits between exactly two faces, so it always has exactly 4 neighbour edges — a fixed patch, like $3\times 3$ round a pixel.

Each edge carries a 5-D feature: the dihedral angle between its two faces, the two inner angles across the edge in each face, and two length-ratios of the edge to the opposite edge in each face:

All five are intrinsic — no change when you turn or move the beast. Snag: the 4 neighbour edges $(a,b,c,d)$ have two orders depending which face starts. So MeshCNN feeds symmetric mixes $(a+c,|a-c|,b+d,|b-d|)$ — swap the pair, these stay same. Kernel has 5 weights $(k_0,\dots ,k_4)$: one for the centre edge, one per mix-channel. Pooling = learned edge-collapse: edges with smallest learned loudness get squashed (ends merge, topology fixes, features average) — task-driven simplification. Mesh unpooling remembers the collapse order so it can undo for segmentation that needs full resolution back.

Ugg remember

• Point cloud breaks three things — unstructured, irregular, unordered — so grid-CNN dies. Cure = order-blind (symmetric) operators.
• PointNet chant: shared MLP → max (not sum/avg!) → MLP, i.e. $f=\gamma ({\max }_{i}h(x_i))$. Max gives critical points; the theorem says it is the universal order-blind net.
• PointNet++ = PointNet made local: FPS for even anchors + ball query for neighbours, stacked for close-and-far sight.
• DGCNN = kNN graph + EdgeConv with offset $x_j-x_i$; rebuild the web in feature space each layer — that is the "Dynamic". Don't confuse it with PointNet++.
• VoxNet cube-memory $\mathcal{O}(N^3)$, mostly empty — win only on thick solid data (CT). MeshCNN = edge-unit, 5 intrinsic features, symmetric conv, learned edge-collapse pooling.