Big cave, one chief, endless shiny stones
Long ago, year 2003, big tribe named Google need cave to hold EVERYTHING — all web, all Gmail scroll, all search-marks. Too much meat for one rock-disk. So they build GFS (Google File System) — one giant shared cave spread across many-many cheap machine-rocks.
Ugg tell you secret: cheap machine-rock DIE, every day. Count it: . So GFS built knowing failure is the norm — machine dying normal like sun rising. Cave must heal self.
What tribe do in this cave
Tribe read BIG — long streaming gulps, whole mammoth at once, not tiny nibbles. Tribe write BIG — stick new meat on END of pile (append), and many hunters append same pile at once (logs, hunt-queues). Small random pokes rare. Tribe want FAT PIPE (high bandwidth) more than FAST ANSWER (low latency). Remember this — it explain every weird choice below.
Three kind of cave-dweller
Single Master — ONE chief rock. Chief hold all the map-knowledge (metadata): names, where meat sit, version numbers. Chief no hold meat itself.
Chunkservers — many worker-rocks. THEY hold actual meat-data. Live in racks (rock-shelves).
Clients — hunters holding a GFS talking-stick (library). Hunter ask chief WHERE, then run straight to worker for meat.
Ugg warn: meat NEVER go through chief! Chief only give directions; data flow hunter-to-worker direct. One chief simple — best global copy-placement — but is single point of fail; fixed by copying chief's log and hunters caching so they no pester chief.
Why meat cut in giant 64-chunk
Other caves cut meat in tiny 4-KB pebbles. GFS cut into HUGE 64 MB chunks. Each chunk get one unique name-scratch — a 64-bit chunk handle. Each chunk copied 3 times across different racks — one shelf catch fire, two copies still safe.
Why so big? Ugg do rock-math: 1 PB file = only 16 million chunks at 64 MB. At tiny pebbles it be 256 BILLION pieces — chief brain explode holding that map! Big chunk = small map, fewer hunter-chief chats, fat streaming gulps. Bad side: one tiny hot file is one chunk, all hunters hammer one worker. Fix: copy hot file MORE times.
What chief keep in head
Chief keep three map-things, ALL in fast head-memory (not slow disk):
1. Names of files and chunks — carved in stone-log (survive chief-nap). 2. Which chunks make which file — logged too. 3. WHERE the chunk-copies sit — NOT logged!
Ugg explain not-logged trick: worker-rocks are the TRUTH about what meat they hold — they lose and gain chunks on own. Stone-log of locations go STALE fast. So on wake-up chief just ASK every worker "what you hold?" by heartbeat — steady thump-message. Chief also log each chunk version number, and remember current primary and lease-time.
How hunter READ meat
1. Hunter find chunk from byte-spot: — divide spot by chunk-size, chop leftover. 2. Hunter → chief: "(file name, chunk index)?" 3. Chief → hunter: "chunk handle + the 3 copy-spots." 4. Hunter REMEMBER this (cache). No pester chief again for same chunk. 5. Hunter go to NEAREST copy, grab byte-range. Done.
How hunter WRITE meat
1. Hunter → chief: "who primary for this chunk?" No primary? Chief pick one fresh copy, give it a lease (chief-blessing lasting 60 seconds), bump version. 2. Chief send primary + secondary spots. 3. Hunter PUSH data down a pipelined chain — hand to nearest worker, that worker forward to next-nearest WHILE still receiving, like water-gourd passed down a line. All hold data in cache. 4. All workers got data? Hunter → primary: "now do write." 5. Primary serialises — primary pick ONE order for all tangled writes, do them, tell secondaries same order with numbers. 6. Secondaries copy that exact order, thump back ACK. 7. All ACK → primary say SUCCESS. Any fail → hunter retry (region messy meanwhile).
Lease clever: for 60 s the primary alone decides order, so chief no touch every write. That how GFS write fast, no chief-choke. Lease renew on heartbeat.
Why split data and command
Control flow (tiny "do write!" command) go star-shape: hunter → primary → secondaries. Data flow (heavy meat-bytes) go pipelined line. Rock-math: star make hunter push bytes alone — choke. Pipeline: each rock use full in-pipe AND out-pipe, total time near — about 3 times faster. Fat pipe win!
The four states of the meat-pile
After a write, a file-spot is one of four:
- Defined — all copies same AND show one writer's meat whole. Best.
- Consistent — all copies show SAME bytes (but maybe not one clean write).
- Undefined — all copies same, BUT bytes are mingled scraps from many writers at once. Consistent yet undefined.
- Inconsistent — copies show DIFFERENT bytes. From failed write or stale copy. Worst.
Rule: one write alone → defined. Many writes at once → consistent but undefined. Failed write → inconsistent. Ugg warn: "consistent" NOT same as "defined"! Consistent only mean same-everywhere; defined mean same AND clean whole write.
Atomic record append — the star trick
Atomic record append — hunter give only the meat, GFS pick the spot and stick it on the end at least once. Many hunters append one log with NO fighting-lock, because primary sort the offsets. If append fail on one copy, hunter retry at a NEW spot — so meat may appear TWICE (duplicate) plus padding at bad spot. That make region defined interspersed with inconsistent. Tribe cope by stamping each record with checksum + unique mark.
Ugg warn: GFS no promise clean linear writes! Append at-least-once, concurrent writes go undefined, failed writes leave inconsistent. App MUST handle duplicate, padding, stale.
Keeping cave clean and safe
Garbage collection — delete a file, GFS just HIDE it with new name, keep 3 days for "oops" recovery, then scan-and-sweep. Orphan chunks and stale copies: chief tell worker by heartbeat "I no know this chunk" → worker toss it.
Stale replica detection — copy that napped through a lease carry OLD version number; chief spot mismatch on heartbeat → mark stale → sweep.
Fault tolerance: fast restart (seconds); 3× chunk copies across racks; chief's log + checkpoints copied to backup rocks, with a shadow master serving read-only map if chief down; data integrity by checksum every 64 KB block, checked on every read, bad block re-copied from good twin; and diagnostic logs of all messages for later sniffing.
Copy-on-write snapshot — want a snapshot? Chief just copy the MAP and bump chunk refcounts — meat NOT copied yet, so snapshot instant (). Only on first write after does worker make a LOCAL meat-copy (no network). Cheap and fast!
Ugg remember
- One chief (master) hold all map in head; many workers (chunkservers) hold meat; hunters (clients) ask chief WHERE then grab meat DIRECT. Meat NEVER through chief.
- Carve these numbers: 64 MB chunk, 3× copies across racks, 60 s lease, 3-day hide, checksum per 64 KB block. Chunk locations NOT logged — rebuilt from heartbeat.
- Read = ask chief → cache → nearest copy. Write = get lease → push data pipelined → primary picks the order → secondaries follow.
- Four states: defined (clean+same) is best; consistent (same); undefined (same but mingled); inconsistent (different). Consistent ≠ defined!
- Atomic record append = at-least-once → "defined interspersed with inconsistent." Snapshot is copy-on-write. Stale copy caught by version number on heartbeat.