Many cave, but tribe think one cave

A distributed system is many thinking-rocks (computers) working together so the tribe THINKS it is one big thinking-rock. Wise elder Tanenbaum say it best: "a group of computers working together to appear as a single computer to the end user."

Many caves, but tribe sees ONE cave. The whole hunt is closing the gap between that look (one cave) and the truth (many caves that break, drift, and lose runners).

Five marks every spread-system carries, want it or not:

• many computers — more than one thinking-rock.
• run concurrently — all at the same time, not in turns.
• fail independently — one cave falls in, other caves keep working. This independent failure is the root of the tribe not starving when one cave dies.
• no global clock — no one big sun-dial for all. Each cave has its own sun-stick, and sun-sticks drift apart. So "WHEN did the mammoth come?" is fuzzy.
• no shared memory — no shared pile of shiny-stones every cave touches. Caves talk ONLY by sending a runner with a message.

Each mark is a PROBLEM before a gift: no shared pile forces far-talk locks (simple semaphores fail), no sun-dial forces smart-count (logical clocks), and eaten runners force an agree-game (consensus).

Why Ugg want many cave

• Horizontal scaling — Ugg carves a bigger rock only so big, then rock-physics says NO; but Ugg can always add MORE small rocks. Adding machines is horizontal scaling. Growing-one-big-rock is vertical scaling, and it hits a ceiling.
• Some hunts are born spread — a bank across two far villages — and cannot squeeze into one cave.
• Fault tolerance — one cave falls, tribe still eats. Low latency — put berries near the hungry tribe. Share the food-map across far tribes. Many cheap rocks beat one fancy rock for the price.

Why many-cave is hard

Six big rocks in the path:

• Unreliable communication — runner drops the message, comes late, comes in wrong order, or a wolf eats runner.
• Lack of global knowledge — each cave sees only its own pile.
• Lack of synchronisation — sun-sticks drift, so "when did it happen?" is fuzzy.
• Concurrency control — no shared pile, so Ugg builds distributed mutual exclusion from runner-talk alone.
• Failure and recovery — caves and runner-paths die; bringing a woke-up cave back in-step is its own puzzle.
• Deadlock, done-detection, cave-file-storage — every old one-cave puzzle gets HARDER when spread out.

One cave, shared-pile cave, spread cave

• Centralised — one rock, one pile, one sun-stick. Simple, but one break kills all and it cannot grow.
• Parallel — many rocks, BUT shared memory and ONE clock (multicore, GPU). Tight-tied; rock-magic keeps the pile the same, so Ugg reasons like one machine.
• Distributed — many rocks, no shared memory, no global clock, only runner-talk. Loose-tied, each cave its own boss. This is the model the course uses.

Three shiny stones, grab only two

CAP theorem — Brewer spoke it (2000), proved (2002). A networked shared-data tribe holds only TWO of three shiny stones:

• Consistency — every cave tells the SAME newest story. Ask any cave "how many mammoth?" and get the same freshest number.
• Availability — every not-dead cave ALWAYS answers, in good time, no error-grunt.
• Partition tolerance — tribe keeps working even when the valley floods and runners cannot cross (a network partition).

Now the deep truth. In the real world the valley WILL flood, so P is not a real choice — P is always must-have. The true pick is: during a flood, keep same-story (C) or keep always-answering (A)? You cannot keep both.

Why not? Caves A and B both hold the mammoth-count. Flood cuts the path. Tribe writes "5" to A, then asks B. A cannot tell B. So either B answers the old "3" (breaks C), or B refuses until the flood ends (breaks A). No third path. That is the proof.

Which cave-system grabs which stones

• CP (drop A, keep same-story): Google Spanner, HBase, Mongo-with-majority, old RDBMS clusters. In a flood they BLOCK to stay correct.
• AP (drop C, keep answering): DNS, Amazon Dynamo, Cassandra, CouchDB, web-caches. In a flood they answer, maybe stale, and fix it later.
• CA only when NO flood ever — a single-site RDBMS. Cannot survive a real flood.

Which stone? Think product, not rock! Airplane seat 6B — need same-story; better refuse than sell one seat twice. Tribe-face-page (social profile) — need answering; a little-stale face is fine, a dark screen is bad.

Two hunt-chiefs on two hills

Two-Generals problem (war-of-chiefs). Two hunt-bands on two hills must attack at the SAME time or both die, but they talk only by runners across a wolf-valley where a wolf might eat the runner.

Chief A sends "attack at dawn." B gets it, but does not know if A knows it arrived, so B sends a back-nod (ack). A gets the nod, but does not know if B knows it arrived, so A sends a nod-of-nod. Forever. No finite runner-game makes BOTH chiefs 100% sure.

Little proof: say the shortest game is $n$ runners. The LAST nod is unconfirmed — its sender never knows it landed. So the game with $n-1$ runners must ALSO work, since the last added no new knowing for one side. But $n$ was shortest — contradiction! So no count of runners is ever enough. ∎

Take-away: sure agreement over an unreliable path is IMPOSSIBLE. Spread-tribes live with fuzzy-sure: time-outs, try-agains, "good-enough" chance-promises. Even TCP's three-hand-shake is not perfect — the last ACK can be lost, yet the greeting counts as done. This drives the bigger FLP impossibility result and every "stuck" moment in 2PC and Raft.

Ugg warn: no do this!

• No say "CAP is just pick any 2 of 3, all equal-trade." NO. P is forced by real floods; the real pick is C-vs-A during a partition.
• No mix CAP-Consistency with ACID-Consistency! CAP-C is linearisability (one newest value shows). ACID-C is keep-the-rule-not-broken. Different animals!
• No say "Two-Generals proves no protocol works." NO — it only kills SURE certainty; real games use bounded-fuzzy and work fine.
• No list an example without saying WHICH stone (C or A) you drop and WHY. The elder marks you down!

Handy trick: even with NO flood, "wait for all caves to nod" (slow, strong) versus "answer from one cave" (fast, stale) is a smooth trade — the PACELC rule extends CAP to it.

Ugg remember

• Distributed system = many computers, appear as one. Five marks: many + concurrent + independent failure + no global clock + no shared memory.
• Six hard-rocks: bad runner-talk, no full-knowing, no sync, far-locks, fail-and-recover, old-OS-puzzles-but-harder.
• CAP: grab 2 of {C, A, P}. Floods are real ⇒ P is forced ⇒ the true pick is C vs A during a partition. CP = Spanner/HBase; AP = Dynamo/Cassandra/DNS.
• Two-Generals: sure agreement over an unreliable path is impossible; live with time-outs and try-agains.
• When giving a CAP example, ALWAYS name which stone you sacrifice and why.