Intuition

A distributed system is a collection of independent computers that *appears* to its users as a single coherent computer. Every interesting property of the field — fault tolerance, scalability, consistency — flows from the gap between that appearance and the reality of independent components that fail independently, communicate over unreliable networks, and share no clock or memory. The CAP theorem is the headline trade-off: under network partitions you must sacrifice either consistency or availability.

Explanation

Tanenbaum's definition (canonical). 'A group of computers working together to appear as a single computer to the end user.' Five required features: (i) many computers, (ii) running concurrently, (iii) failing independently (one crashes, others keep working), (iv) no global clock (each node has its own clock that drifts), (v) no shared memory (only message passing).

Motivation — why bother? *Horizontal scaling* (vertical scaling has CPU/memory limits; horizontal scales by adding more machines). *Inherently distributed applications* (banking across cities, video conferencing). *Data sharing* (a file lives where it makes sense; clients query). *Enhanced reliability* (one node down ≠ system down). *Better performance/cost ratio* (commodity hardware in parallel beats a single expensive box). *Fault tolerance*. *Low latency* (place data near users).

Challenges — what makes DS hard. (i) Unreliable communication — messages drop, get delayed, arrive out of order, can be intercepted. (ii) Lack of global knowledge — each node sees only its own memory. (iii) Lack of synchronisation — local clocks drift, so 'when did X happen' is ambiguous. (iv) Concurrency control — no shared memory ⇒ no semaphores; must do distributed mutual exclusion. (v) Failure & recovery — nodes/channels fail; bringing a recovered node back into sync is non-trivial. (vi) Deadlocks, termination detection, file system issues — every classical OS topic gets harder.

Centralised vs Parallel vs Distributed. Centralised: single CPU, single memory, single clock. Parallel: multiple CPUs with shared memory and a single clock (tightly coupled). Distributed: multiple CPUs, no shared memory, no global clock, communicate by message passing — loosely coupled, autonomous nodes.

CAP theorem (Brewer 2000, proved 2002). A networked shared-data system can simultaneously guarantee only TWO of: Consistency — every node returns the same most-recent successful write. Availability — every non-failing node responds to every request in reasonable time. Partition Tolerance — system continues operating despite arbitrary message loss / network partitions. Real networks WILL partition, so P is non-negotiable in practice — meaning every real distributed system must choose between C and A *during a partition*.

CAP examples. *CP (sacrifice A):* Google Spanner, HBase, MongoDB with majority writes, traditional RDBMS clusters. *AP (sacrifice C):* DNS, Amazon Dynamo, Cassandra, CouchDB, web caches. *CA only without partitions:* single-site RDBMS — cannot survive a real network partition.

Consistency-critical vs availability-critical examples. Consistency-critical: an airline seat sold once (6B can't be assigned to two passengers); last item of stock on Amazon. Availability-critical: social media profile (slightly stale is fine; downtime hurts); adding a file to a movie server.

Two-Generals problem (also 'War of Generals'). Two armies coordinating an attack via a courier who may be captured. No matter how many acknowledgements are sent, the LAST ack itself is unconfirmed — so neither general can be 100% sure the other knows it's safe to attack. Take-away: deterministic consensus over an unreliable channel is impossible. Motivates the FLP impossibility result and explains why distributed protocols always work with bounded uncertainty (timeouts, retries, probabilistic guarantees).

Two-Generals applied to TCP. Even TCP's three-way handshake doesn't guarantee perfect mutual agreement. The final ACK from the receiver to the sender might be lost — but the connection is still considered established. We accept 'good enough' probabilistic agreement because perfect agreement is provably impossible.

Definitions

• Distributed system (Tanenbaum) — A group of computers working together to appear as a single computer to the end user.
• Consistency (CAP) — Every node returns the same most-recent successful write — all nodes have the same view of data at every instant.
• Availability (CAP) — Every non-failing node responds to every request in reasonable time, with a non-error response.
• Partition tolerance — System continues to operate despite arbitrary message loss or network partitions between groups of nodes.
• Two-Generals problem — Classical impossibility result showing two parties cannot reach deterministic agreement over an unreliable channel — every ack needs its own ack ad infinitum.
• Independent failure — One node crashing does not directly cause others to crash; the system as a whole keeps operating. Foundation of fault tolerance.

Formulas

• $$\text{Tanenbaum:}\text{distributed system}=\text{many computers}\Rightarrow \text{single-system image}$$
• $$\text{CAP:}\text{pick 2 of}\{C,A,P\};P\text{is unavoidable}\Rightarrow \text{trade}C\text{vs}A$$

Derivations

Why partitions force C-vs-A. During a partition, two sides of the network can't communicate. Each side can either (a) refuse writes to keep replicas in sync (sacrificing availability) or (b) accept writes locally and reconcile later (sacrificing consistency at the moment). There is no third option: returning a stale value sacrifices C, refusing the request sacrifices A.

Two-Generals informal proof. Suppose generals can agree. Let n be the smallest number of acknowledgements they exchange. The last ack is unconfirmed; the sender doesn't know it arrived. So the protocol with n-1 messages must also work (since the last message added no information for one side). Contradiction with minimality. Therefore no finite protocol can guarantee agreement.

Examples

• Airline seat reservation — consistency critical. Database must guarantee seat 6B is sold exactly once; better to refuse the second request than to double-sell.
• Social media profile update — availability critical. Better to serve a slightly stale profile than to refuse the request. Eventual consistency is acceptable.
• Banking transactions across two cities — inherently distributed; cannot be served from a single machine without unacceptable latency.
• Hawk-Eye tennis ball tracking — combination of inherently distributed sensors (cameras around the court) + real-time 3D reconstruction.

Diagrams

• CAP triangle with C, A, P at vertices. Real systems sit on edges: CP (Spanner, HBase, RDBMS), AP (Dynamo, Cassandra, DNS). The 'CA' point is theoretical only.
• Three-tier comparison table: Centralised (1 CPU, shared mem, 1 clock) vs Parallel (many CPUs, shared mem, 1 clock) vs Distributed (many CPUs, no shared mem, no global clock).
• Two-Generals message exchange: each ack needs its own ack — a recursive impossibility.

Edge cases

• 'CA' is only achievable in a non-partitioned single-node system. Anyone claiming CA in a real distributed system has redefined either 'partition' or 'consistent'.
• Partitions are not always full disconnects — they include slow links, asymmetric routing failures, and one-way packet loss. Any of these forces a CAP choice.
• Even TCP doesn't guarantee consensus — only probabilistic 'good enough' agreement bounded by retries.
• Latency-vs-consistency. Even without partitions, the choice 'wait for all replicas to confirm' (high latency, strong consistency) vs 'reply from one replica' (low latency, possibly stale) is a continuous CAP-like trade-off (PACELC framework extends CAP).

Common mistakes

• Saying 'CAP means pick 2 from 3, all equally tradeable.' No — P is non-negotiable in real networks. The real choice is C vs A *during partitions*.
• Confusing 'consistency' in CAP with 'consistency' in ACID. CAP-C is linearisability (single most-recent value visible). ACID-C is constraint preservation (no invariant violated).
• Saying 'Two-Generals proves no DS protocol works.' No — it shows no protocol gives DETERMINISTIC certainty. Real protocols accept bounded uncertainty.
• Listing examples without explaining the trade-off. When asked for a CAP example, always state WHICH of C or A is being sacrificed and WHY.

Shortcuts

• Tanenbaum: 'many computers, appear as one'.
• Five features: many + concurrent + independent failure + no global clock + no shared memory.
• CAP: pick 2; P is real ⇒ trade C vs A.
• CP examples: Spanner, HBase. AP examples: Dynamo, Cassandra, DNS.
• Two-Generals takeaway: deterministic consensus over unreliable channels is impossible.

Proofs / Algorithms

CAP informal proof. Consider two nodes A and B holding a replica of value x. A partition severs the link between them. A client writes x = 5 to A; another reads x from B. Under partition: A can either propagate the write (impossible, link severed) or accept it locally (B's read returns stale, sacrificing C); B can refuse the read until link restored (sacrificing A) or return stale (sacrificing C). No option preserves both C and A while staying P-tolerant. ∎

Two-Generals impossibility. Suppose protocol P with n messages achieves agreement. Consider the sub-protocol P' = P minus the last message. By symmetry the receiver of message n is in the same epistemic state as before message n; otherwise message n carries no decision-relevant information. By induction, no number of messages suffices. ∎