Distributed Systems

A distributed system is a collection of independent computers that *appears* to its users as a single coherent computer. Every interesting property of the field — fault tolerance, scalability, consistency — flows from the gap between that appearance and the reality of independent components that fail independently, communicate over unreliable networks, and share no clock or memory. The CAP theorem is the headline trade-off: under network partitions you must sacrifice either consistency or availability.

Without a global clock, we need a discipline for talking about 'before' and 'after'. Lamport's happened-before relation $(\to)$ captures *causal* ordering; logical clocks (scalar, vector, matrix) implement that relation in code. The trade-off is storage vs information: scalar = 1 int (cheap, but loses converse); vector = n ints (strongly consistent); matrix = n × n ints (also tracks what others know about each other — enables obsolete-message GC).

We can't freeze time across a distributed system to take a snapshot. Instead, each process records its own local state independently, and the algorithm records the in-transit messages on each channel such that the combined picture is one the system *could have been in* in an equivalent execution — even if it wasn't actually in that exact state at any single instant. The channel-type assumption (FIFO / non-FIFO / causal) determines which algorithm applies.

FIFO says 'messages from the same sender arrive in order'. Causal order says 'if $\text{send}(m_1) \to \text{send}(m_2)$ and both target the same destination, $m_1$ is delivered before $m_2$' — even if the senders are different. This matters when an effect's cause comes through a different path. Total order goes further: all processes deliver every message in the same global order.

Mutual exclusion in a distributed system is the same idea as a critical section in OS — except there's no shared memory, no semaphore, no central arbitrator. Everything is messages. Five algorithms attack the problem in two families: **non-token** (request-permission from peers — Lamport, R-A, Maekawa) and **token** (a single token grants entry — Suzuki-Kasami, Raymond). Trade-offs: message count, synchronisation delay, FIFO requirement, fault behaviour.

A deadlock in a distributed system is the same idea as in an OS — a cycle of processes each waiting for a resource held by another. But the DETECTION criterion depends on the resource model: in Single-resource and AND models a cycle in the wait-for graph suffices; in OR (a process needs any one of many resources) you need a stronger structure called a **knot** (an SCC with no outgoing edges). Three classical algorithms — Chandy-Misra-Haas probe, Mitchell-Merritt label-passing, Chandy diffusion — implement detection for different models.

Consensus = a group of nodes agree on a single value despite some being faulty. The difficulty depends on the failure model. **Crash-failure** (silent halts, no lies): solvable in $f + 1$ rounds with $n > f$ processes. **Byzantine** (arbitrary lying, conflicting messages): requires $n \ge 3f + 1$ and at least $f + 1$ rounds — and is **impossible in async systems** (FLP). Two flagship Byzantine algorithms: OM(m) (exponential messages, recursive) and Phase King (polynomial, needs more processes).

A distributed transaction touches data at multiple sites. ACID requires that it either commits at ALL sites or aborts at ALL sites — never inconsistent partial outcomes. **2PC** is the textbook protocol: a coordinator gathers votes, then broadcasts the decision. Its weakness — and the entire reason 3PC exists — is the **blocking problem**: if the coordinator crashes after participants have voted but before announcing the decision, participants hold all their locks and wait indefinitely.

Raft is **Paxos rewritten for understandability** (Ongaro and Ousterhout, 2014). Same correctness as Paxos; simpler decomposition. The cluster keeps a replicated log via three independent mechanisms: random timeouts elect a single leader per term; the leader replicates each new entry to a majority before committing; an 'election restriction' ensures the new leader has every committed entry from prior terms.

Computing a minimum spanning tree centrally is easy (Prim, Kruskal). Doing it without a central coordinator — each node only knows its incident edges and their weights — is harder. GHS is **distributed Kruskal**: many small fragments grow in parallel, repeatedly absorbing or merging via their minimum outgoing edge (MWOE). Three combining rules manage the growth: smaller fragments rush into larger ones immediately (A); equal-level fragments merge only if they both chose the same edge (B); otherwise wait (C).

GFS (2003) is the design Google used to store all of search and Gmail at the time. The radical choices: a **single master** for all metadata (vs many believed this couldn't scale); **64 MB chunks** (vs typical 4 KB block sizes); **failure is the norm** (commodity hardware in racks); **atomic record append** as the primary write operation (vs random in-place writes). The 'inconsistent' / 'undefined' regions in the consistency model are unusual — applications are expected to cope.