A Practical Primer on Distributed Systems

Building software that runs on a single machine is hard. Building software that runs reliably across dozens of machines—while each can fail independently—is a different kind of hard.

This post is a ground-up introduction to the ideas that matter most.

The Fundamental Problem

A distributed system is a collection of independent computers that work together to appear as a single coherent system. The challenge is that:

1. Partial failures are the norm, not the exception
2. Networks are unreliable — messages can be delayed, duplicated, or lost
3. Clocks drift — you can't rely on timestamps across machines

CAP Theorem

Eric Brewer's CAP theorem states that a distributed system can guarantee at most two of three properties:

• Consistency: Every read gets the most recent write
• Availability: Every request gets a response (though not necessarily the most recent data)
• Partition tolerance: The system continues operating despite network partitions

In practice, network partitions happen — so the real choice is between CP (consistency during partitions) and AP (availability during partitions).

Important nuance: CAP is often misapplied. It only applies during network partitions. When the network is healthy, you can have both consistency and availability.

The PACELC Model

A more useful framework is PACELC:

• During a Partition: choose between Availability and Consistency
• Else (no partition): choose between Latency and Consistency

This captures the real engineering tradeoff most systems face.

Consensus

When multiple nodes need to agree on a value (e.g., "who is the leader?"), you need a consensus protocol. The key insight is:

A consensus algorithm must be safe (never returns wrong result) and live (eventually terminates).

Paxos was the first practical consensus algorithm, but it's notoriously hard to understand. Raft was designed specifically to be understandable — and it is.

Raft decomposes consensus into:

1. Leader election — one node becomes the authoritative writer
2. Log replication — the leader replicates entries to followers
3. Safety — committed entries are never overwritten

What Comes Next

This primer barely scratches the surface. The real complexity is in the details:

• How do you handle membership changes?
• What happens when a node rejoins after a long partition?
• How do you build transactions on top of a replicated log?

I'll cover each of these in follow-up posts.

---

If you found this useful, I'd love to hear what you're building — reach out on Twitter.