Distributed Locks

Overview

A distributed lock lets multiple processes on different machines agree that only one of them may act on a shared resource at a time. Unlike an in-process mutex, it must survive crashes, network delays, and the reality that the lock holder might freeze at the worst possible moment - so correct designs use expiring leases and fencing tokens, not just a flag in a shared store.

🧠 Mental model: A distributed lock is the single key to a meeting room in a building with unreliable phones. You must be able to reclaim the key if someone walks off with it (a lease that expires), and the door must reject an old key after the locks are changed (a fencing token), so a forgetful ex-holder can't wander back in.

Key Concepts

A safe distributed lock needs three properties: mutual exclusion (at most one holder), deadlock freedom (a crashed holder's lock is eventually released), and fault tolerance (the lock service itself survives node loss).

Leases provide deadlock freedom: a lock is granted for a bounded time and auto-expires, so a dead holder cannot block others forever. But leases create a danger - the holder may not know its lease expired.

Fencing tokens solve that danger. Each lock grant returns a monotonically increasing number. The protected resource records the highest token it has seen and rejects any write carrying a smaller one, so a stalled old holder that wakes up is fenced off.

Locks are typically built on a strongly consistent coordination service that already solves distributed consensus and leader election:

Backing store	Mechanism	Notes
ZooKeeper	Ephemeral sequential znodes	Auto-release on session loss; battle-tested
etcd	Leases + compare-and-swap	Used by Kubernetes for coordination
Redis (single)	`SET key val NX PX ttl`	Fast but not safe under failover without care
Redis (Redlock)	Quorum across N masters	Debated; needs fencing for correctness

Whenever a lock exists mainly to prevent duplicate side effects, prefer making the operation idempotent so a rare double-execution is harmless.

Trade-offs

A lock on a strongly consistent store (ZooKeeper, etcd) is safe but adds latency and a dependency on a CP system that may reject requests during a partition. A lock on Redis is fast and simple but trades safety: under failover a lock can be granted twice, so it needs fencing tokens or an idempotent resource to stay correct. The deeper trade-off is philosophical - distributed locks are expensive and fragile, so the strongest designs avoid needing them by partitioning work (each key owned by one worker) or by making operations idempotent.

Interview Tips

Never propose a bare "SET NX in Redis" lock without mentioning lease expiry and fencing tokens.
Say the safest lock is the one you avoid: partition ownership or idempotency often removes the need entirely.
For strong safety, name ZooKeeper/etcd; for speed with caveats, name Redis and state the risk.
Distinguish a lock (mutual exclusion) from leader election (a durable single leader) - related but not the same.

Summary

A distributed lock enforces single-holder access to a resource across machines.
It must provide mutual exclusion, deadlock freedom (via expiring leases), and fault tolerance.
Leases prevent dead holders from blocking forever but let stalled holders act late.
Fencing tokens (monotonic numbers the resource enforces) are what make locks actually safe.
Prefer avoiding locks via partitioned ownership or idempotency; use ZooKeeper/etcd when you need real safety.