Distributed Locking: Redis Redlock vs. Zookeeper vs. Database Constraints

Distributed Locking: Coordinating at Scale

In a distributed microservice cluster, multiple instances of a stateless application frequently compete to access a shared, single-writer physical resource. Typical concurrency primitives like Java’s synchronized, Go’s sync.Mutex, or operating-system-level semaphores are bound strictly to a single server's address space. When coordination must span across virtual machines, container pods, and network boundaries, we require a Distributed Lock Service.

Distributed locks prevent race conditions on shared components, such as adjusting a customer's digital ledger balance, processing ticket reservations, or updating dynamic pricing models. However, implementing distributed locking safely under the presence of network partitions, virtual machine pauses, and system clock drift is one of the most notoriously complex challenges in distributed systems engineering.

1. System Requirements & Goals

Before comparing engines, we define the strict specifications of our locking system:

Functional Requirements

• Mutual Exclusion: Only one client can hold the lock on a specific resource at any given time.
• Deadlock Prevention: The system must eventually release the lock even if the holding client crashes, loses network connectivity, or enters a virtual machine freeze loop.
• Lock Reentrancy: The same thread or process should be able to re-acquire the lock without blocking itself (optional but common in LLD frameworks).

Non-Functional Requirements

• Safety (Safety Level 1): The lock manager must guarantee correctness under all asynchronous timing conditions. It is unacceptable for two clients to hold the same lock simultaneously.
• High Availability & Fault Tolerance: The lock manager must remain operational even if a subset of its consensus nodes crash.
• Low Latency: Lock acquisition and release operations must take less than 5ms under medium load.

2. HLD & API Endpoints Request Flow

Clients interact with a lock manager via high-performance RPCs (gRPC) or REST APIs. The request flow differs significantly depending on whether we choose an AP (High Availability) lock manager like Redis or a CP (Strong Consistency) lock manager like Apache ZooKeeper.

sequenceDiagram
    autonumber
    participant ClientA as Client A
    participant Manager as Lock Manager (Redis/ZooKeeper)
    participant ClientB as Client B
    participant Storage as Shared Storage (Database)

    ClientA->>Manager: AcquireLock(resource_id="booking_123", ttl=30000ms)
    Manager-->>ClientA: LockAcquired(token=101)
    
    ClientB->>Manager: AcquireLock(resource_id="booking_123", ttl=30000ms)
    Manager-->>ClientB: LockDenied(AlreadyHeld)
    
    ClientA->>ClientA: Process long task (JVM enters Garbage Collection pause...)
    Note over ClientA: GC Stop-the-World Pause (TTL Expires!)
    Manager->>Manager: Lock TTL Expires / Ephemeral Node Deleted
    
    ClientB->>Manager: Re-try: AcquireLock(resource_id="booking_123")
    Manager-->>ClientB: LockAcquired(token=102)
    
    ClientB->>Storage: Write data (with Fencing Token=102)
    Storage->>Storage: Token 102 > 101 (Write Succeeded)
    Storage-->>ClientB: OK
    
    Note over ClientA: GC ends. Client A wakes up.
    ClientA->>Storage: Write data (with expired Fencing Token=101)
    Storage->>Storage: Reject Token 101 (Stale Token Error!)
    Storage-->>ClientA: Write Failed

RPC API Definition (Protobuf Spec)

syntax = "proto3";

package lockservice;

service LockManager {
  rpc AcquireLock(AcquireLockRequest) returns (AcquireLockResponse);
  rpc ReleaseLock(ReleaseLockRequest) returns (ReleaseLockResponse);
}

message AcquireLockRequest {
  string resource_id = 1;
  string client_id = 2;
  int64 ttl_ms = 3;
}

message AcquireLockResponse {
  bool success = 1;
  string fencing_token = 2; // Monotonically increasing lock generation counter
  int64 expires_at_ms = 4;
}

message ReleaseLockRequest {
  string resource_id = 1;
  string client_id = 2;
  string fencing_token = 3;
}

message ReleaseLockResponse {
  bool success = 1;
}

3. Low-Level Design (LLD) & Storage Engines

We dissect the concrete implementation details across three storage mechanisms: Redis, ZooKeeper, and Relational SQL Database Constraints.

A. Redis Single-Instance Locking (Lua Scripting)

To acquire a lock in Redis safely, we must execute a single atomic operation. We set a unique client identifier as the value so that only the owner can release the lock.

Lock Acquisition Command:

SET booking_123 "client_uuid_456" NX PX 30000

• NX: Set key only if it does not already exist.
• PX 30000: Expire the key in 30,000 milliseconds (lease safety fallback).

Lock Release Lua Script:

We cannot use a simple DEL command because a client might release a lock that it lost due to a lease expiry, deleting a lock acquired by a subsequent client. We must use an atomic Lua script that compares the key value first:

-- Lua script to release lock safely
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end

B. ZooKeeper Ephemeral Nodes (CP Consensus)

ZooKeeper coordinates locking using Ephemeral Sequential Nodes and ZAB (ZooKeeper Atomic Broadcast) consensus. This guarantees absolute correctness under partition failures (CP posture) by eliminating clock drift dependencies.

sequenceDiagram
    autonumber
    participant Client1 as Client 1
    participant Client2 as Client 2
    participant ZK as ZooKeeper Ensemble

    Client1->>ZK: Create ephemeral sequential `/locks/booking_123/lock-`
    ZK-->>Client1: Created: `/locks/booking_123/lock-00000001`
    Client2->>ZK: Create ephemeral sequential `/locks/booking_123/lock-`
    ZK-->>Client2: Created: `/locks/booking_123/lock-00000002`
    
    Client1->>ZK: getChildren("/locks/booking_123")
    ZK-->>Client1: List: [lock-00000001, lock-00000002]
    Note over Client1: Lowest sequence (00000001) -> LOCK ACQUIRED!

    Client2->>ZK: getChildren("/locks/booking_123")
    ZK-->>Client2: List: [lock-00000001, lock-00000002]
    Note over Client2: Sequence 00000002 is not lowest -> LOCK DENIED!
    Client2->>ZK: Register Watcher on `/locks/booking_123/lock-00000001`
    ZK-->>Client2: Watcher Confirmed

    Client1->>ZK: Delete `/locks/booking_123/lock-00000001` (or Client1 disconnects)
    ZK-->>Client2: Node Deletion Event Triggered!
    Client2->>ZK: getChildren("/locks/booking_123")
    ZK-->>Client2: List: [lock-00000002]
    Note over Client2: Lowest sequence (00000002) -> LOCK ACQUIRED!

Detailed Locking Algorithm on ZooKeeper:

1. Node Creation: A client requests creation of an ephemeral sequential node path (e.g., /locks/booking_123/lock-). The sequential flag tells ZK to automatically append a monotonically increasing 10-digit number.
2. Lowest Sequence Check: The client queries all children of /locks/booking_123. If the sequence appended to its node is the lowest, it holds the lock.
3. Avoid Thundering Herd: If its sequence is not the lowest, the client registers a Watcher specifically on the node that has the sequence number immediately preceding its own (e.g., node 002 watches node 001). This ensures that when node 001 is deleted, only node 002 is notified, avoiding wake-up stampedes across thousands of waiting servers.
4. Session Heartbeating: The lock is kept alive via persistent heartbeats between client and ZooKeeper. If the client experiences a crash, or gets partitioned out for longer than the session timeout duration, ZooKeeper automatically destroys the ephemeral node, releasing the lock cleanly.

C. SQL Relational Constraints (DDL & Fencing Schema)

In legacy setups, you can implement high-safety locking utilizing standard database engine guarantees.

-- DDL for Database Distributed Locking
CREATE TABLE distributed_locks (
    resource_name VARCHAR(255) PRIMARY KEY,
    owner_id VARCHAR(255) NOT NULL,
    fencing_token BIGINT NOT NULL,
    expires_at TIMESTAMP WITH TIME ZONE NOT NULL
);

-- Safe Lock Acquisition query
INSERT INTO distributed_locks (resource_name, owner_id, fencing_token, expires_at)
VALUES ('booking_123', 'client_uuid_456', 102, NOW() + INTERVAL '30 seconds')
ON CONFLICT (resource_name) DO UPDATE 
SET owner_id = 'client_uuid_456',
    fencing_token = distributed_locks.fencing_token + 1,
    expires_at = NOW() + INTERVAL '30 seconds'
WHERE distributed_locks.expires_at < NOW();

4. Scaling Challenges & Estimations

Lock Transaction Throughput Sizing

• Target Load: $100,000 \text{ checkout operations/sec}$.
• Lock duration: $10\text{ms}$ average database write lease duration.
• Storage Sizing for Redis Cache Nodes:
  • Active locks at any millisecond: $100,000 \text{ checkout/s} \times 0.01 \text{s} = 1,000 \text{ active concurrent locks}$.
  • Lock key memory size: $256 \text{ Bytes}$ per key payload in Redis memory.
  • RAM consumption: $1,000 \text{ locks} \times 256 \text{ Bytes} \approx 256 \text{ KB}$ (Negligible).
• IOPS Scale:
  • $100,000 \text{ acquisitions/sec} + 100,000 \text{ releases/sec} = 200,000 \text{ operations/sec}$ on the lock manager.
  • Redis Cluster Partition Sharding: We partition the lock resource keys using consistent hashing ({lock:booking_123}) to distribute locks uniformly across 10 cluster shards, with each handling a comfortable $20,000 \text{ IOPS}$.

5. Architectural Trade-offs & Comparisons

Choosing the right distributed lock engine is a direct choice between Performance (Latency) and Safety (Correctness).

6. Failure Scenarios, Resiliency & Mitigations

A. The "Stop-the-World" Garbage Collection Pause

Suppose Client A acquires a Redis lock with a 30-second TTL. The client immediately enters a major JVM Garbage Collection pause or system-level VM migration freeze that lasts 45 seconds. During this pause, the lock expires in Redis. Client B acquires the lock. Client A wakes up, unaware that it has lost the lock, and writes corrupted changes to the database.

• Mitigation (Fencing Tokens): A fencing token is a monotonically increasing counter returned with every lock acquisition. The shared storage engine (e.g. database table) tracks the latest active fencing token. When a client performs a write, it includes its token. If the database receives a write with a token lower than the latest recorded, it aborts the operation.

B. The Redlock Clock-Drift Vulnerability (Kleppmann Controversy)

Redlock relies on the assumption that independent Redis masters expire keys uniformly. However, system clocks drift due to NTP synchronization jumps or hardware anomalies. If a NTP step-change dynamically alters the clock on two of the five Redis masters, a key can expire early, leading to multiple clients holding the lock.

• Mitigation: Use a consensus-backed distributed coordinator like ZooKeeper or etcd for mission-critical operations where correctness is non-negotiable (such as moving bank funds).

7. Staff Engineer Perspective: Production Resiliency

[!CAUTION] Lease Extension (The Keep-Alive Loop): When performing operations with unpredictable latency (like generating massive media files), you should never guess the lock TTL. Under-estimating it causes dirty concurrency states; over-estimating it blocks other workers for hours if a node crashes.

To resolve this, always implement a background keep-alive heartbeating loop (often called a lock renewer or watchdog). When the lock is acquired, spawn a lightweight background fiber/thread. Every $\frac{1}{3}$ of the TTL duration, the worker checks if the parent thread is still alive and issues an atomic Lua script to extend the TTL in the background.

// JavaScript Keep-Alive Watchdog renewal loop blueprint
async function startWatchdog(lockKey, clientId, ttlMs) {
  const interval = setInterval(async () => {
    const success = await redis.eval(`
      if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("pexpire", KEYS[1], ARGV[2])
      else
        return 0
      end
    `, 1, lockKey, clientId, ttlMs);

    if (!success) {
      clearInterval(interval);
      throw new Error("Lock lease renewal failed - Lock lost!");
    }
  }, ttlMs / 3);
  return () => clearInterval(interval);
}

8. Verbatim Mock Interview Script

Interviewer Dialogue

• Interviewer: "You mentioned using Redis's Redlock. If a network partition occurs and splits the cluster, can two clients acquire the same lock under Redlock?"

• Candidate: "Yes, under specific edge cases, Redlock can fail to guarantee mutual exclusion. Redlock requires a client to acquire locks from a majority (e.g., 3 out of 5) of independent Redis masters. If a network partition isolates Client A with 3 masters, and Client B with the other 2, Client B will fail to acquire the lock because it cannot reach a majority.

However, if a master node in Client A's partition crashes and restarts without durable disk persistence enabled (AOF), it will wake up with an empty memory space. If the partition heals, Client B can now request the lock, and because the restarted master has lost its state, it will grant the lock to Client B. Client B can now achieve a majority of 3 nodes, resulting in both Client A and Client B holding the lock simultaneously.

For high-integrity workloads, I would steer away from Redlock and utilize etcd or ZooKeeper, which rely on the Raft or ZAB consensus protocols. These engines enforce strict monotonic log indexing and state replication that prevent restarted nodes from losing transaction state."

• Interviewer: "What is a fencing token, and why is it necessary if our lock manager is 100% correct?"

• Candidate: "A fencing token is a monotonic counter returned upon lock acquisition that acts as a downstream verification guard. Even if we use a mathematically perfect CP lock manager like ZooKeeper, the client itself is still subject to asynchronous runtime pauses (like a JVM Stop-the-World GC pause, OS hypervisor context switching, or network interface queue buffers).

If a client acquires the lock, verifies its lease, and then pauses for 10 seconds, the lock manager will detect a session timeout and release the lock. Another client acquires it. When the first client wakes up, there is no code-level check that can prevent it from sending its already-computed database write command across the socket.

By passing a fencing token (e.g., token 102) with the database write, the target database can reject the command because it has already processed a write with token 103 from the second client. Thus, fencing tokens protect the system from client-side failures rather than lock manager failures."

Read Next

• System Design: Designing a Distributed Key-Value Store (RocksDB / Cassandra)
• System Design: Building a Collaborative Editor (Google Docs Internals)
• Consistent Hashing: The Secret Sauce of Distributed Scalability