A distributed lock looks deceptively simple.

You have multiple instances of a service, but only one of them should perform a particular operation at a time.

So you put a key in Redis:

lock:invoice:123 = worker-1

If the key exists, another worker waits.

If it does not exist, the worker creates it and starts processing.

Problem solved.

Except it isn't.

The difficult part of distributed locking is not acquiring the lock.

The difficult part is answering questions like:

What happens if the process crashes?
What happens if the lock expires while the process is still running?
What happens if another worker acquires the lock?
What happens if the original worker continues executing?
What happens if the network connection disappears temporarily?
Can a worker accidentally release another worker's lock?
What happens when the protected operation takes longer than the lease?
Can the system prevent a stale worker from modifying the resource?

These are not edge cases.

They are the reason distributed locking is fundamentally different from using sync.Mutex.

A Distributed Lock Is Not a `sync.Mutex` in Redis

A Go mutex protects shared memory inside one process.

var mu sync.Mutex func update() { mu.Lock() defer mu.Unlock() // critical section }

Every goroutine using that mutex is part of the same process and shares the same memory.

A distributed lock operates across:

 ┌───────────────┐ │ Redis │ └───────┬───────┘ │ ┌─────────┴─────────┐ │ │ Worker A Worker B Instance 1 Instance 2

The workers do not share memory.

They communicate through a remote system.

That introduces failure modes that a local mutex does not have:

process crash network partition packet delay Redis failover clock differences lease expiration long-running operations stale clients

Redis itself documents the basic single-instance pattern as an atomic SET with NX and an expiration, using a unique random value for ownership.

That distinction — ownership over a remote lease rather than a local mutex — is the foundation for everything that follows.

The Basic Redis Lock

The simplest useful primitive looks like this:

SET resource_name random_token NX PX 30000

The important parts are:

NX: only create the key if it does not exist
PX 30000: give it a 30-second expiration
random_token: uniquely identifies the owner

Redis supports combining the conditional set and expiration in a single SET command. SETNX itself is considered deprecated for new locking patterns in favor of SET ... NX.

In Go, using go-redis:

type Lock struct { client *redis.Client key string ttl time.Duration } func (l *Lock) Acquire(ctx context.Context, token string) error { ok, err := l.client.SetNX( ctx, l.key, token, l.ttl, ).Result() if err != nil { return err } if !ok { return ErrLockNotAcquired } return nil }

Usage:

token := uuid.NewString() if err := lock.Acquire(ctx, token); err != nil { return err } defer func() { releaseCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() if err := lock.Release(releaseCtx, token); err != nil { log.Printf("lock release failed: %v", err) } }() return processInvoice(ctx)

Use a bounded context for release and report failures rather than silently
discarding them. The TTL remains the recovery mechanism if release cannot
complete.

There is also an ambiguous-outcome case: Redis may apply SET while the reply
is lost or times out. An acquisition error therefore does not always prove
that no lock was created. Do not blindly retry with a new ownership token; that
can leave the first lease held until it expires. Generate the token before the
command and retain it. If recovering from an ambiguous result, an owner check
must also account for the remaining lease time; a matching GET alone cannot
guarantee ownership after the check. Otherwise treat the result as unknown and
rely on the lease to expire.

At first glance this looks good.

But there is already a subtle bug waiting for us.

Never Release a Lock You Don't Own

Imagine this timeline.

Worker A: acquire lock token = "aaa" TTL = 10 seconds Worker A: starts processing 10 seconds later: lock expires Worker B: acquires lock token = "bbb" Worker A: finishes processing Worker A: DEL lock

That final DEL is dangerous.

Worker A no longer owns the lock.

Worker B does.

If Worker A blindly deletes the key, it just deleted Worker B's lock.

This is why the lock value needs to identify the owner.

The release operation must effectively mean:

if current_owner == my_token: delete lock

And that check must be atomic.

Not:

owner, _ := rdb.Get(ctx, key).Result() if owner == token { rdb.Del(ctx, key) }

Because another worker can acquire the lock between those two operations.

You need one atomic operation.

A Lua script is a straightforward way to express it:

if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0

In Go:

var releaseScript = redis.NewScript(` if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0 `) func (l *Lock) Release(ctx context.Context, token string) error { result, err := releaseScript.Run( ctx, l.client, []string{l.key}, token, ).Int() if err != nil { return err } if result == 0 { return ErrLockNotOwned } return nil }

This gives us an important invariant:

A worker may release the lock only if the lock still contains its ownership token.

Redis documents this owner-checking release pattern explicitly.

That solves one class of race.

It does not solve the hardest one.

The Dangerous Part: Your Work Can Outlive Your Lock

Consider a job that normally takes around three seconds.

You configure:

lock TTL = 10 seconds

That sounds reasonable.

Until one day the downstream database becomes slow.

The job takes 15 seconds.

Now:

T=0s Worker A acquire lock └── TTL = 10s T=8s Worker A still processing T=10s Redis lock expires T=11s Worker B acquires lock T=12s Worker A still processing T=15s Worker A writes result

We now have two workers performing the supposedly exclusive operation.

The lock did exactly what we asked.

It prevented another worker from acquiring the lock while the lease was valid.

The mistake was assuming that the lock guaranteed ownership for the entire lifetime of the operation.

It doesn't.

A TTL-based distributed lock is better understood as a lease.

The ownership is valid only for a bounded period.

Why TTLs Are Necessary

You might ask:

Why not just remove the expiration?

Because then a crashed process can leave the system permanently locked.

Worker A acquire lock Worker A crash 💥 Redis lock remains forever

Every other worker now sees:

lock exists

and waits forever.

The TTL solves the dead-owner problem:

Worker A acquire lock Worker A crash 💥 TTL expires Worker B acquire lock

But TTL introduces another problem:

The lease can expire while the owner is still alive.

Distributed locking is therefore a trade-off between two failure modes:

No TTL ↓ dead worker can block everyone forever TTL ↓ live but slow worker can become stale

This is why choosing a TTL is not simply:

"How long does this job normally take?"

It is a correctness decision.

Lock Renewal

For long-running work, one option is to renew the lease.

For example:

TTL = 30 seconds renew every 10 seconds

The worker periodically verifies that it still owns the lock and extends the expiration.

Conceptually:

if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("PEXPIRE", KEYS[1], ARGV[2]) end return 0

The important detail is the ownership check.

Never do:

PEXPIRE lock 30000

without verifying ownership.

Otherwise a stale worker could extend a lock that now belongs to another worker.

A renewal loop might look like:

func (l *Lock) RenewLoop( ctx context.Context, token string, ) <-chan error { errors := make(chan error, 1) go func() { defer close(errors) ticker := time.NewTicker(l.ttl / 3) defer ticker.Stop() for { select { case <-ctx.Done(): return case <-ticker.C: ok, err := l.Renew(ctx, token) if err != nil { errors <- err return } if !ok { errors <- ErrLockLost return } } } }() return errors }

But renewal doesn't magically make the system safe.

You still have to define what happens when renewal fails.

For example:

Worker A lock acquired ↓ processing ↓ renewal fails ↓ lock may expire ↓ Worker B acquires lock

What should Worker A do?

The safe answer is generally:

Stop treating yourself as the owner.

This is where context.Context becomes useful. Go's context model is specifically designed to propagate cancellation and deadlines across goroutines and API boundaries.

For example:

workCtx, cancel := context.WithCancel(ctx) defer cancel() renewErrors := lock.RenewLoop(workCtx, token) workDone := make(chan error, 1) go func() { workDone <- processInvoice(workCtx) }() select { case workErr := <-workDone: cancel() return workErr case err, ok := <-renewErrors: cancel() if !ok { return errors.New("lock renewal stopped unexpectedly") } if err != nil { return fmt.Errorf("lock lost: %w", err) } return errors.New("lock renewal stopped unexpectedly") case <-ctx.Done(): cancel() return ctx.Err() }

This assumes processInvoice observes cancellation. If it cannot stop promptly,
downstream writes still need fencing.

The exact implementation will depend on how the work is structured, but the principle is more important:

Losing the lease should become a first-class state in the worker lifecycle.

But What If the Worker Cannot Stop Immediately?

This is the question that takes distributed locking from "Redis tutorial" to distributed-systems engineering.

Imagine:

Worker A token = 41 lock expires Worker B token = 42 acquires lock Worker A continues running Worker A writes to database

Even if Worker A knows it lost the lock, there may be no way to instantly stop every operation.

Maybe it is blocked inside:

db.ExecContext(...)

Maybe a network request is already in flight.

Maybe the process is paused.

Maybe the runtime is under heavy CPU pressure.

Maybe the operation cannot be interrupted.

So cancellation alone isn't enough.

We need another mechanism.

Fencing Tokens

This is where fencing tokens become extremely useful.

Instead of treating the lock token only as an ownership identifier, every successful acquisition receives a monotonically increasing number.

For example:

Worker A → fencing token 41 Worker B → fencing token 42

Now the downstream resource can reject stale operations.

Suppose Worker A loses its lease:

Worker A token = 41 Worker B token = 42

Worker A tries:

WRITE resource WITH token 41

The resource knows:

latest token = 42

Therefore:

41 < 42

Reject it.

Worker B's operation:

WRITE resource WITH token 42

is accepted.

The key idea is:

The lock determines who may start the work. The fencing token determines whether an operation is still authorized to affect the resource.

This distinction is critical for long-running distributed operations. Redis's own distributed-lock documentation explicitly calls out fencing tokens as important for processes whose work can take significant time.

Generating Fencing Tokens

A monotonically increasing token can be generated with Redis INCR, but it
must be tied atomically to successful lock acquisition. Incrementing in a
separate command can reverse the intended order: an old worker can pause after
acquiring its lease, lose it, then increment after the new owner and receive a
higher token.

For a single Redis primary, one Lua script can perform both operations:

local acquired = redis.call( "SET", KEYS[1], ARGV[1], "NX", "PX", ARGV[2] ) if not acquired then return 0 end return redis.call("INCR", KEYS[2])

The caller generates and retains the random ownership token in ARGV[1]; a
positive result is the fencing token, while 0 means the lease was not
acquired. The lock key and counter key must be distinct, and the counter must
remain numeric.

Atomic execution on one primary does not make Redis failover strongly
consistent. With asynchronous replication, a failover can lose a successful
lock or counter write, allowing overlapping owners or a token to move
backward. If stale writes would violate a critical invariant, use a coordinator
and downstream enforcement whose durability and ordering guarantees meet that
invariant; a Lua script alone does not provide that guarantee.

Then:

Worker A → 41 Worker B → 42 Worker C → 43

But generating the token is only half of the solution.

The downstream system must actually enforce it.

For example, suppose we store the latest fencing token with the protected resource:

UPDATE invoices SET status = $1, fencing_token = $2 WHERE id = $3 AND fencing_token <= $2;

The column should be NOT NULL with an initial value below the first issued
token. The inclusive comparison permits multiple writes by the same lease;
the stored value still advances when a newer token arrives.

Now a stale worker cannot overwrite a newer state.

A more complete example:

func updateInvoice( ctx context.Context, db *sql.DB, invoiceID string, status string, fenceToken int64, ) error { result, err := db.ExecContext(ctx, ` UPDATE invoices SET status = $1, fencing_token = $2 WHERE id = $3 AND fencing_token <= $2 `, status, fenceToken, invoiceID, ) if err != nil { return err } rows, err := result.RowsAffected() if err != nil { return err } if rows == 0 { return ErrStaleWorker } return nil }

This permits more than one update by the same owner token, while rejecting an
older token after the resource has recorded a newer one. Fencing is not
instantaneous revocation: if the old worker's write reaches the resource before
any write carrying the new token, the resource has not yet observed the newer
token and cannot reject the old one on that basis alone.

Now consider:

Worker A → token 41 Worker B → token 42

Worker B updates first:

fencing_token = 42

Worker A eventually wakes up:

UPDATE ... WHERE fencing_token <= 41

That condition is false.

The stale operation is rejected.

This is fundamentally stronger than trusting the lock's TTL.

Lock Tokens and Fencing Tokens Are Different

It is useful to keep these concepts separate.

A lock ownership token answers:

"Does this client currently own this lease?"

A fencing token answers:

"Is this operation newer than the operations that came before it?"

For example:

Lock token: "550e8400-e29b-41d4-a716-446655440000" Fencing token: 42

The first is useful for safe release:

GET lock == my-token

The second is useful for downstream ordering:

incoming-token >= stored-token

They solve different problems.

A More Complete Lock Lifecycle

A production workflow might therefore look like this:

 ┌──────────────────┐ │ Acquire lease │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Get fence token │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Start work │ └────────┬─────────┘ │ ┌────────────┴────────────┐ │ │ ▼ ▼ renew succeeds renewal fails │ │ │ ▼ │ cancel work │ │ ▼ ▼ continue work stop writes │ ▼ fenced downstream writes │ ▼ owner-checked release

This is much closer to a real distributed coordination mechanism than simply:

SET lock doWork() DEL lock

PostgreSQL Can Sometimes Be a Better Lock Manager

Redis is not the only option.

If the resource you're coordinating already lives in PostgreSQL, adding Redis solely for locking can introduce unnecessary infrastructure.

For example, database row locking can provide transactional coordination:

SELECT id FROM jobs WHERE status = 'pending' ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1;

This can be extremely useful for worker queues backed by PostgreSQL.

The database already knows:

who owns the row what transaction owns it when the transaction ends

You may not need a distributed lock at all.

PostgreSQL advisory locks are another option when the coordination is conceptually tied to the database.

The broader design question should therefore be:

Where does ownership already live?

If your work is already represented as a database row, using a separate Redis lock may be unnecessary complexity.

Distributed Lock vs Queue Ownership

Sometimes the correct solution isn't a lock.

Suppose you have:

10 workers 100,000 jobs

You might initially think:

lock job process job unlock job

But a queue can model ownership directly.

For example:

Job 123 → Worker A Job 124 → Worker B Job 125 → Worker C

The system is not asking:

"Who currently owns this global lock?"

It is asking:

"Who owns this piece of work?"

That distinction often produces simpler systems.

Locks are especially attractive when multiple workers need to coordinate access to the same shared resource.

Queues are often more natural when the problem is work distribution.

Distributed Lock vs Leader Election

These are also different concepts.

A distributed lock might say:

Only one worker should process this resource right now.

Leader election says:

One instance should act as the active controller.

For example:

Worker A Worker B Worker C

might elect:

Worker B = leader

Worker B then performs:

scheduler configuration refresh control-plane reconciliation

This is not necessarily the same problem as:

lock:customer:123

Trying to solve both problems with the same primitive often creates unnecessary complexity.

Context-Aware Lock Acquisition

Lock acquisition should also respect the caller's deadline.

This is especially important for HTTP handlers and request-driven workflows.

Imagine:

HTTP timeout = 2 seconds lock acquisition timeout = 30 seconds

A request might spend the entire 2 seconds waiting for a lock and then fail anyway.

Instead:

func (l *Lock) AcquireWithRetry( ctx context.Context, ) (string, error) { delay := 50 * time.Millisecond const maxDelay = time.Second for { token := uuid.NewString() err := l.Acquire(ctx, token) if err == nil { return token, nil } if !errors.Is(err, ErrLockNotAcquired) { return token, err } jitter := time.Duration(rand.Int63n(int64(delay / 2))) timer := time.NewTimer(delay + jitter) select { case <-timer.C: if delay < maxDelay { delay *= 2 if delay > maxDelay { delay = maxDelay } } case <-ctx.Done(): timer.Stop() return "", ctx.Err() } } }

The exact bounds are workload-dependent. Exponential backoff with jitter avoids
having contending workers retry in lockstep. The important part is that waiting
for ownership is part of the operation's deadline.

Go's context package is designed to propagate cancellation and deadlines through a chain of operations, including across goroutines and external calls.

Don't Create a Lock That Lives Forever

A tempting design is:

Acquire lock Renew forever

This can be dangerous.

Imagine the renewal goroutine survives while the actual worker is stuck.

You can accidentally create a lock that remains alive even though the operation is no longer making progress.

The lifecycle should therefore be explicit:

worker starts ↓ lock acquired ↓ renewal starts ↓ work starts ↓ work finishes / fails / is canceled ↓ renewal stops ↓ lock released

The renewal mechanism should belong to the same lifecycle as the work it protects.

This is one of the reasons structured concurrency principles are useful even when implementing distributed coordination.

What About Redlock?

Redis also documents the Redlock algorithm for using multiple independent Redis masters.

The idea is roughly:

Redis A ──┐ Redis B ──┤ Redis C ──┼── acquire majority Redis D ──┤ Redis E ──┘

The goal is to reduce dependence on a single Redis instance.

But Redlock is not a magic word that turns distributed coordination into a formally solved problem.

There has been substantial discussion around its safety assumptions, timing model, failure modes, and whether it provides the guarantees applications actually need.

Redis's own documentation notes that the analysis of Redlock should be considered carefully, including clock behavior and the use of fencing tokens.

The important engineering lesson is:

Don't choose Redlock because "distributed locks require Redlock."

Choose a coordination mechanism based on the consistency guarantees your application actually needs.

If stale work can corrupt data, you need to think about stale work independently of the lock algorithm.

Observability Matters

A distributed lock without metrics is difficult to operate.

At minimum, I would want to know:

lock_acquisition_attempts lock_acquisition_failures lock_acquisition_latency lock_hold_duration lock_renewal_failures lock_release_failures lock_lost_events stale_operation_rejections

For example:

Source：https://dev.to/serifcolakel/distributed-locks-in-go-correctness-failure-modes-and-production-patterns-4mdg

My favorites

PreviousJev: What Happens When AI Stops Generating and Starts Deciding?NextLaya a decision making system one model.