Most backend services run as several identical copies behind a load balancer, so they can take more traffic and survive a crashed server. Each request reads some data, decides what to do and writes the result back. Inside one process, you'd guard that read-decide-write with a mutex. Across servers there's no shared memory to put a mutex in, so the guard has to live somewhere every copy can reach, such as a database row or a Redis distributed lock.
The gap shows up the first time two requests touch the same record at once. A user taps "Pay" twice on a slow connection, or a mobile client retries a payment the moment the first request times out. Two requests for the same wallet land on two different servers within the same millisecond. Both read a balance of 100, both decide 10 is affordable, and both write back 90. The user spent 20, you recorded 10, and nothing in the logs looks wrong.
The standard answer is a distributed lock : one shared place every server can reach that gives one request at a time exclusive access to a record. A database row lock, etcd or ZooKeeper can play that role; Redis is a popular choice because it's fast and often already in the stack. The naive version, one SetNX ("set if not exists": create the key only if nobody holds it) before the work and one Del (delete the key) after, breaks quietly: a crash leaves a lock that never expires, or a slow request deletes someone else's.
A lock kept in Redis is a lease, a key that expires after a set time whether or not you've finished. Take it with SET NX PX (set the key only if it doesn't exist, with an expiry in milliseconds) and a random token, release it only if the token matches, and keep the work shorter than the TTL (time to live, how long the key survives before Redis deletes it). When money is involved, also make the database refuse a write computed from a stale read; no lease can prove you still hold it. If you own the roadmap, the correctness work is a version column and a conditional UPDATE in the database you already run.
In this tutorial, you'll build a distributed lock on Redis and implement it in Go. You'll reproduce the race, fix it by hand, break the fix with a short TTL, switch to the bsm/redislock library, add fencing tokens that PostgreSQL checks, and load-test the result. Everything ran against Redis 8.10.2 and PostgreSQL 18, with go-redis v9.23.0 and bsm/redislock v0.10.0.
Prerequisites
To follow along, you need:
- Go 1.26 or later, which go-redis v9.23.0 requires. With the default
GOTOOLCHAIN=auto,go getdownloads a newer toolchain if yours is older. - Docker, to run Redis and PostgreSQL locally.
- Working knowledge of goroutines,
context.Contextand SQL. No prior Redis experience is needed. - Ports 6379 and 5432 free, or adjust the
-pmappings and addresses to match.
Step 1: Reproduce the Race Without a Lock
This step reproduces the double payment on your machine, so every later fix has a baseline. One wallet balance lives in Redis, and 50 goroutines, each a "Pay" request on a different server, try to spend 10 from it at once. Predict how many will believe they can afford it.
Start Redis in a container and check that it answers:
docker run -d --name redis -p 6379:6379 redis:8.10.2 docker exec redis redis-cli ping Output
PONG Create a module and add go-redis, bsm/redislock and pgx, the PostgreSQL driver for Step 5. The second output line only appears if your local Go is older than 1.26:
mkdir walletlock && cd walletlock go mod init example.com/walletlock go get github.com/redis/go-redis/v9@v9.23.0 github.com/bsm/redislock@v0.10.0 github.com/jackc/pgx/v5@v5.11.0 Output
go: creating new go.mod: module example.com/walletlock go: github.com/redis/go-redis/v9@v9.23.0 requires go >= 1.26.0; switching to go1.26.9 go: upgraded go 1.24.4 => 1.26.0 go: added github.com/bsm/redislock v0.10.0 go: added github.com/jackc/pgx/v5 v5.11.0 go: added github.com/redis/go-redis/v9 v9.23.0 Spend is a read-modify-write: it reads the balance, checks it and writes the new value back, so its decision rests on a value that can change before the write lands. The sleep stands in for the real work in between.
wallet/wallet.go
// Package wallet holds a demo balance in Redis and spends from it the naive way. package wallet import ( "context" "fmt" "os" "sync" "time" "github.com/redis/go-redis/v9" ) const BalanceKey = "wallet:42:balance" // Connect returns a client for REDIS_ADDR, or localhost:6379 by default. func Connect() *redis.Client { addr := os.Getenv("REDIS_ADDR") if addr == "" { addr = "localhost:6379" } return redis.NewClient(&redis.Options{ Addr: addr, // Let context deadlines cut socket reads short. Without this, // go-redis waits out its own 5-second ReadTimeout instead. ContextTimeoutEnabled: true, }) } // Spend is a read-modify-write: read the balance, check it, write it back. // The sleep stands in for real work between the read and the write // (validation, a fraud check, a call to another service). func Spend(ctx context.Context, rdb *redis.Client, amount int64, work time.Duration) (bool, error) { bal, err := rdb.Get(ctx, BalanceKey).Int64() if err != nil { return false, err } if bal < amount { return false, nil } time.Sleep(work) return true, rdb.Set(ctx, BalanceKey, bal-amount, 0).Err() } // Hammer starts n goroutines at the same moment and counts how many // calls to fn reported success. func Hammer(n int, fn func() (bool, error)) (accepted int) { var wg sync.WaitGroup var mu sync.Mutex start := make(chan struct{}) for range n { wg.Add(1) go func() { defer wg.Done() <-start ok, err := fn() if err != nil { fmt.Println("error:", err) return } if ok { mu.Lock() accepted++ mu.Unlock() } }() } close(start) wg.Wait() return accepted } // Check compares the final balance with what the accepted spends imply. func Check(initial, amount int64, accepted int, balance int64) { want := initial - amount*int64(accepted) status := "OK" if balance != want || balance < 0 { status = "BROKEN" } fmt.Printf("accepted=%d balance=%d expected=%d %s\n", accepted, balance, want, status) } Hammer releases every goroutine at once by closing a channel they all wait on. Check is the yardstick for every later step: 10 spends of 10 from 100 must leave 0.
The first program spends with no lock at all:
cmd/step1-race/main.go
package main import ( "context" "log" "time" "example.com/walletlock/wallet" ) func main() { ctx := context.Background() rdb := wallet.Connect() defer rdb.Close() if err := rdb.Set(ctx, wallet.BalanceKey, 100, 0).Err(); err != nil { log.Fatal(err) } // 50 requests try to spend 10 each from a balance of 100. accepted := wallet.Hammer(50, func() (bool, error) { return wallet.Spend(ctx, rdb, 10, 2*time.Millisecond) }) balance, _ := rdb.Get(ctx, wallet.BalanceKey).Int64() wallet.Check(100, 10, accepted, balance) } go run ./cmd/step1-race Output
accepted=50 balance=80 expected=-400 BROKEN accepted=50 balance=70 expected=-400 BROKEN accepted=50 balance=60 expected=-400 BROKEN Three of five runs are shown. All 50 requests were accepted, so 500 was spent from 100, yet the balance only dropped to 60 to 80: the ledger shows 50 approved payments and a wallet charged for two to four. Each goroutine read 100, decided it could pay and wrote its own new balance over the others. That failure is a lost update , and the rest of this tutorial fixes it.
Go's race detector doesn't catch it:
go run -race ./cmd/step1-race Output
accepted=50 balance=70 expected=-400 BROKEN There's no WARNING: DATA RACE and the exit code is 0. The race detector watches memory inside one Go process, and this shared state lives in Redis. The reliable test is the one Check does: compare the final balance with what the accepted operations imply.
Step 2: Write a Correct Lock by Hand
This step builds the lock: a way for one request to announce "I'm working on wallet 42, wait your turn" where every server can see it. Picture a sign on a door: whoever hangs "busy" first gets the room. In Redis the sign is a key, and it needs two properties: it disappears by itself if the holder crashes, and only the holder may take it down.
Open a Redis shell, take the lock as caller A, try it as caller B, and check A's remaining time:
docker exec -it redis redis-cli redis-cli
127.0.0.1:6379> SET lock:wallet:42 token-a NX PX 30000 OK 127.0.0.1:6379> SET lock:wallet:42 token-b NX PX 30000 (nil) 127.0.0.1:6379> PTTL lock:wallet:42 (integer) 29884 NX sets the key only if it doesn't exist, so B gets (nil). PX 30000 attaches a 30-second expiry in the same command, which makes the lock the lease from the introduction: if the holder crashes, Redis removes the key. The value is a token unique to this caller, as on the official Redis distributed locks page.
Both must go in one command. The common SETNX then EXPIRE is two, and a crash between them leaves a lock that never expires. The SETNX page still documents an older SETNX-based lock only because existing code links to it, and points you to SET with a Lua release instead; the SET page even discourages the single-command lock in favor of Redlock, covered in the safety section.
The token matters on release. If A finishes late, its lease may have passed to B, and deleting the key by name would remove B's lock:
Using just DEL is not safe as a client may remove another client's lock.
Redis runs a Lua script atomically, so a script can compare the token and delete the key with nothing in between. Try a release with the wrong token, then the right one:
redis-cli
127.0.0.1:6379> EVAL 'if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0' 1 lock:wallet:42 token-b (integer) 0 127.0.0.1:6379> EVAL 'if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0' 1 lock:wallet:42 token-a (integer) 1 B's release returns 0 and leaves A's lock alone; A's returns 1. The key goes in KEYS, as the scripting docs require for every key a script touches. Exit the shell with exit.
In Go, Acquire hangs the sign, retrying until it succeeds or the context ends, and Release is the token-checked compare-and-delete:
lock/lock.go
// Package lock is a single-node Redis lease: SET NX PX plus a random token. package lock import ( "context" "crypto/rand" "errors" "fmt" mrand "math/rand/v2" "time" "github.com/redis/go-redis/v9" ) var ( ErrNotHeld = errors.New("lock not held (expired or taken by someone else)") ErrBadTTL = errors.New("lock TTL must be at least 1ms") ) // Delete the key only if it still holds our token. var releaseScript = redis.NewScript(` if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0`) type Lock struct { rdb *redis.Client Key string Token string Fence int64 // set by AcquireFenced, zero otherwise } // Acquire retries SET key token NX PX ttl until it wins or ctx ends. func Acquire(ctx context.Context, rdb *redis.Client, key string, ttl time.Duration) (*Lock, error) { // A zero or negative TTL would create a lock that never expires. if ttl < time.Millisecond { return nil, ErrBadTTL } token := rand.Text() // 26 random base32 characters for { ok, err := rdb.SetNX(ctx, key, token, ttl).Result() if err != nil { return nil, err } if ok { return &Lock{rdb: rdb, Key: key, Token: token}, nil } // Busy: wait 5 to 15 ms, randomized so waiters don't retry in lockstep. wait := 5*time.Millisecond + mrand.N(10*time.Millisecond) select { case <-ctx.Done(): return nil, fmt.Errorf("acquire %s: %w", key, ctx.Err()) case <-time.After(wait): } } } // Release deletes the lock only if we still own it. func (l *Lock) Release(ctx context.Context) error { n, err := releaseScript.Run(ctx, l.rdb, []string{l.Key}, l.Token).Int() if err != nil { return err } if n == 0 { return ErrNotHeld } return nil } rand.Text (Go 1.24 and later) returns at least 128 bits of randomness. releaseScript.Run falls back from EVALSHA (run a script Redis has cached, by its hash) to EVAL (send the whole script) when a restart or failover has emptied the volatile script cache. While the lock is busy, Acquire waits a random 5 to 15 ms; without that randomness, waiters retry in lockstep, and Step 4 measures the cost.
A zero TTL means no expiry
In go-redis, SetNX with a TTL of 0 or redis.KeepTTL sends no expiry, so a new lock never expires. An unset config value can do this silently, which is why Acquire rejects anything under 1 ms.
Wrap the spend in the lock and run it:
cmd/step2-lock/main.go
package main import ( "context" "log" "time" "example.com/walletlock/lock" "example.com/walletlock/wallet" ) const lockKey = "lock:wallet:42" func main() { ctx := context.Background() rdb := wallet.Connect() defer rdb.Close() if err := rdb.Set(ctx, wallet.BalanceKey, 100, 0).Err(); err != nil { log.Fatal(err) } accepted := wallet.Hammer(50, func() (bool, error) { // Give up waiting for the lock after 5 seconds. actx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() l, err := lock.Acquire(actx, rdb, lockKey, 2*time.Second) if err != nil { return false, err } defer func() { if err := l.Release(ctx); err != nil { log.Println("release:", err) } }() return wallet.Spend(ctx, rdb, 10, 2*time.Millisecond) }) balance, _ := rdb.Get(ctx, wallet.BalanceKey).Int64() wallet.Check(100, 10, accepted, balance) } go run ./cmd/step2-lock Output
accepted=10 balance=0 expected=0 OK accepted=10 balance=0 expected=0 OK accepted=10 balance=0 expected=0 OK Exactly 10 spends were accepted and the balance was 0 on all five runs (three shown). The lock turned the burst into a queue, so the double tap now ends as the user expects: the second request waits, reads 90 and pays from that. Confirm no lock was left behind:
docker exec -it redis redis-cli EXISTS lock:wallet:42 Output
(integer) 0 On Redis 8.4 and later, DELEX does the compare-and-delete natively and the Redis docs recommend it there; go-redis v9.23.0 exposes it as DelExArgs, marked experimental:
Lua script (any Redis)
n, err := releaseScript.Run(ctx, l.rdb, []string{l.Key}, l.Token).Int() Works on any Redis that allows scripting, so it's the portable default.
DELEX (Redis 8.4 and later)
// releaseDelEx is the Redis 8.4+ alternative to the Lua release script. // go-redis marks DelExArgs as experimental, so its signature may change. func releaseDelEx(ctx context.Context, rdb *redis.Client, l *lock.Lock) error { n, err := rdb.DelExArgs(ctx, l.Key, redis.DelExArgs{Mode: "IFEQ", MatchValue: l.Token}).Result() if err != nil { return err } if n == 0 { return lock.ErrNotHeld } return nil } Calling it twice on the same lock prints:
first release: <nil> second release: lock not held (expired or taken by someone else) Against Redis 7.4, the same code fails with ERR unknown command 'delex'. Redis Cloud and Redis Software support it; Valkey 9.0 and later uses its own DELIFEQ key value, so check your provider before switching.
Step 3: Handle Work That Outlives the TTL
This step breaks the lock by letting the work outlast the lease, then fixes it. A slow query or a long garbage-collection pause can stretch work past the TTL, and the lease expires regardless: the sign falls off while you're still inside. The fix is a watchdog , a background goroutine that keeps refreshing the lease while the work runs.
Add a token-checked refresh to lock/lock.go, plus a Watch that refreshes every third of the TTL and cancels a context the moment a refresh fails:
lock/lock.go (add to the file)
// Extend the TTL only if the key still holds our token. var refreshScript = redis.NewScript(` if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("PEXPIRE", KEYS[1], ARGV[2]) end return 0`) // Refresh resets the TTL only if we still own the lock. func (l *Lock) Refresh(ctx context.Context, ttl time.Duration) error { n, err := refreshScript.Run(ctx, l.rdb, []string{l.Key}, l.Token, ttl.Milliseconds()).Int() if err != nil { return err } if n == 0 { return ErrNotHeld } return nil } // Watch refreshes the lock every ttl/3 and cancels the returned context // as soon as a refresh fails. Do the protected work with that context. func (l *Lock) Watch(parent context.Context, ttl time.Duration) (context.Context, context.CancelFunc) { ctx, cancel := context.WithCancelCause(parent) go func() { t := time.NewTicker(ttl / 3) defer t.Stop() for { select { case <-ctx.Done(): return case <-t.C: // Bound each refresh: a hung call must not outlast the lease. rctx, rcancel := context.WithTimeout(ctx, ttl/3) err := Source：https://dev.to/shahidyousuf/distributed-locks-in-redis-with-go-2oglRelated posts
Building AgentMesh: Exploring a Shared Compute Infrastructure for AI Workloads
Exploratory Testing with AI: Beyond Predefined Test Cases
Goravel file uploads and downloads with Storage
Reorder PDF Pages: Free Online PDF Page Reordering Tool — How-To Guide
Firecrawl in Production: Surviving 10K Concurrent Scrapes on Cloudflare Workers
