demo

Hi everyone! I'd like to share PgLoom (v0.1.1, Apache-2.0), a deterministic concurrency model checker written in Rust (Edition 2024, #![forbid(unsafe_code)]) for catching transaction-level race conditions in services backed by PostgreSQL.

Motivation

Tools like tokio-rs/loom are great at exploring thread and atomic interleavings inside a single Rust process, while most financial and state bugs in production backends happen across network SQL transactions under PostgreSQL's default READ COMMITTED isolation: check-then-act double spends, lost updates, write skew, phantom inserts, and lock-order deadlocks.

Sequential integration tests never interleave concurrent transactions, and load tests only hit the microsecond race window by luck without a way to replay it.

PgLoom sits between any backend service (Rust, Go, Python, TypeScript, Java, etc.) and a real PostgreSQL 12..17 server as a v3 wire-protocol proxy, holding SQL round trips at a gate and systematically exploring how transactions interleave. No SDK or application code changes are needed: you point DATABASE_URL at PgLoom and define concurrent workers + invariants in YAML.

Sample output (pgloom demo)

[FAIL] RACE CONDITION DETECTED [demo-wallet-withdraw] Explored 3/50 schedules in 66ms (1-preemption trace) Observed DB state: id=1, balance=-100 (req_a=200, req_b=200) +------+------------------------+------------------------+ | Step | Worker req_a (pid 205) | Worker req_b (pid 204) | +------+------------------------+------------------------+ | t3 | SELECT balance -> 100 | | | t4 | UPDATE balance - 100 | | | >> | --- context switch (preempt) ----> | | t5 | [paused before COMMIT] | SELECT balance -> 100 | | t6 | | UPDATE balance - 100 | | | | [blocked by req_a] | | t7 | COMMIT -> HTTP 200 OK | | | t9 | | lock granted -> UPD 1 | | t10 | | COMMIT -> HTTP 200 OK | +------+------------------------+------------------------+ Fix Option A (Pessimistic Lock): SELECT balance FROM wallets WHERE id = $1 FOR NO KEY UPDATE; Fix Option B (Atomic Guarded Update): UPDATE wallets SET balance = balance - $1 WHERE id = $2 AND balance >= $1;

Crate architecture & technical highlights

The workspace is split into 6 crates:

  • pgloom-wire: Stateful PostgreSQL v3 protocol proxy (tokio + bytes). Parses Simple and Extended Query pipelines (Parse, Bind, Describe for statements and portals, Execute, Flush, Sync, plus COPY streaming), caches column layouts to decode binary parameters/results, and gates completed SQL turns before forwarding them upstream.
  • pgloom-engine: Implements Source-DPOR (dynamic partial-order reduction with sleep sets) over SQL read/write footprints enriched with pg_catalog foreign-key and trigger metadata to prune Mazurkiewicz-equivalent schedules, alongside randomized and PCT (Probabilistic Concurrency Testing) explorers.
  • pgloom-pgctrl: Queries pg_blocking_pids() when a backend blocks on a row lock in Postgres so the scheduler parks the waiting worker instead of hanging the proxy, hands execution back to the lock holder, and detects true lock cycles as deadlocks. Also resets mutated tables between schedules in <1ms from an UNLOGGED snapshot schema so every schedule executes real COMMITs (rather than faking transactions with SAVEPOINTs, which would alter READ COMMITTED visibility).
  • pgloom-driver / pgloom-report / pgloom-cli: Concurrent HTTP & SQL worker drivers, response/SQL invariant evaluators, static SQL Fix Advisor, and deterministic .pgloom-repro.json schedule replay.
  • Testing: Tested across PostgreSQL 12 through 17 on CI (MSRV 1.89), with proptest property suites and cargo-fuzz targets covering the wire frame/message decoders, SQL classifier, and Fix Advisor.

Try it locally

pgloom demo runs a self-contained vulnerable wallet handler, catches the double-spend in ~60ms, patches the query with FOR NO KEY UPDATE, and re-explores every schedule to verify the fix (uses a local Postgres or starts a throwaway postgres:17-alpine container):

cargo install pgloom --locked pgloom demo

Prebuilt static binaries for Linux (musl x86_64/aarch64) and macOS (Apple Silicon/Intel) are also attached on the GitHub Releases page.

Feedback on the workspace design, pgwire edge cases across drivers, or Source-DPOR over SQL is very welcome!

1 post - 1 participant

Read full topic