There is a class of bugs that never shows up in your unit tests and rarely appears in staging.
It appears in production, at 2am, when your app saves a record to the database, then tries to
call an external service — and one of them fails. One side committed. The other didn't. Nobody
knows until a user complains.

This is the dual-write problem, and the Transactional Outbox Pattern is the standard answer
to it. The idea is simple: instead of calling the external system directly, write a task into the
same database transaction as your domain change. A background worker picks it up and executes it.
The write is atomic. The task cannot be lost.

What's less documented is everything that comes after the basic idea — what happens when tasks
fail, when two workers race for the same row, when your app crashes mid-execution, when you have
ten different task types living in one table, and when your stakeholders ask why the queue depth
spiked at 3am last Tuesday.

That's what this article covers.

I'll work through a single concrete scenario — a report generation service — and evolve it
step by step from the simplest possible outbox to something I'd be comfortable running in
production. Every code snippet is from a real working repo linked at the end.

Stack: Java, Spring Boot, PostgreSQL.

TL;DR

  • The Transactional Outbox Pattern fixes the dual-write problem by writing a task row in the same DB transaction as your domain change, then executing it from a background worker.
  • Poll the outbox table with SELECT ... FOR UPDATE SKIP LOCKED so multiple workers can run concurrently without double-processing.
  • Production-grade requires more than the basic idea: a state machine, retries with backoff, heartbeats/leases to recover crashed workers, ordering guarantees, and monitoring of queue depth.
  • You don't need Kafka — PostgreSQL alone is enough for reliable async execution.

The Scenario 

sequenceDiagram actor User as User (Backoffice) participant API as ReportGenerationService participant DB as PostgreSQL User->>API: POST /reports (from, to) API->>DB: INSERT report_generation_tasks (PENDING) DB-->>API: task saved API-->>User: 202 Accepted loop Scheduler every 1s API->>DB: SELECT ... WHERE state = PENDING FOR UPDATE SKIP LOCKED DB-->>API: task row API->>API: generateReport() API->>DB: UPDATE state = COMPLETED end

We're building a CRM. Users can request monthly analytics reports from a backoffice UI.
Generating a report means aggregating tens of millions of rows — it can take minutes.
Obviously we can't do this synchronously in an HTTP request.

The user clicks a button. The backend accepts the request, saves it, and responds immediately.
A background worker picks up the task, runs the generation, and updates the status when done.

This is the simplest case where an outbox makes sense: work that is genuinely async by nature,
where the user already expects to wait.

Part 1 — The Basic Outbox

What Every Outbox Needs

At its core, any outbox table needs three things:

  • State — where is this task in its lifecycle?
  • Payload — what exactly needs to be executed?
  • Timestamps — when was it created, when did it last change? 
CREATE TYPE task_state AS ENUM ('PENDING', 'COMPLETED'); CREATE TABLE report_generation_tasks ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), state task_state NOT NULL DEFAULT 'PENDING', payload JSONB NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT now(), updated_at TIMESTAMP NOT NULL DEFAULT now() );

The payload for our use case is a date range:

@Value @Builder @Jacksonized public class ReportGenerationPayload { Instant from; Instant to; }

The JPA entity:

@Entity @Table(name = "report_generation_tasks") @Data @Builder @NoArgsConstructor @AllArgsConstructor public class ReportGenerationEntity { @Id @GeneratedValue(strategy = GenerationType.UUID) @Column(updatable = false, nullable = false) private UUID id; @Enumerated(EnumType.STRING) @Column(nullable = false, columnDefinition = "task_state") private TaskState state; @JdbcTypeCode(SqlTypes.JSON) @Column(nullable = false, updatable = false, columnDefinition = "jsonb") private ReportGenerationPayload payload; @CreationTimestamp @Column(nullable = false, updatable = false) private Instant createdAt; @UpdateTimestamp @Column(nullable = false) private Instant updatedAt; }

Submitting a task:

@Service @RequiredArgsConstructor public class ReportGenerationService { private final ReportGenerationRepository repository; public void submit(ReportGenerationCommand command) { repository.save(ReportGenerationEntity.builder() .state(TaskState.PENDING) .payload(ReportGenerationPayload.builder() .from(command.getFrom()) .to(command.getTo()) .build()) .build()); } }

The Worker

A scheduled job polls for pending tasks and executes them:

@Component @RequiredArgsConstructor public class ReportGenerationScheduledJob { private final ReportGenerationTaskService taskService; @Scheduled(fixedDelay = 1000) public void run() { taskService.fetchAndProcess(); } } 
@Service @RequiredArgsConstructor public class ReportGenerationTaskService { private final ReportGenerationRepository repository; public void fetchAndProcess() { repository.findFirstByStateOrderByCreatedAt(TaskState.PENDING) .ifPresent(task -> { generateReport(task); task.setState(TaskState.COMPLETED); repository.save(task); }); } private void generateReport(ReportGenerationEntity task) { // actual report generation logic } } 
public interface ReportGenerationRepository extends JpaRepository<ReportGenerationEntity, UUID> { Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state); }

This works in a single-instance app with no failures. In the real world, it has two problems:
two workers can claim the same task simultaneously, and tasks can fail without any recovery path.
Let's fix both.

Part 2 — Concurrency: Two Workers, One Task

When two scheduler threads — or two app instances — run the fetch query at the same moment,
both can read the same PENDING row and both will try to execute it. The result is duplicate work
at best, corrupted state at worst.

The fix is a database lock taken at fetch time. PostgreSQL's SELECT ... FOR UPDATE SKIP LOCKED
is purpose-built for exactly this use case: it locks the row for the current transaction and skips
any rows that are already locked by another transaction, rather than waiting.

public interface ReportGenerationRepository extends JpaRepository<ReportGenerationEntity, UUID> { @Lock(LockModeType.PESSIMISTIC_WRITE) @QueryHint(name = AvailableSettings.JAKARTA_LOCK_TIMEOUT, value = "-2") // -2 = SKIP_LOCKED Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state); }

The SKIP_LOCKED hint value -2 is what Hibernate translates to the SKIP LOCKED clause.
Since there's no first-class annotation for this, it's worth extracting into a reusable annotation
so you don't scatter the magic number across your repositories:

@Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD, ElementType.ANNOTATION_TYPE}) @QueryHints(@QueryHint( name = AvailableSettings.JAKARTA_LOCK_TIMEOUT, value = Timeouts.SKIP_LOCKED_MILLI + "" )) public @interface SkipLocked {}

Much cleaner. Now the repository reads like intent, not plumbing:

@SkipLocked @Lock(LockModeType.PESSIMISTIC_WRITE) Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state);

You can keep the lock while processing by wrapping the whole fetch-execute-update cycle in a
single transaction, but you generally should not:

// Technically works, but do not use this pattern for outbox processing. @Transactional public void fetchAndProcess() { repository.findFirstByStateOrderByCreatedAt(TaskState.PENDING) .ifPresent(task -> { generateReport(task); task.setState(TaskState.COMPLETED); repository.save(task); }); } 
sequenceDiagram participant DB as PostgreSQL participant A as Worker A participant B as Worker B Note over DB: task row: PENDING (unlocked) par Worker A and Worker B fetch simultaneously A->>DB: SELECT ... FOR UPDATE SKIP LOCKED B->>DB: SELECT ... FOR UPDATE SKIP LOCKED end DB-->>A: row locked ✓ — returns task DB-->>B: row locked by A — SKIP LOCKED returns empty Note over B: no task found, exits cleanly A->>A: generateReport() A->>DB: UPDATE state = COMPLETED A->>DB: COMMIT — lock released Note over DB: task row: COMPLETED (unlocked)

Even if generateReport() is small enough that this technically works, running an outbox action
under an open transaction is an antipattern. Transaction duration should not depend on outbox
execution duration: the action can become slower over time, block on I/O, or call a downstream
service, leaving a database connection and row lock held for an unpredictable period.

Instead, claim the row in a short transaction by moving it to PROCESSING, commit and release the
lock, and only then execute the outbox action. Record COMPLETED or FAILED in another short
transaction afterward. The next section implements that state transition and its recovery path.

Part 2.5 — Reducing Latency with Application Events

The polling scheduler has an inherent latency floor. If your job fires every second, a task
submitted at t=0ms sits PENDING until the next tick — up to 1000ms later. For most async
workloads that's acceptable. For cases where you want near-immediate pickup — sending a
transactional email, triggering a webhook — it's not.

The pattern borrowed from Celery: save the outbox entity to the database, then immediately
publish a Spring ApplicationEvent carrying the new task's ID. A listener picks it up and
dispatches processing right away, without waiting for the next scheduler tick.

sequenceDiagram participant S as Service participant DB as PostgreSQL participant EL as Event Listener participant W as Worker rect rgb(240, 240, 240) Note over S,W: Polling approach — up to 1s latency S->>DB: INSERT task (PENDING) Note over W: ... waiting for next scheduler tick (0–1000ms) ... W->>DB: SELECT PENDING tasks DB-->>W: task found W->>W: process task end rect rgb(230, 247, 240) Note over S,W: Application event approach — near-zero latency S->>DB: INSERT task (PENDING) S->>EL: publishEvent(taskId) — fires AFTER_COMMIT EL->>W: trigger fetchAndProcess() immediately W->>DB: SELECT PENDING tasks DB-->>W: task found W->>W: process task end

The scheduler still runs as a safety net — it catches anything the event listener missed (app
restart, listener exception, missed event). But the happy path bypasses the polling delay
entirely.

Implementation

Publish the event inside the service method, after saving:

@Service @RequiredArgsConstructor public class ReportGenerationService { private final ReportGenerationRepository repository; private final ApplicationEventPublisher eventPublisher; @Transactional public void submit(ReportGenerationCommand command) { var entity = repository.save(ReportGenerationEntity.builder() .state(TaskState.PENDING) .payload(ReportGenerationPayload.builder() .from(command.getFrom()) .to(command.getTo()) .build()) .build()); eventPublisher.publishEvent(new ReportGenerationTaskSubmittedEvent(entity.getId())); } } 
public record ReportGenerationTaskSubmittedEvent(UUID taskId) {}

The listener reacts and triggers processing immediately:

@Component @RequiredArgsConstructor public class ReportGenerationTaskEventListener { private final ReportGenerationTaskService taskService; @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT) public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) { taskService.fetchAndProcess(); } }

The Classic Mistake

The most common error here is using @EventListener instead of
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT).

@EventListener fires the moment publishEvent() is called — which is inside the open
transaction, before the INSERT has committed. The listener calls fetchAndProcess(),
which queries the database for the task by ID. The row does not exist yet. The task is silently
skipped, the event is lost, and you're left wondering why processing never started — even though
the scheduler eventually picks it up seconds later and everything looks fine in the logs.

// ❌ fires inside the transaction — entity not visible to other connections yet @EventListener public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) { taskService.fetchAndProcess(); } // ✅ fires after COMMIT — entity is visible, fetch succeeds @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT) public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) { taskService.fetchAndProcess(); }

AFTER_COMMIT guarantees the listener runs only after the transaction that published the event
has fully committed. By that point the row is visible to all connections and the fetch will
find it.

The scheduler remains unchanged — it acts as the recovery mechanism for any tasks the event
listener missed. The two approaches complement each other: events for latency, polling for
reliability.

Part 3 — Failures, Retries, and the PROCESSING State 

stateDiagram-v2 [*] --> PENDING: submit() PENDING --> PROCESSING: worker claims task\n(TX1 commits) PROCESSING --> COMPLETED: generateReport() succeeds\n(TX2 commits) PROCESSING --> FAILED: generateReport() throws\n(TX2 commits) PROCESSING --> PENDING: app crash detected\nstuck-task recovery job resets FAILED --> PROCESSING: retry eligible\n(retryCount ≤ MAX and executeAt ≤ now) FAILED --> [*]: retryCount > MAX\n(dead — manual ops) COMPLETED --> [*]: cleanup job removes\nafter retention period

Outbox actions should not run under an open database transaction. The locked-transaction approach
holds a database lock for the entire
duration of generateReport(). For a report that takes two minutes, that's two minutes of a
locked row — a connection tied up, other DB operations potentially queuing behind it. With
SKIP LOCKED workers don't block each other on the lock, but long-running transactions still
put pressure on the connection pool and make the database unhappy under load. A very small action
may technically complete without causing visible trouble, but coupling transaction duration to
outbox execution duration is still an antipattern.

The fix is to separate claiming a task from executing it. Flip the task to PROCESSING in a
short transaction, release the lock, then do the actual work outside any transaction. This way
the DB is only touched briefly at the start and briefly at the end — the expensive part happens
in between without holding anything.

First, extend the enum:

ALTER TYPE task_state ADD VALUE 'PROCESSING'; ALTER TYPE task_state ADD VALUE 'FAILED'; ALTER TABLE report_generation_tasks ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0, ADD COLUMN error_message VARCHAR;

retry_count tracks how many times execution was attempted. We cap retries at 10 — beyond that,
something is genuinely broken and needs manual intervention. We'll wire alerting around this
later. error_message stores the last exception so you know why a task is stuck without
digging through logs.

The Java enum grows with it:

public enum TaskState { PENDING, PROCESSING, COMPLETED, FAILED }

The service now runs two separate short transactions with the actual work in between:

@RequiredArgsConstructor public class ReportGenerationTaskService { private final ReportGenerationRepository repository; private final TransactionTemplate transactionTemplate; private static final int MAX_RETRY_COUNT = 10; public void fetchAndProcess() { transactionTemplate .execute(status -> repository.findFirstEligibleTask( List.of(TaskState.PENDING, TaskState.FAILED), MAX_RETRY_COUNT ).map(this::markProcessing)) .ifPresentOrElse( task -> { try { generateReport(task); markCompleted(task); } catch (Exception e) { markFailed(task, e); log.error("Task {} failed on attempt {}", task.getId(), task.getRetryCount(), e); } }, () -> log.debug("No eligible tasks to process") ); } private ReportGenerationEntity markProcessing(ReportGenerationEntity task) { task.setState(TaskState.PROCESSING); return repository.save(task); } private void markCompleted(ReportGenerationEntity task) { task.setState(TaskState.COMPLETED); repository.save(task); } private void markFailed(ReportGenerationEntity task, Exception e) { task.setState(TaskState.FAILED); task.setRetryCount(task.getRetryCount() + 1); task.setErrorMessage(e.getMessage()); // don't skip this — it's the only trace of what went wrong without digging through logs repository.save(task); } }

The query fetches both PENDING and FAILED tasks — a failed task with retryCount < 10 is
just as eligible as a fresh one. Once retry count hits the cap, the task stops being fetched and
sits as a permanent FAILED record until someone looks at it. That's intentional — beyond 10
attempts something is genuinely broken and needs eyes on it, not another retry.

@SkipLocked @Lock(LockModeType.PESSIMISTIC_WRITE) @Query(""" SELECT t FROM ReportGenerationEntity t WHERE t.state IN :states AND t.retryCount <= :maxRetryCount ORDER BY t.createdAt ASC LIMIT 1 """) Optional<ReportGenerationEntity> findFirstEligibleTask( @Param("states") List<TaskState> states, @Param("maxRetryCount") Integer maxRetryCount );

But wait — should a just-failed task really be picked up again on the very next scheduler tick,
seconds later? If the downstream system is down, hammering it every second makes things worse.
That's what the next section covers.

Part 4 — Exponential Backoff

Without a delay between retries, a failed task re-enters the queue on the very next scheduler
tick — usually seconds later. If the downstream system is down, you're hammering it at full
polling rate. That's bad for the downstream system and bad for your DB.

The fix is an execute_at column: the task is not eligible until this timestamp is in the past.
Fresh tasks get execute_at = now() so they're picked up immediately. Failed tasks get
execute_at = now() + delay, where the delay grows with each retry.

ALTER TABLE report_generation_tasks ADD COLUMN execute_at TIMESTAMP NOT NULL DEFAULT NOW();

The delay schedule drives everything — MAX_RETRY_COUNT is derived from it, not declared
separately. This way the two can never drift out of sync:

private static final Map<Integer, Duration> RETRY_DELAYS = Map.of( 0, Duration.ofMinutes(1), 1, Duration.ofMinutes(5), 2, Duration.ofMinutes(15), 3, Duration.ofMinutes(30), 4, Duration.ofHours(1), 5, Duration.ofHours(2), 6, Duration.ofHours(4), 7, Duration.ofHours(8), 8, Duration.ofHours(12), 9, Duration.ofDays(1) ); private static final int MAX_RETRY_COUNT = RETRY_DELAYS.size();

Add a new entry to RETRY_DELAYS and the retry cap automatically updates. No magic number to
keep in sync elsewhere.

In markFailed(), capture the current retry count before incrementing — that selects the
delay for the next attempt:

private void markFailed(ReportGenerationEntity task, Exception e) { int currentRetry = task.getRetryCount(); task.setState(TaskState.FAILED); task.setRetryCount(currentRetry + 1); task.setErrorMessage(e.getMessage()); task.setExecuteAt(Instant.now().plus( RETRY_DELAYS.getOrDefault(currentRetry, RETRY_DELAYS.get(RETRY_DELAYS.size() - 1)))); repository.save(task); }

The scheduler keeps running at the same rate. It just skips tasks whose window hasn't opened.
Failed tasks sit quietly until their time comes, then re-enter naturally.

Part 5 — Stuck Tasks and Heartbeats 

sequenceDiagram participant W as Worker participant DB as PostgreSQL participant R as Recovery Job W->>DB: UPDATE state = PROCESSING (TX1 commits) W->>W: generateReport() — running... Note over W: 💥 app crashes mid-execution Note over DB: task st...