Abstract
Apache Kafka supports transactions, and many developers who use Kafka daily would be unable to state precisely what those transactions cover. The term is familiar from relational databases, where a transaction groups several writes into a single atomic outcome. In enterprise grade applications, the same outcome might be extended from databases to message brokers. For example, one of the most common business case is having transactional operations spanning databases like PostgreSQL, Oracle, etc. and message brokers like ActiveMQ, Artemis, etc.
But Kafka isn't a message broker like the others and, as opposed to classical ones, it doesn't provide any XA adapter that would allow it to participate in 2PC transactions. However, implementing transactional operations enlisting database updates and Kafka messaging as a single work unit is possible and this article demonstrates how.
It establishes what a Kafka transaction actually contains, explains the protocol by which it reaches its outcome, and then measures how much atomicity becomes available when a Kafka transaction is chained around a database transaction in the same handler. The working implementation is the transfer-kafka module of the quarkus-distributed-tx repository, where it joins an XA and a saga implementation of the same money transfer use case, all three exposing the same REST contract and exercised by the same shared test suite.
The article is illustrated by a use case implementating a money transfer application. It manages standing orders between two accounts, by debiting the source and crediting the destination account, and records the movement as a transfer. It exposes a REST contract and satisfies an abstract test class.
1. The Question This Article Answers
The scenario is ordinary. A service consumes a message describing a money transfer, debits one account, credits another, records the transfer, and publishes an event announcing the result. Three kinds of state change occur: rows are written in PostgreSQL, a record is produced to a Kafka output topic, and the consumer position on the input topic advances so that the message is considered handled.
The question is what happens to those databases and topics when something interrupts the sequence. A developer accustomed to the X/Open Distributed Transaction Processing model will expect a transaction manager to enlist both the database and the broker as resources under one coordinator. However, as already mentioned, Apache Kafka isn't a message broker, like ActiveMQ, Artemis, etc., but a distributed log, and this changes everything. Nevertheless, Apache Kafka is designed to participate in a transaction protocol of its own design, with its own coordinator, its own log and its own commit decision, and that protocol is defined over the cluster.
Understanding the shape of the available guarantee therefore requires understanding the Kafka protocol first. Section 2 supplies that, in more detail.
2. The Anatomy of a Kafka Transaction
Kafka transactions were specified in KIP-98, "Exactly Once Delivery and Transactional Messaging", and shipped in Kafka 0.11 in 2017. The specification combines two mechanisms that are frequently confused with each other, so it is worth separating them before describing either.
2.1 The idempotent producer
The first mechanism addresses duplication caused by producer retries. A producer that sends a record and then loses network connectivity before the acknowledgement arrives has no way to distinguish a lost record from a lost acknowledgement, so it retries, and the broker may append the record twice.
The idempotent producer solves this by giving each producer a numeric identifier, obtained from the broker, and by numbering every record it sends within a partition with a monotonically increasing sequence number. The broker remembers, for each producer identifier and partition, the sequence number it last accepted. A record arriving with a sequence number already seen by the broker is acknowledged and discarded. A record arriving with a gap in the sequence numbers is rejected as well. The effect is that a retry within a producer session appends exactly one record.
This mechanism operates per partition and per producer session. It is enabled by enable.idempotence proprty, and it is a prerequisite for the second mechanism.
2.2 The transactional producer and the transaction coordinator
The second mechanism groups several writes, possibly spanning several topics and partitions, into one atomic unit. A producer becomes transactional by being configured with a transactional.id, that survives restarts of the process.
That identifier is the key to a broker side component called the transaction coordinator. The transactional.id is hashed to a partition of an internal topic named __transaction_state, which KIP-98 calls the transaction log, and the leader of that partition acts as the coordinator for that producer. The coordinator holds the authoritative state of every transaction that producer opens, and it writes each state change durably into the transaction log before acting on it.
Two consequences follow immediately. The first is that the coordinator is a Kafka broker rather than a component of the application, so the decision to commit survives the possible death of the application process. The second is that a stable transactional.id allows a restarted producer to be recognized as the same logical producer, which is what makes recovery possible.
Recognition is enforced by a producer Linux epoch that the transaction coordinator stores in the transaction log beside the producer identifier for a given transactional.id.
Each transaction initiation, materialized by a call to initTransactions(), increments the stored epoch and returns the new value to the producer that caused the increment. Every request that producer issues afterward carries the producer identifier together with that epoch, whether the request produces records, adds partitions to the transaction, commits offsets or ends the transaction. A broker receiving a request whose epoch is lower than the value it holds refuses that request with an InvalidProducerEpoch error.
The purpose is to resolve the case in which two processes believe themselves to be the same producer. A service may be paused by a long garbage collection, or separated from the cluster by a network partition, while the platform running it concludes that it has failed and starts a replacement configured with the same transactional.id. Both processes are then live, and the older one remains unaware that it has been replaced. KIP-98 describes the remedy plainly, stating that the epoch is bumped "so that any previous zombie instance of the producer is fenced off". Because the replacement initialized later, it holds the higher epoch, and the broker refuses everything the older process sends. Authority is therefore determined by the broker, which observes both processes.
Developers familiar with optimistic locking will recognize the shape of the mechanism. The epoch plays the part that a version column plays in a JPA entity, where an update arriving with a stale version is refused rather than applied. The same initialization also resolves whatever transaction the previous incarnation left open, aborting it when it had yet to be decided.
2.3 The lifecycle, step by step
The protocol that reaches the outcome is itself a two phase commit, executed inside Kafka between the coordinator and the partition leaders. The sequence below follows the data flow specified in KIP-98.
Reading that sequence as a two phase commit makes its structure clear. The EndTxn request is the application asking for an outcome. The write of PREPARE_COMMIT to the transaction log is the point at which the decision becomes durable and irrevocable, exactly as a transaction manager writing its decision to an object store makes an XA outcome irrevocable. The markers written afterward are the second phase, instructing each participant to make the decision visible. The final write of COMMITTED records that the second phase completed.
An abort follows the same shape with PREPARE_ABORT and abort markers in place of their commit counterparts.
2.4 Control markers and the read_committed isolation level
The records a transaction produces are appended to their partitions as the transaction proceeds, before any outcome is known. Visibility, rather than placement, is what the outcome governs.
The mechanism is a control message, which KIP-98 defines as a special message "written to user topics, processed by clients, but never exposed to users". At commit or abort time the coordinator instructs each participating partition leader to append such a marker, carrying the producer identifier and a type of commit or abort. Markers may be written only through the internal WriteTxnMarkers request, since the specification states that "the writing of control messages is not permitted through the Produce API".
A consumer configured with isolation.level=read_committed uses those markers to decide what to deliver to the application. It reads only as far as the Last Stable Offset, which is the position beyond which a transaction remains undecided, and it discards records belonging to transactions whose marker says abort. The fetch response carries a list of aborted transactions so the consumer can discard that data directly.
The practical consequence is that a record produced inside a transaction that later aborts is physically present in the log and is withheld from every read_committed consumer. The default isolation level is read_uncommitted, under which such records are delivered, so the guarantee requires the setting to be made deliberately on the consuming side.
2.5 Consumer offsets as a transactional resource
A Kafka transaction can include the commit of consumer offsets. The internal topic __consumer_offsets is an ordinary Kafka topic, and a transactional producer may write to it as a participant in its transaction, through sendOffsetsToTransaction(...). KIP-98 states the effect directly: "The consumed offsets are not visible externally until the transaction is committed." When the transaction commits, a marker is written to the offsets partition as well, the group coordinator materializes the offsets, and the consumer position advances. When the transaction aborts, the offsets are discarded and the consumer position remains where it was.
A consumer whose position has not advanced receives the same records again on its next fetch. Aborting a transaction therefore returns the input to the state it occupied before the handler ran. This is the property which makes a Kafka transaction to support operations that genuinely resemble rollbacks.
2.6 The scope of the guarantee
A Kafka transaction covers exactly three categories of state, all of them inside the Kafka cluster:
- records appended to data partitions
- consumer offsets written to
__consumer_offsets - the transaction state itself
Within that boundary the guarantee is strong, durable and coordinated by a broker that survives the application process.
3. The Boundary of the Kafka Protocol
The X/Open model achieves atomicity across heterogeneous resources by defining a common interface, XA, that every participating resource manager implements, and by placing a transaction manager above them to drive the protocol. A JDBC datasource configured with jdbc.transactions=xa supplies connections whose XAResource the transaction manager enlists, and the manager then calls prepare and commit on that resource alongside every other branch.
Kafka reaches its outcome through the protocol described in section 2, in which the coordinator is a broker, the participants are partition leaders, and the decision is recorded in a Kafka topic. The client API that exposes this protocol is the producer API, with beginTransaction(...), commitTransaction(...) and abortTransaction(...). An external transaction manager has no interface through which to make Kafka a branch of its own transaction, and the Kafka coordinator has no interface through which to make a database a participant in its own.
This arrangement is a design decision rather than an omission. KIP-98 aimed at a protocol that performs well at Kafka's throughput and remains available under broker failure, and a generic cross resource protocol would have imposed costs that the stream processing use cases driving the design did not require.
The practical conclusion for an application performing transactional database and Kafka operations is therefore specific. Two coordinators exist. The first is the JTA transaction manager, which is Narayana in the case of Quarkus, running inside the application process and authoritative over the XA resources enlisted in its transaction. In our project these XA resources are the two PostgreSQL databases. The second is the Kafka transaction coordinator, which is the broker leading the __transaction_state partition selected for the configured transactional.id, authoritative over the produced records, the consumer offsets and the transaction state itself, which are the categories section 2.6 enumerates. Each decides the outcome of its own resources alone, and the application decides how to arrange work between them. That arrangement is the subject of the remainder of this article.
4. Choosing the Direction: Consume, Write, Produce
Two arrangements are available, and the choice between them determines how much of the guarantee survives.
In the outbound arrangement the application writes to the database, publishes an event, and the Kafka transaction covers only the produced record. The two resources reach their outcomes through two separate commits, taken one after the other, and the interval between them is a window in which exactly one of the two commits has taken effect. An interruption inside that window leaves the resource that committed first in a status that isn't propagated to the other. Where the database commits first, the window ends with a committed row and its associated event unpublished, so every downstream consumer stays unaware of a money transfer that the database considers complete. Where the Kafka transaction commits first, the interruption finds the database transaction still open, and a transaction that remains open when its client connection ends is discarded by the server. The window therefore ends with a published event that consumers may already have acted upon, concerning a money transfer that the database retains no record of. The second case is the more troublesome of the two, because a published event is a fact that other systems have observed and erroneously acted upon, whereas an unpublished event is a fact that a later reconciliation can still supply.
In the inbound arrangement the application consumes a message, writes to the database, and publishes an outcome event, committing the consumed offset inside the same Kafka transaction. Here the Kafka transaction covers a resource that can be returned to an earlier value, namely the consumer position. An abort returns the consumer to its previous position, the broker redelivers the message, and the handler runs again.
The inbound arrangement is therefore the strongest of the two, for a reason that can be stated precisely. Its worst case is the repetition of work, which an idempotent handler absorbs, whereas the outbound worst case is the publication of an event that cannot be retracted. The implementation that follows uses the inbound arrangement.
5. The Implementation
5.1 The project the module joins
The module described here belongs to a repository that implements one use case three times. The use case is a money transfer between two accounts, which debits the source account, credits the destination account and records the movement as a transfer. The three implementations expose the same REST contract, satisfy the same abstract test class, and differ in the manner by which they make the operation atomic.
The data of that use case is divided between two PostgreSQL databases. The database named accounts holds one row per account, carrying the balance that a debit and a credit modify. The database named xfers holds one row per transfer, carrying the amount, the currency, the two account identifiers and a status. The division is deliberate, because it makes a single transfer a distributed transaction before any messaging is considered. One business operation writes to two independent database servers, each with a transaction manager of its own, and the operation is correct only when both writes take effect together or both are discarded together.
The first implementation, which the first article of this series describes, resolves that with the X/Open model. Both datasources are declared with jdbc.transactions=xa, which causes each connection to expose an XAResource, and Narayana enlists the two as branches of one global transaction and drives prepare and commit across them. The second implementation replaces the global transaction with a saga, in which each step owns a compensating action and a coordinator invokes those compensations when a later step fails. The third implementation is the subject of this article. It retains the XA arrangement for the two databases and places a Kafka transaction around it.
Two consequences of that inheritance govern the sections that follow. The first is that the database work of the handler is one transaction spanning two databases, so a rollback returns both of them to their previous state together, and a commit makes both sets of changes visible at the same instant. The second is that this two database transaction is itself nested inside a Kafka transaction, so the complete arrangement is a two phase commit over two relational databases chained within a scope that also covers a topic and the consumer offsets.
5.2 Module layout
The transfer-kafka module reuses the domain, the persistence layer, the transfer service and the REST contract of the XA implementation through a shared transfer-core module, so that the only difference between the two is the transaction strategy. It holds two XA datasources, accounts and xfers, exactly as the XA module does, which means Narayana already provides one atomic unit across both databases. The Kafka transaction is chained around that unit.
What the module adds is the Kafka material: a listener, a payload type, an idempotency ledger and an observer that records what a read_committed consumer actually receives. The idempotency ledger is the PROCESSED_REQUESTS table, which holds one row for each request identifier the handler has applied, and whose purpose is to make a second delivery of an already applied request harmless. It is a record of requests rather than a record of money transfers, which distinguishes it from the XFERS table holding the transfers themselves. Both tables reside in the datasource named xfers, while the balances reside in the datasource named accounts. Section 9 explains why the ledger is required.
5.3 The listener
The handler code is reproduced in the listing below.
@ApplicationScoped public class TransferRequestListener { @Inject TransferService transferService; @Inject TransferRepository transferRepository; @Inject ProcessedRequestRepository processedRequestRepository; @Inject FailureInjection failureInjection; @Channel("transfer-events") KafkaTransactions<TransferEvent> emitter; private final Map<String, AtomicInteger> deliveries = new ConcurrentHashMap<>(); public Map<String, Integer> deliveries() { Map<String, Integer> snapshot = new HashMap<>(); deliveries.forEach((requestId, count) -> snapshot.put(requestId, count.get())); return snapshot; } public int deliveriesOf(String requestId) { AtomicInteger count = deliveries.get(requestId); return count == null ? 0 : count.get(); } @Incoming("transfer-requests") @Blocking public Uni<Void> onTransferRequest(Message<TransferRequest> message) { String requestId = requestIdOf(message); deliveries.computeIfAbsent(requestId, key -> new AtomicInteger()) .incrementAndGet(); return emitter.withTransactionAndAck(message, tx -> { try { apply(requestId, message.getPayload()).ifPresent(tx::send); return Uni.createFrom().voidItem(); } catch (RuntimeException runtimeException) { return Uni.createFrom().failure(runtimeException); } }); } private Optional<TransferEvent> apply(String requestId, TransferRequest request) { if (QuarkusTransaction.requiringNew() .call(() -> processedRequestRepository.alreadyProcessed(requestId))) return Optional.empty(); try { return Optional.of(complete(requestId, request)); } catch (ApplicationException rejection) { return Optional.of(reject(requestId, request, rejection.getMessage())); } } private TransferEvent complete(String requestId, TransferRequest request) { return QuarkusTransaction.requiringNew().call(() -> { TransferDto transfer = transferService.processTransfer(request); processedRequestRepository.record(new ProcessedRequest( requestId, transfer.getTransferId(), LocalDateTime.now())); if (failureInjection.consume(requestId)) throw new IllegalStateException("### injected failure between " + "the database write and the Kafka commit"); return TransferEvent.of(requestId, transfer); }); } private TransferEvent reject(String requestId, TransferRequest request, String reason) { return QuarkusTransaction.requiringNew().call(() -> { Transfer rejected = new Transfer(Common.getRandomId(), request.getSourceAccountId(), request.getDestinationAccountId(), request.getAmount(), request.getCurrency(), TransferStatus.REJECTED, LocalDateTime.now()); transferRepository.createTransfer(rejected); processedRequestRepository.record(new ProcessedRequest( requestId, rejected.getTransferId(), LocalDateTime.now())); return TransferEvent.of(requestId, rejected.toDto(), reason); }); } private String requestIdOf(Message<TransferRequest> message) { return message.getMetadata(IncomingKafkaRecordMetadata.class) .map(metadata -> String.valueOf(metadata.getKey())) .orElseGet(Common::getRandomId); } } The five injected members cover database access, failure injection and the Kafka side of the handler. TransferService and TransferRepository are inherited from the shared transfer-core module, and ProcessedRequestRepository is declared in this module to reach the PROCESSED_REQUESTS table that section 9 describes, these three members being the ones through which the handler reaches the two databases. FailureInjection is a small application scoped bean that defines a counter through which a test requests a deterministic interruption, and section 7 explains the position at which that interruption takes effect. The emitter field is the Kafka side of the handler, declared as KafkaTransactions<TransferEvent> and bound by @Channel to the outgoing channel named transfer-events. That declaration is what causes Quarkus to configure a transactional producer for the channel, as section 6 shows.
The deliveries map counts the arrivals of each request identifier, and the two accessor methods publish that count so that a test may assert upon it. The count is the observable on which the demonstration of rollback rests, because an aborted transaction is established by a second delivery of the same message and the count reflects that. A ConcurrentHashMap holds the counters because the Kafka worker threads access the map concurrently.
The method onTransferRequest is the entry point. The annotation @Incoming binds it to the incoming channel named transfer-requests, and @Blocking moves its execution from the I/O thread to a worker one. This is because JDBC calls block the calling thread. The method obtains the request identifier, records the delivery, and hands the remaining work to withTransactionAndAck(...).
That single call performs four operations. It begins a Kafka transaction, it supplies a TransactionalEmitter to the given function, it enlists the offset of the consumed record in the transaction, and it determines the outcome from the Uni the function returns. A Uni that completes causes the transaction to commit and the message to be acknowledged. A Uni that carries a failure causes the transaction to abort and the message to be negatively acknowledged.
The body of the function is wrapped in a try block whose catch clause converts a RuntimeException into a failed Uni. The conversion matters because the function executes inside a Mutiny pipeline, in which an exception thrown synchronously and a failure carried by the returned Uni reach different handlers. Returning the failure keeps the outcome on the path that withTransactionAndAck(...) observes. This is the path that aborts the transaction and leaves the offset uncommitted.
The call apply(requestId, message.getPayload()).ifPresent(tx::send) performs the work and publishes its outcome. The method apply returns an Optional, and ifPresent(...) sends an event exactly when one is present, through the TransactionalEmitter and therefore inside the open transaction. The empty case belongs to a request already recorded in PROCESSED_REQUESTS, which produces no event.
The method apply arranges the three possible outcomes of a request. It first consults the PROCESSED_REQUESTS table in a transaction of its own and returns an empty Optional when the request identifier is already present, which is the idempotent path of section 9. Otherwise, it delegates to complete(...). An ApplicationException, which is the type the shared transfer service throws when the business refuses an operation, is caught and routed to reject(...), whereas every other runtime exception leaves the method and reaches the catch(...) clause of the caller. That one distinction is the policy stated in section 8.
The method complete(...) performs the successful path inside one JTA transaction, begun by QuarkusTransaction.requiringNew(), which Narayana commits when the lambda function returns a value and rolls back when it throws. Three operations share that transaction, and they reach both databases. The call transferService.processTransfer(...) updates the two account rows in accounts and writes the transfer row in xfers. The row written to PROCESSED_REQUESTS records the request identifier together with the resulting transfer identifier, in xfers beside the transfer. Narayana therefore enlists two XA branches for this single transaction and drives prepare and commit across both, which is the two phase commit occurring inside a Kafka transaction. The call failureInjection.consume(requestId) occupies the final position, which places any injected exception before the commit and therefore within the scope that the rollback discards.
The method reject(...) writes a refused outcome, likewise in a fresh transaction. It creates a transfer row carrying the status REJECTED directly through the repository, and it records the request identifier in PROCESSED_REQUESTS so that a redelivery of a refused request produces a single rejection event. The reason supplied by the exception is copied into the event, which allows a consumer to learn the ground on which a transfer was refused as well as the fact of the refusal.
The method requestIdOf(...) reads the identifier from the Kafka record key through IncomingKafkaRecordMetadata, and generates one when a record arrives without a key. The key is therefore the name under which a logical request is known across its deliveries, a property to which section 9 returns.
5.4 The configuration
The incoming channel carries two settings whose purpose is frequently misread.
mp.messaging.incoming.transfer-requests.commit-strategy=ignore mp.messaging.incoming.transfer-requests.failure-strategy=ignore The first instructs SmallRye Reactive Messaging to leave offset commits entirely to the transaction. Any other commit strategy would advance the consumer position independently of the transaction outcome.
The second instructs the channel to let the transaction decide what happens after a failure. A message whose processing succeeds is acknowledged, and a message whose processing fails is negatively acknowledged. This operation is named "nacking". Under the ignore failure strategy a negatively acknowledged message causes the consumer to reset to the last committed offsets and to resume from there, which is the redelivery described in section 7. The log line the application emits at that moment reads as follows.
SRMSG18204: A message sent to channel `transfer-requests` has been nacked, ignored failure is: ### injected failure between the database write and the Kafka commit The producer side requires no explicit client configuration, because Quarkus recognises a KafkaTransactions channel and supplies the necessary settings itself.
Transactional producer detected for channel 'transfer-events', setting following default config values: mp.messaging.outgoing.transfer-events.transactional.id= ${quarkus.application.name}-${channelName} mp.messaging.outgoing.transfer-events.enable.idempotence=true mp.messaging.outgoing.transfer-events.acks=all Those three settings are precisely defining a stable transactional identifier, the idempotent producer underneath it, and the full acknowledgement so that a committed record is durable on every in sync replica.
One further setting requires explanation. Quarkus Dev Services starts the broker in a container, and it offers a choice among several providers. Apache Kafka is available in two forms, a conventional build running on the Java virtual machine and a build compiled ahead of time with GraalVM, which starts faster and is the default in Quarkus 3.37. Redpanda is an independent implementation of the Kafka protocol, written in C++ and distributed as a single binary, which presents the same client API over a different internal architecture. Strimzi is the project that operates Apache Kafka on Kubernetes, and the container images it publishes for testing hold a genuine Apache Kafka broker running on the Java virtual machine.
The choice among the above mentioned providers matters for one reason. The Quarkus documentation states that Redpanda transactions do not support exactly once processing, and it records the absence of KIP-447 and KIP-360. KIP-447, titled "Producer scalability for exactly once semantics", is the proposal through which a transactional producer commits consumer offsets while carrying the metadata of its consumer group, which allows the broker to fence a member that has left the group. That commit is the mechanism section 2.5 described and the one on which this implementation relies. KIP-360, titled "Improve reliability of idempotent/transactional producer", governs the recovery of a producer whose identifier the broker has discarded.
The module therefore names its provider explicitly, rather than inheriting a default value that has changed across Quarkus versions.
quarkus.kafka.devservices.provider=upstream-kafka-native The test run confirms the choice, reporting the image apache/kafka-native:4.2.0.
5.5 The nesting order
The Kafka transaction is the outer one, and the inner one is the JTA transaction that spans the two databases. That ordering is deliberate, and reversing it would change the guarantee.
Placing Kafka outside means the consumed offset belongs to the transaction that is decided last, so the input remains replayable until everything else has succeeded. The inner transaction commits first, which through the X/Open protocol makes the changes in accounts and in xfers visible at one instant, and it does so within the lifetime of the Kafka transaction. The Kafka transaction then commits the produced event together with the offset.
The complete arrangement therefore nests three levels. The two relational databases form the branches that Narayana enlists and that constitute one JTA transaction. That unit of work is entirely contained in one Kafka transaction covering the produced event and the consumed offset. Atomicity within each level is guaranteed by the protocol.
6. The Committing Path
The ordinary case proceeds as follows.
Source:https://dev.to/nicolasduminil/rdbms-and-kafka-distributed-transactions-with-quarkus-7l0
