An order gets saved to Postgres. The next line of code publishes an "order created" event to a message broker, so the payments service can charge the customer, shipping can start prepping a label, and notifications can send a confirmation email. That's the whole design. It works in every test, in staging, and in production, for months.

Then one day the process gets killed between those two lines. Deploy rolled a pod mid-request. The broker connection timed out for four seconds during a network blip. A GC pause pushed the request past a load balancer timeout and the client retried against a different instance while the first one was still finishing up. The order is sitting in the database, fully committed, real, billable. Nobody downstream ever hears about it. No exception was thrown. No alert fired. The system looks healthy. A customer just placed an order that will never ship.

Why You Can't Just Wrap It in a Transaction

The instinct is to reach for a transaction: begin, insert the order, publish the event, commit. That doesn't work, and it's worth being precise about why, because the reason is architectural, not a missing try/catch.

A database transaction gives you atomicity over one resource: the database. Postgres can guarantee the order row either exists or doesn't, with nothing in between visible to other readers. A message broker (Kafka, RabbitMQ, SQS) is a separate system with its own commit protocol, its own durability guarantees, and no shared transaction coordinator with your database. There's no operation that atomically says "commit this Postgres row and this Kafka message together, or neither." You are making two independent network calls to two independent systems, and between them there is always a window where one has succeeded and the other hasn't.

Move the event publish before the database commit and you get the opposite failure: the event goes out, downstream services start acting on an order that doesn't exist yet, and then the database write fails (constraint violation, connection drop, whatever) and now payments has charged a card for an order that was never actually created. Order doesn't matter. Two independent writes to two independent systems, with no atomicity across the boundary, will eventually diverge no matter which one goes first. This is the dual-write problem, and it's not a bug in your code, it's a gap in what a database transaction can promise you.

Why this matters more than it looks like it should: this isn't a rare edge case you can accept as background noise. Every deploy is a process restart. Every autoscaling event is new pods coming up while old ones drain mid-request. Every network blip between your service and your broker is a window for this to happen. At low traffic it might occur once a month and get written off as "weird, must've been a fluke." At real scale, with thousands of writes a minute, this gap fires constantly, and it fires exactly during the conditions you're least equipped to notice it: deploys and incidents, when your attention is already somewhere else.

Two-Phase Commit: The Theoretically Correct Answer Nobody Uses

The textbook fix for "atomically commit across two systems" is Two-Phase Commit (2PC). A coordinator asks every participant (the database, the broker) to prepare, meaning "lock this resource and confirm you can commit, but don't commit yet." Once every participant says yes, the coordinator tells everyone to commit for real. If any participant says no, the coordinator tells everyone to roll back.

It's a real protocol with a real correctness proof, and it's almost never what production systems actually use for this problem, for reasons that show up the moment you operate it instead of just reading about it:

• It's blocking. Once a participant says "yes, I can commit," it has to hold that lock until the coordinator's final decision arrives. If the coordinator crashes after collecting votes but before sending the commit decision, every participant is stuck holding locks indefinitely, waiting for a coordinator that might not come back.
• Most message brokers don't implement the participant side of the protocol at all. Kafka has no native 2PC participant role. You'd be building and maintaining that coordination layer yourself, on top of a system that was never designed to expose it.
• It doesn't survive network partitions gracefully. The entire protocol assumes the coordinator can eventually reach every participant. A partition during the decision phase leaves things in exactly the indeterminate, locked state the protocol was supposed to prevent.

2PC is the right answer to a narrower question: coordinating multiple databases that all speak a compatible protocol, in a controlled environment where you own the failure modes. It's the wrong tool for "coordinate my database with my message broker," which is the shape of the dual-write problem almost everyone actually hits.

The Outbox Pattern: Make the Second Write Boring

The pattern that actually gets used starts from a reframe: stop trying to make two different systems commit atomically, and instead make the second write something the same database transaction can own.

Instead of publishing to the broker directly, you write the event as a row in an outbox table, in the exact same transaction as the order insert:

Now atomicity is trivial, because both writes are ordinary rows in the database you already have transactional guarantees for. Either both rows exist or neither does. There is no window where the order exists but its event doesn't, because "the event exists" now just means "a row in the same table transaction says so."

A separate relay process, running continuously, polls the outbox table for unpublished rows (or reads Postgres's write-ahead log directly via logical replication, which is how tools like Debezium do it without polling), publishes each one to the real message broker, and marks it published. If the relay crashes, it resumes from the last unpublished row on restart. If the broker is briefly unavailable, the events just sit in the table until it recovers. The order write itself never blocks on the broker being up at all.

The trade-off is honest, not hidden: downstream consumers now see events after a short delay (however often the relay polls, typically sub-second to a few seconds), and they can receive the same event more than once if the relay publishes successfully but crashes before marking the row as sent. That second part is why every consumer of these events has to be idempotent, not because it's good hygiene in the abstract, but because at-least-once delivery is the actual guarantee the outbox gives you, and "exactly once" isn't achievable without it.

Sagas: The Same Problem, Stretched Across Multiple Services

The outbox pattern solves atomicity for one write plus one event. A saga is what you reach for when the operation itself spans multiple services, each with its own local database, and there's no way to wrap the whole thing in a transaction even in principle: reserve inventory, charge the payment, create the shipment. Three services, three databases, one logical operation.

A saga runs this as a sequence of local transactions, each one committing on its own, with a compensating action defined for each step in case a later step fails:

1. Inventory service reserves the item. Commits locally.
2. Payment service charges the card. Commits locally.
3. Shipping service creates the shipment. If this fails...
4. ...run the compensations in reverse: refund the payment, release the inventory reservation.

Nothing here is atomic in the ACID sense. There's a real window, between steps 1 and 3, where inventory is reserved and no shipment exists yet. A saga doesn't hide that window, it makes it explicit and time-bounded, and gives you a defined path back to a consistent state if the last step doesn't complete. That's a fundamentally different consistency model than a database transaction (eventual, with defined compensations, instead of immediate and atomic), and pretending otherwise is where sagas go wrong in practice: teams build the happy path, skip writing the compensating actions because "that won't really happen," and then an incident forces someone to write the refund logic live, under pressure, for a case they never tested.

Each step publishing its "I'm done" or "I failed" signal is itself a dual-write problem at a smaller scale, which is why saga implementations lean on the outbox pattern internally for each step, rather than being a separate mechanism from it. The two patterns compose: outbox solves atomic local write-plus-event, sagas solve the multi-step orchestration built on top of that primitive.

What This Actually Demonstrates

None of this is about picking the "advanced" pattern to look sophisticated. It's about recognizing that "write to the database, then tell everyone else" is not one operation, it's two, and the honest response to that is either to make the second write ride inside the first one's transaction (Outbox) or to make the multi-step version of that gap explicit and recoverable (Sagas) — not to assume the gap won't matter until an incident proves otherwise.