Designing reliable event-driven systems

Event-driven architecture can loosen the dependencies between services, but it does not make complexity disappear. It moves some of that complexity into message contracts, delivery guarantees, and the operational life of events.

The most reliable designs begin with a clear answer to one question: what fact has happened that other parts of the system may care about?

Model facts, not commands

An event describes something that already happened: OrderPlaced, PaymentCaptured, or ShipmentDispatched. A command asks a particular service to do something. Keeping those meanings distinct helps consumers react independently without pretending they own the producer’s workflow.

{
  "eventId": "evt_01J9...",
  "type": "OrderPlaced",
  "occurredAt": "2026-10-12T13:00:00Z",
  "data": { "orderId": "ord_4821", "customerId": "cus_193" }
}

Include stable identifiers and a timestamp, and make the payload explicit. Consumers should not need to infer a business fact from a database row or an internal implementation detail.

Assume delivery can repeat

Most practical brokers provide at-least-once delivery. A consumer can finish its work and fail before acknowledging the message, so the broker may deliver it again. Make handlers idempotent: processing the same event twice should have the same effect as processing it once.

For example, record the event ID alongside a consumer’s durable result. A uniqueness constraint can turn a repeated delivery into a safe no-op.

Keep the database and event in step

Writing a database change and publishing an event are two separate operations. If the process stops between them, the system can end up with a changed record and no event. The transactional outbox pattern stores the business change and an outgoing event in the same database transaction; a separate publisher forwards pending events and marks them delivered.

This does not remove retries. It makes the source of truth recoverable and the failure mode visible.

Make operations part of the design

Track consumer lag, processing failures, retry counts, and the age of the oldest unprocessed message. Define where poison messages go and how a human can inspect and replay them. A queue that accepts events but silently accumulates work is not a reliable system.

Good event-driven architecture is less about adding a broker and more about making ownership, failure, and recovery explicit. Start with a small set of meaningful events, document their contracts, and let real boundaries—not fashion—determine where asynchronous communication helps.