Event-Driven Architectures That Survive Peak Traffic
How we design pub/sub systems, outbox patterns, and idempotent consumers so message storms do not become outages.
Events are contracts, not logs
Teams adopt Kafka hoping for decoupling and get distributed debugging instead. The difference is contract discipline: versioned schemas, explicit ownership, and clear delivery semantics. An event storm without schemas is just a more expensive way to couple services through tribal knowledge.
We define producers, consumers, and SLOs per topic. If nobody owns poison messages, peak traffic will invent an owner at the worst time.
Transactional outbox over dual writes
Writing to the database and then publishing to a broker is a classic consistency trap. The outbox pattern records the event in the same transaction as business state, then a relay publishes safely.
This pattern survives broker outages and process crashes. It also makes replay and audit trails honest because the source of truth for "what happened" stays with the domain data.
Idempotent consumers and backpressure
At least-once delivery means duplicates will happen. Consumers must key on event IDs or natural business keys and treat retries as normal. Side effects — emails, charges, provisioned resources — need dedupe tables or provider idempotency keys.
Backpressure matters as much as throughput. Consumer lag alerts should trigger load shedding or buffer expansion before partitions fall hours behind and create cascading retries.
Peak traffic playbooks
Load tests should include producer bursts, consumer slowdowns, and poison pill scenarios. We rehearse pausing consumers, draining DLQs, and replaying from offsets with product awareness of user-visible delay.
Event-driven systems that survive peaks are boring: strong schemas, outboxes, idempotency, and operators who have practised failure — not just drawn boxes on a whiteboard.
James Okonkwo
Platform Engineer
Platform engineer building event-driven systems, observability, and edge architectures.
