Event Driven Microservices with Kafka and the Outbox Pattern

The appeal of event driven microservices is that services react to facts about the world rather than being coordinated by a central orchestrator. But that freedom creates a consistency problem: how do you reliably publish an event when your database commit and your Kafka produce are two separate operations?

The transactional outbox pattern solves this by writing events into an outbox table inside the same ACID transaction as your business data. A separate relay process reads the outbox and forwards events to Kafka, giving you at least once delivery even if the application crashes between the database write and the publish.

Exactly once semantics in Kafka requires producer idempotence combined with transactional producers and consumers set to read committed isolation. That eliminates duplicate processing on the Kafka side, but your consumer logic still needs to be idempotent for the general case.

Schema evolution is the long term maintenance concern teams tend to underestimate. Avro with a Schema Registry gives you backward and forward compatibility guarantees, but it requires discipline: adding fields with defaults is safe, renaming or removing fields needs a migration window.

Kafka consumer group lag is your main operational signal. A lag that keeps growing and does not recover means a processing bottleneck is forming before it shows up as a user facing problem. Track it with Prometheus and alert on sustained increases rather than absolute values.