Idempotent consumers: the Kafka lesson I keep relearning

Most of the event-driven services I built at Mercedes-Benz R&D India ran on Kafka, in Go, Kotlin, Rust and Python. Across all of them, one lesson came up more than any other: your consumer will see the same event more than once, so it had better not matter.

This post explains why duplicates happen and the patterns I use so that every event is processed exactly once in effect, even when it’s delivered twice.

Why duplicates happen

A Kafka consumer reads a message, processes it, and commits its offset so it won’t read that message again. The trouble is the gap between processing and committing. If the consumer crashes, is restarted by a deploy, or loses its partition in a rebalance after processing a message but before committing the offset, the next consumer to own that partition starts from the last committed offset and processes the message again.

Committing before processing just swaps one problem for another: crash at the wrong moment and the message is never processed at all. Producers add duplicates of their own when they retry a send whose acknowledgement got lost.

So the realistic guarantee for the system as a whole is at-least-once delivery. Kafka does offer exactly-once semantics through idempotent producers and transactions, but that covers reading from Kafka and writing back to Kafka. The moment your consumer writes to a database or calls another service, you’re outside it.

Make processing idempotent

An operation is idempotent if doing it twice has the same effect as doing it once. The goal is to make each consumer’s effect idempotent, so redelivery is harmless.

Give every event an ID. The producer assigns a unique ID when the event is created, not when it’s sent, so retries carry the same ID. Put it in a header or in the payload.

Record processed IDs in the same transaction as the effect. If the consumer writes to a relational database, keep a table of processed event IDs and insert into it in the same transaction as the real change. A duplicate hits the primary key and the whole thing is skipped. Because both writes commit together, there’s no window where the effect happened but the ID wasn’t recorded.

func (h *Handler) Handle(ctx context.Context, msg kafka.Message) error {
	tx, err := h.db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer tx.Rollback()

	// the event ID is the idempotency key; a redelivered event inserts nothing
	res, err := tx.ExecContext(ctx,
		`INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING`,
		eventID(msg))
	if err != nil {
		return err
	}
	if n, err := res.RowsAffected(); err != nil || n == 0 {
		return err // nil when it's a duplicate: already handled
	}

	if err := h.apply(ctx, tx, msg); err != nil {
		return err
	}
	return tx.Commit()
}

The offset is committed only after Handle returns without an error. If the process dies anywhere in between, the transaction rolls back and the message is simply processed again later, safely.

Or make the write itself idempotent. Sometimes you don’t need a separate table. Setting a value is naturally idempotent where incrementing one isn’t: “set the order status to shipped” can run twice, “add 10 to the balance” can’t. Upserts keyed on a natural ID, and updates guarded by a version number so an older event can’t overwrite a newer state, cover a lot of cases.

Producing events safely: the outbox

The mirror-image problem appears when a service changes its database and publishes an event about it. Write to the database and then publish, and a crash in between loses the event. Publish first, and a failed database write leaves an event announcing something that never happened.

The outbox pattern solves it. In the same transaction as the business change, insert the event into an outbox table. A separate relay reads the outbox and publishes to Kafka, marking rows as sent. The relay might publish an event twice if it crashes after sending but before marking, which is fine, because the consumers are idempotent.

Ordering, retries and poison messages

• Ordering. Kafka orders messages only within a partition. If events about the same entity must be processed in order, use the entity’s ID as the message key so they land on the same partition.
• Retries. Transient failures, like a database timeout, deserve a few retries with backoff, inside the consumer, before giving up.
• Poison messages. A message that can never be processed, because it’s malformed or breaks an assumption, will block its partition forever if you keep retrying it. After a bounded number of attempts, move it to a dead-letter topic with the error attached, alert on it, and keep going.

Watch consumer lag

Consumer lag, the distance between the newest message in a partition and the consumer’s committed offset, is the single most useful health metric for an event-driven service. Growing lag means the consumer can’t keep up or is stuck on something. A partition whose lag climbs while the others stay flat usually points to a poison message or a hot key.

The habit

None of these patterns is complicated. The lesson I keep relearning is to assume duplicates from the first line of a consumer, not to add idempotency after the first incident. Ask “what happens if this runs twice?” in every review, and event-driven systems become a lot less surprising.