Building Resilient Systems with Kafka and Event-Driven Architecture

  • kafka
  • java
  • spring-boot
  • event-driven
  • architecture

Kafka is one of those tools that feels like magic until it doesn't. The promise — decoupled, resilient, scalable messaging — is real. But so are the failure modes.

Here's what I've learned from running Kafka in production across several services.

Why Events

The core value proposition of event-driven architecture is decoupling in two dimensions.

Spatial decoupling: The order service publishes an OrderPlaced event. It doesn't know or care whether the notification service, the inventory service, and the analytics pipeline are listening. Adding a new consumer requires no change to the producer.

Temporal decoupling: Consumers don't need to be running when the event is published. Kafka retains events on disk (configurable retention, typically 7 days). If the notification service goes down for an hour, it picks up from where it left off when it restarts. Compare this to synchronous REST calls where a downstream service being down means your API call fails.

The audit log is a free byproduct. Every state change that flows through Kafka is a durable record. Replay it to reconstruct state, feed it to data pipelines, or debug production issues after the fact.

The At-Least-Once Problem

Kafka's delivery guarantee is at-least-once. A message may be delivered more than once under failure conditions — if a consumer processes a message but crashes before committing its offset, the message will be redelivered on restart.

Idempotent consumers are not optional. Your message handler must be safe to call multiple times with the same event.

The standard approach is a deduplication table. Before processing an event, check whether you've seen it before:

@Transactional
public void handleOrderPlaced(OrderPlacedEvent event) {
    if (processedEventRepository.existsByEventId(event.getEventId())) {
        log.debug("Skipping duplicate event: {}", event.getEventId());
        return;
    }
    // Process the event
    inventoryService.reserve(event.getProductId(), event.getQuantity());
    // Mark as processed in the same transaction
    processedEventRepository.save(new ProcessedEvent(event.getEventId()));
}

The deduplication check and the processing must happen in the same database transaction. If you save the processed marker separately, there's a window where the process crashes between the two operations and you get double-processing.

For high-throughput cases where a DB round trip per message is too expensive, a Redis SET with an expiry matching your Kafka retention period works well.

Dead Letter Topics and Retry Handling

Some messages can't be processed due to transient errors (downstream service unavailable) and some can't be processed at all (malformed payload, business logic rejection). You need to handle both without blocking the partition.

Spring Kafka's @RetryableTopic makes this straightforward:

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 1000, multiplier = 2.0, maxDelay = 10000),
    dltTopicSuffix = ".DLT",
    autoCreateTopics = "false"
)
@KafkaListener(topics = "orders.placed", groupId = "inventory-service")
public void handleOrderPlaced(OrderPlacedEvent event) {
    inventoryService.reserve(event.getProductId(), event.getQuantity());
}

@DltHandler
public void handleDlt(OrderPlacedEvent event, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
    log.error("Event landed in DLT after all retries exhausted. topic={}, eventId={}",
        topic, event.getEventId());
    alertingService.notifyDltMessage(topic, event);
}

This retries three times with exponential backoff. After three failures, the event goes to orders.placed.DLT. The DLT handler gives you a hook to alert, store for manual inspection, or attempt a different recovery path.

Monitor your DLTs. A DLT that grows silently means real events are being dropped. Set up a Prometheus alert on consumer group lag for your DLT topics.

Schema Evolution

JSON is the path of least resistance — no tooling required, easy to read, works everywhere. It's also the path that will hurt you when you need to evolve your event schema without breaking existing consumers.

Avro with the Confluent Schema Registry enforces compatibility contracts. When you try to register a new schema version, the Registry checks it against compatibility rules — BACKWARD (new schema can read old data), FORWARD (old schema can read new data), or FULL (both).

For services that evolve frequently and have multiple consumers, this enforcement is worth the operational cost of running a Schema Registry. For smaller setups, JSON with explicit versioning ("version": 2 in the payload) and additive-only changes is pragmatic.

The rule I apply: if you're adding a field, add it as optional with a default. If you're removing a field, treat it as a multi-phase migration — first make consumers ignore it, then stop publishing it, then remove the consumer code. Never change a field type in place.

Ordering Guarantees

Kafka guarantees ordering within a partition. Events with the same partition key will always be delivered to the same partition in the order they were published.

The common mistake: publishing all events to partition 0, or using no partition key. If order matters for a given entity (e.g., all events for order #12345 must be processed in sequence), use the entity ID as the partition key:

ProducerRecord<String, OrderEvent> record = new ProducerRecord<>(
    "orders.events",
    order.getId().toString(), // partition key
    event
);

Events for different orders land on different partitions and are processed in parallel. Events for the same order land on the same partition and are processed sequentially by a single consumer thread.

Be careful with partition count changes. Changing the partition count after a topic is created changes the mapping from key to partition. Events for the same entity may end up on different partitions depending on whether they were published before or after the change. For ordered topics, plan your partition count at creation time.

Operational Concerns

Consumer lag is your primary Kafka health metric. If a consumer group is consistently falling behind, you're producing faster than you're consuming. Add consumers (up to the partition count) or optimise processing time.

Consumer group restarts trigger partition rebalancing. All consumers in the group pause processing, the broker reassigns partitions, and consumers resume. For large consumer groups with slow max.poll.interval.ms settings, this can take tens of seconds. Tune your max.poll.interval.ms to be longer than your worst-case processing time per batch.

Heap pressure from large batches. Spring Kafka's default max.poll.records is 500. If each record is a large JSON payload, you're loading 500 × payload size into heap on each poll. Start with max.poll.records=100 and profile under load before increasing it.

Kafka solves coupling, not complexity. It trades the complexity of synchronous service dependencies for the complexity of distributed message delivery, schema evolution, and consumer lag management. That trade is often worth it — but go in with eyes open.

The patterns above — idempotent consumers, dead letter topics, schema versioning, correct partition key design — are what separate production-ready event-driven systems from ones that look fine until they fail.

← back to blog