Manual Kafka offset commits: the difference between received and done

Auto-commit is the default in Kafka for a reason: it's fine for most workloads. If you lose a click event or a metrics sample during a crash, nobody files an incident.

Debit card transactions are not that workload.

With enable.auto.commit=true, the consumer commits offsets on a timer — every five seconds by default — regardless of what your code has actually done with the messages. The commit means "I received these." It does not mean "I processed these." If the service dies between the commit and the end of your processing logic, those transactions are gone. Kafka believes you handled them, so it will never redeliver them. There is no error, no log line, no dead letter. Just a gap that reconciliation finds three days later.

The fix is to make the commit the last thing that happens:

@KafkaListener(topics = "card.transactions", containerFactory = "manualAckFactory")
public void onTransaction(ConsumerRecord<String, Transaction> record, Acknowledgment ack) {
    try {
        transactionService.process(record.value()); // idempotent — may run twice
        ack.acknowledge();                          // commit only after success
    } catch (NonRetryableException e) {
        deadLetterProducer.send(record, e);         // park it, don't lose it
        ack.acknowledge();
    }
    // Retryable failure? Don't ack. The record is redelivered.
}

With AckMode.MANUAL_IMMEDIATE, a crash before acknowledge() means the record is redelivered on restart. The failure mode flips from silent data loss to possible duplicate processing — which is why process() must be idempotent. In our case, an upsert keyed on the transaction id.

That flip is the entire point. Duplicates are a problem you can engineer away with an idempotency key. Silence is a problem you find in an audit.

Two supporting pieces make this production-grade rather than just correct:

• Dead Letter Topics for poison messages — a record that can never succeed shouldn't block the partition, but it also can't be dropped. Park it where reconciliation can see it.
• Automated reconciliation that compares source counts against processed counts, so "zero data loss" is a measured property, not a hope.

received is what the broker knows. done is what your business logic knows. In a bank, only one of those is allowed to move the offset.