TIL: Kafka vs Redpanda – different defaults for data durability

If you switch from Kafka to Redpanda expecting identical behavior, the default durability guarantees might surprise you.

Kafka: page cache, no fsync

By default, Kafka does not call fsync after writing messages. Instead, it writes to the Linux kernel page cache – a region of RAM that the OS periodically flushes to disk via its own writeback mechanism. Both writes and reads are often served directly from memory, making this path fast.

This means acknowledged messages survive a Kafka process crash – the data is still in the page cache and will eventually be flushed. But a kernel panic or power loss wipes the page cache, and any data not yet written to disk is gone.

Kafka does expose log.flush.interval.messages and log.flush.interval.ms settings, but they are effectively disabled by default. The Kafka documentation itself recommends relying on replication rather than fsync for durability.

Redpanda published a detailed critique of this approach, demonstrating how under certain failure scenarios a single node’s data loss after a crash can propagate through the replication protocol, causing acknowledged messages to disappear across the entire cluster.

Redpanda: fsync before acknowledgment

Redpanda takes the opposite approach: it always writes through O_DIRECT, bypassing the page cache. When a producer sets acks=all, Redpanda ensures data is persisted to disk before sending an acknowledgment. Even a kernel panic or sudden power loss won’t cause data loss.

Under the hood, Redpanda achieves this without the typical fsync overhead. Built on the Seastar framework, it uses O_DIRECT for all disk I/O, writing directly to the storage device without going through the kernel page cache. A final fdatasync call ensures data reaches persistent storage.

The tradeoff is that write throughput is bounded by disk speed. This is why Redpanda requires NVMe drives for production deployments - the drives need to deliver at least 16,000 IOPS to keep up.

In short