The Kafka defaults that page you at 3am.

Since 3.0 the producer default is acks=all — but acks=all waits for the in-sync replicas, not all replicas. When a follower lags or restarts it’s ejected from the ISR rather than blocking writes, so “all” silently shrinks toward just the leader. With min.insync.replicas left at 1, Kafka happily acknowledges a write that lives on exactly one disk; if that leader then dies, the acknowledged record is gone. The strong-looking default is neutralised by the weak one next to it.

# broker / topic — the trap that survived the 3.0 change
min.insync.replicas = 1◂ acks=all now means “one disk”
# producer (already the 3.0+ default — not the problem)
acks = all

default.replication.factor is 1 and auto.create.topics.enable is true. Any topic that gets auto-created — a typo’d topic name from a producer, a Connect connector, a mis-set consumer — lands as a single unreplicated partition. There’s no second copy: a single disk or broker loss is unrecoverable, and the partition is fully unavailable while that broker is down. No one ever ran a create-topic command.

default.replication.factor = 1      # one copy only
auto.create.topics.enable  = true◂ a typo mints a topic
num.partitions             = 1

The broker default for offsets.topic.replication.factor is actually 3 — but the single-broker example compose files teams copy override it to 1, because a one-node cluster can’t do more. __consumer_offsets stores every consumer group’s committed positions; at RF=1 it lives on one broker. Lose that broker and every group’s offsets vanish, so on recovery consumers fall back to auto.offset.reset (Finding 4) and either replay everything or skip everything. Teams copy the single-broker example, add brokers, and never revisit the internal-topic replication factor.

# what the single-broker quickstart forces — and you inherit
offsets.topic.replication.factor         = 1◂ every group’s offsets, one disk
transaction.state.log.replication.factor = 1

auto.offset.reset defaults to latest. When a group has no committed offset — a brand-new group, or offsets that expired or were lost per Finding 3 — latest starts at the end of the log. Every record already in the topic is skipped, silently, with no exception, and lag reads ~0. A newly deployed consumer group looks perfectly healthy while dropping its entire backlog.

auto.offset.reset = latest◂ new group starts at the END

group.protocol defaults to classic, and the default partition.assignment.strategy is a list — [RangeAssignor, CooperativeStickyAssignor] — with Range used first. Under the classic protocol with Range active, a rebalance is stop-the-world: every consumer revokes all of its partitions and blocks until reassignment completes. A rolling deploy bounces N instances, so a single deploy can cause N stop-the-world pauses and a cascade of duplicate JoinGroup rounds — and you get that eager disruption even though CooperativeSticky is sitting right there in the list, because Range is listed first.

group.protocol                = classic◂ stop-the-world rebalance
partition.assignment.strategy = [RangeAssignor,   # used FIRST
                                 CooperativeStickyAssignor]

max.poll.interval.ms is 300000 (5 minutes) and max.poll.records is 500. A background thread heartbeats every 3 seconds, so the consumer looks alive — but if the application takes longer than 5 minutes to process one poll() batch of up to 500 records (a slow DB write, an external API, a GC pause), Kafka assumes the instance is stuck, evicts it, and reassigns its partitions. The consumer that inherits them gets the same slow batch and can also time out — a self-reinforcing eviction/rebalance loop that reads as flapping.

max.poll.interval.ms = 300000   # 5 min — one batch must finish inside this
max.poll.records     = 500◂ 500 slow records/poll blows the budget

enable.auto.commit is true and commits on a 5-second timer, based on what poll() returned — not on what your code finished processing. If the consumer crashes after poll() but before handling the records, those offsets may already be committed, so on restart they’re skipped: silent loss. Crash the other way — between processing and the next auto-commit — and they replay. Either way, your delivery semantics are decoupled from your actual processing boundary and set by a timer.

enable.auto.commit      = true◂ commits on a timer, not on success
auto.commit.interval.ms = 5000

log.retention.hours is 168 (7 days) and log.retention.bytes is -1 (unbounded). Two traps in one. Size retention is off by default, so a topic’s disk usage is bounded only by time — a traffic spike or a stuck compaction can fill the disk and take brokers down. And the 7-day time bound means a consumer that falls more than a week behind loses data even with auto.offset.reset=earliest, because the records are already deleted; the offset it wants no longer exists. “earliest = never lose data” is false once retention has passed.

log.retention.hours = 168   # 7 days — a >1wk-behind consumer loses data
log.retention.bytes = -1◂ size retention OFF — disk bounded by time only