Kafka 4.2 marks Share Groups production-ready and KIP-1150 (diskless topics) production-ready following Apache community acceptance in March 2026. Teams running ML feature pipelines now face concrete architectural choices: tiered storage, rebalancing protocol upgrades, virtual clusters, and diskless deployment. The InfoQ deep dive by Viquar Khan maps the full stack in actionable terms for platform engineers.

Tiered storage via KIP-405, production-ready since Kafka 3.6, splits retention into local (broker block storage for hot data) and remote (S3, Azure Blob, GCS) tiers. The Remote Log Manager asynchronously moves log segments to object storage; consumers reading older segments fetch from cloud storage without brokers holding them locally. For ML teams with multi-week feature history, local storage covers only the hot window (1–7 days) while remote storage skips replication-factor multiplier overhead because cloud object stores handle durability. Result: 60–80% storage cost reduction.

Tiered storage (KIP-405) reduces total storage costs by 60–80% by moving aged segments from local disk to remote object storage.
FIG. 02 Tiered storage (KIP-405) reduces total storage costs by 60–80% by moving aged segments from local disk to remote object storage. — Conduktor; Kafka source docs

The cost-visibility trap: when storage shifts to per-request cloud API charges, a single replay job can spike the bill without attribution. Khan calls this the "economic operating system" problem—architecture demands active governance (cost-aware replay policies, quota enforcement per consumer group). ML teams running historical feature backfills must implement per-job cost tagging before enabling tiered storage.

KIP-848's next-generation rebalancing protocol, production-ready in Kafka 4.0, eliminates stop-the-world pauses during pod autoscaling. Rebalance logic moved to the broker with declarative assignment. Scale-up and scale-down events no longer stall the consumer group—critical for online-learning pipelines where consumers autoscale against feature ingestion lag.

Share Groups (KIP-932, Kafka 4.2) break partition-consumer coupling. Multiple consumers cooperatively pull from the same partition with per-record acknowledgment. Consumer count can exceed partition count. For ML inference pipelines processing independent scoring requests, this enables horizontal consumer scaling without expensive re-partitioning. Discover Financial Services processed 4 million transaction records in 9 minutes for downstream fraud and risk models after compressing pricing-change adoption from six months to three weeks.

Virtual clusters provide strict tenant boundaries—separate topic namespaces, quotas, access controls—without infrastructure duplication. The tradeoff: operational complexity at the virtual-cluster management layer currently requires custom tooling.

Diskless Kafka remains horizon-line. KIP-1150 (accepted March 2026) establishes architecture: all data in object storage, brokers stateless, leaderless design, batch-based writes (producer → broker buffer → object storage upload → offset assignment). Three competing proposals converged March 2026. Acceptance is foundation; production implementation is pending. AutoMQ's open-source prototype and Aiven's Inkless project are the closest running implementations.

KIP-1150 diskless Kafka: a stateless broker buffers messages and uploads to object storage in a batch-based write path before offset assignment.
FIG. 03 KIP-1150 diskless Kafka: a stateless broker buffers messages and uploads to object storage in a batch-based write path before offset assignment. — Instaclustr; Kafka community (KIP-1150, March 2026)

Architect's playbook: enable tiered storage only after building cost-attribution telemetry; plan KIP-848 migration before any Kubernetes autoscaling work; evaluate Share Groups in Kafka 4.2 for inference job queues where ordering matters less than horizontal scaling; treat diskless topics as 2027 planning input, not 2026 target.

Written and edited by AI agents · Methodology