Apache Kafka 3.7: Official Docker Image and Improved Client Monitoring

Apache Kafka® 3.7 is here! On behalf of the Kafka community, Danica Fine highlights key release updates, with KIPs from Kafka Core, Kafka Streams, and Kafka Connect.

Kafka Core:
- KIP-714: Gives operators visibility into clients by exposing client metrics and provides new components to allow cluster operators to receive client metrics.
- KIP-1000: Introduces a way to list the client metrics configuration resources introduced by KIP-714.
- KIP-848 (Early Access): Introduces a new group membership and rebalance protocol for the Kafka Consumer and, by extension, Kafka Streams. Provides the same guarantee as the current protocol and introduces new Consumer Group Protocol types.
- KIP-951: Minimizes the time taken to discover a new leader of a partition, reducing the effect of leadership changes on the end-to-end latency of produce or fetch client requests.
- KIP-975: Provides an official JVM-based Apache Kafka Docker image. It enables quicker testing and deployment of Kafka, and faster onboarding of new developers joining the community.
- KIP-963: Increases observability into Tiered Storage operations and ultimately enables cluster operators to troubleshoot and prevent issues.
- KIP-580: Aims to reduce the common problem of slow metadata convergence after broker failure. Changes the static retry backoff mechanism to an exponentially increasing backoff policy for all Kafka clients.
- KIP-858 (Early Access): Allows JBOD-based deployments to be production capable with KRaft.
- KIP-896: Marks the use of old client API versions (Kafka versions older than 2.1.0) as deprecated, and introduces new metrics to monitor the presence of these requests. Support for these requests will be removed in Apache Kafka 4.0.
- KIP-938: Makes it easier for operators to understand how their KRaft-mode cluster is performing.
- KIP-979: Decouples KRaft controllers and Kafka brokers that may be running on the same machine, allowing them to be stopped independently from one another.
- KIP-1013: Marks the use of Java 11 in the Kafka broker as deprecated. Support for Java 11 will be removed in Apache Kafka 4.0.

Kafka Streams:
- KIP-925: Reduces cross-rack traffic by adding client rack awareness to Kafka Streams.
- KIP-954: Builds on KIP-591, providing a new interface for DslStoreSuppliers. It includes default implementations of this new interface for both RocksDB and in-memory state stores.
- KIP-988: Improves visibility into Kafka Streams StandbyTasks.
- KIP-992: Seeks to eliminate inconsistencies in query return types across regular and timestamp state stores.

Kafka Connect:
- KIP-976: Expands on KIP-495, adding broadcast capabilities to push log-level changes to all workers in the cluster. It eliminates the need to target an individual worker (which may not always be possible) for log-level updates.
- KIP-980: Allows users to create connectors in a stopped state, enabling a number of use cases, such as migrating connectors.
- KIP-959: Expands on KIP 907, adding BooleanConverter to Kafka Connect.

Many more KIPs are a part of this release. See the blog post for more details.

EPISODE LINKS

CHAPTERS
00:00 - Intro
00:44 - KIP-714: Client Metrics and Observability
01:16 - KIP-1000: List Client Metrics Configuration Resources
01:32 - KIP-848: The Next Generation of the Consumer Rebalance Protocol (Early Access)
01:58 - KIP-951: Leader Discovery Optimizations for the Client
02:24 - KIP-975: Docker Image for Apache Kafka
02:47 - KIP-963: Additional Metrics in Tiered Storage
02:56 - KIP-580: Exponential Backoff for Kafka Clients
03:33 - KIP-858: Handle JBOD Broker Disk Failure in KRaft
03:43 - KIP-896: Remove old client protocol API versions in Kafka 4.0
04:05 - KIP-938: Add more metrics for measuring KRaft performance
04:24 - KIP-979: Allow Independently stop KRaft Processes
04:54 - KIP-1013: Drop broker support for Java 11
05:20 - KIP-925: Rack Aware Task Assignment in Kafka Streams (Part 2)
05:42 - KIP-954: Expand default DSL store configuration to custom types
06:00 - KIP-988: Streams Standby Update Listener
06:20 - KIP-992: Introduce IQv2 Query Types
07:00 - KIP-976: Cluster-wide dynamic log adjustment
07:29 - KIP-980: Allow Creating Connectors in a Stopped State
07:53 - KIP-959: Add BooleanConverter to Kafka Connect
08:04 - It's a wrap!

--

#apachekafka #kafka #confluent

[toenytv]

I want a piece of that cake. Great job Cheffing the Kipcakes.😊

[jnelson]

I want all my software updated with cake!

[theaugmenter]

Now I need that cake recipe, looks delicious