KAFKA-19648: Formatter refactoring #20707

mannoopj · 2025-10-15T15:43:45Z

KIP-1170
Change formatter code to ensure that when the directories are formatted through the kafka storage command, metadata records normally written to the bootstrap.checkpoint file are instead located in the 00000000000000000000-0000000000.checkpoint file. Then if any kraft records generated during this format they will be located after the metadata records in the 00000000000000000000-0000000000.checkpoint file. This PR is currently a draft and opened to allow for reviews to code changes to formatter for this kip.

kevin-wu24

Thanks for the changes @mannoopj. Some high level comments:

kevin-wu24 · 2025-10-15T16:14:11Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

    static void writeDynamicQuorumSnapshot(
        String writeLogDir,
        DynamicVoters initialControllers,
        short kraftVersion,
        String controllerListenerName
    ) {
-        File parentDir = new File(writeLogDir);
-        File clusterMetadataDirectory = new File(parentDir, String.format("%s-%d",
-                CLUSTER_METADATA_TOPIC_PARTITION.topic(),
-                CLUSTER_METADATA_TOPIC_PARTITION.partition()));
        VoterSet voterSet = initialControllers.toVoterSet(controllerListenerName);
        RecordsSnapshotWriter.Builder builder = new RecordsSnapshotWriter.Builder().
            setLastContainedLogTimestamp(Time.SYSTEM.milliseconds()).
            setMaxBatchSizeBytes(KafkaRaftClient.MAX_BATCH_SIZE_BYTES).
            setRawSnapshotWriter(FileRawSnapshotWriter.create(
-                clusterMetadataDirectory.toPath(),
+                Paths.get(writeLogDir),
                Snapshots.BOOTSTRAP_SNAPSHOT_ID)).
            setKraftVersion(KRaftVersion.fromFeatureLevel(kraftVersion)).
            setVoterSet(Optional.of(voterSet));
        try (RecordsSnapshotWriter<ApiMessageAndVersion> writer = builder.build(new MetadataRecordSerde())) {
            writer.freeze();
        }
    }


There is a clearer way to make these changes. This method is what writes the 0-0.checkpoint currently. We should pass the bootstrap metadata object here and append the metadata records using writer.append.append(bootstrapMetadata.records()) before calling writer.freeze().

kevin-wu24 · 2025-10-15T16:15:53Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

                if (directoryTypes.get(writeLogDir).isDynamicMetadataDirectory()) {
-                    writeDynamicQuorumSnapshot(writeLogDir,
+                    writeDynamicQuorumSnapshot(clusterMetadataDirectory.getPath(),


We should rename writeDynamicQuorumSnapshot to writeZeroSnapshot, and write it when formatting metadata directories. The semantics that change here are that we should not write the KRaft control records (KRaft version and voter set) when !isDynamicMetadataDirectory().

kevin-wu24 · 2025-10-15T16:21:17Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

+                    File createdBoostrapCheckpoint = new File(clusterMetadataDirectory.getPath() + "/" + BootstrapDirectory.BINARY_BOOTSTRAP_FILENAME);
+                    File created000Checkpoint = new File(clusterMetadataDirectory.getPath() + "/" + BootstrapDirectory.BINARY_CHECKPOINT_FILENAME);
+                    Files.write(
+                            createdBoostrapCheckpoint.toPath(),
+                            Files.readAllBytes(created000Checkpoint.toPath()),
+                            StandardOpenOption.APPEND);
+                    try {
+                        created000Checkpoint.delete();
+                        createdBoostrapCheckpoint.renameTo(created000Checkpoint);
+                    } catch (Exception ex) {
+                        throw new RuntimeException("Failed operation to combine metadata and kraft records: ", ex);
+                    }
+                } else {
+                    File createdBoostrapCheckpoint = new File(clusterMetadataDirectory.getPath() + "/" + BootstrapDirectory.BINARY_BOOTSTRAP_FILENAME);
+                    File created000Checkpoint = new File(clusterMetadataDirectory.getPath() + "/" + BootstrapDirectory.BINARY_CHECKPOINT_FILENAME);
+                    try {
+                        createdBoostrapCheckpoint.renameTo(created000Checkpoint);
+                    } catch (Exception ex) {
+                        throw new RuntimeException("Failed to rename file: ", ex);
+                    }


This code is confusing. Instead of doing this renaming and deleting. We should instead remove the call to write the bootstrap metadata to disk on Line 447, since we're no longer writing bootstrap.checkpoint anymore, and follow the other comments for writing metadata records to 0-0.checkpoint.

We can check if an old bootstrap.checkpoint exists and delete it, since IIRC that was part of the KIP.

kevin-wu24

Thanks for the changes @mannoopj. Left some more comments

kevin-wu24 · 2025-10-20T13:34:27Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

+        try (RecordsSnapshotWriter<ApiMessageAndVersion> writer = builder.build(new MetadataRecordSerde(), Optional.of(bootstrapMetadata.records()))) {
            writer.freeze();


Suggested change

try (RecordsSnapshotWriter<ApiMessageAndVersion> writer = builder.build(new MetadataRecordSerde(), Optional.of(bootstrapMetadata.records()))) {

writer.freeze();

try (RecordsSnapshotWriter<ApiMessageAndVersion> writer = builder.build(new MetadataRecordSerde()))) {

writer.append(bootstrapMetadata.records());

writer.freeze();

Wouldn't this write the bootstrap records after the control records, since in RecordsSnapshotWriter.build() is where we append the kraft records and we would be calling that ahead of writer.append in this scenario? we want the bootstrap records ahead correct?

It doesn't matter what order we write them in because KRaft only reads the control records, and the metadata module will only read the "data" records. When we read the 0-0.checkpoint back into memory, we only deal with either its control records or its data records, not both in the same code.

kevin-wu24 · 2025-10-20T13:34:55Z

core/src/test/scala/unit/kafka/tools/DumpLogSegmentsTest.scala

        .setKraftVersion(KRaftVersion.KRAFT_VERSION_1)
        .setVoterSet(Optional.of(VoterSetTest.voterSet(VoterSetTest.voterMap(IntStream.of(1, 2, 3), true))))
-        .build(MetadataRecordSerde.INSTANCE)
+        .build(MetadataRecordSerde.INSTANCE, emptyOptional)


We should be able to remove this. This applies to other instances where we build the RecordsSnapshotWriter.

kevin-wu24 · 2025-10-20T13:37:17Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

-                }
+                writeBoostrapSnapshot(writeLogDir,
+                    bootstrapMetadata,
+                    initialControllers.get(),


We need to pass the optional for initialControllers here. We can only do a .get() if initialControllers.isPresent().

kevin-wu24 · 2025-10-20T13:41:08Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

        File clusterMetadataDirectory = new File(parentDir, String.format("%s-%d",
                CLUSTER_METADATA_TOPIC_PARTITION.topic(),
                CLUSTER_METADATA_TOPIC_PARTITION.partition()));
        VoterSet voterSet = initialControllers.toVoterSet(controllerListenerName);


We should only construct the VoterSet object if initialControllers.isPresent().

kevin-wu24 · 2025-10-24T17:06:09Z

metadata/src/main/java/org/apache/kafka/controller/QuorumController.java

+                                // For bootstrap snapshots, extract feature levels from all data records
+                                if (batch.controlRecords().isEmpty()) {
+                                    bootstrapMetadata = BootstrapMetadata.fromRecords(messages, "bootstrap");
+                                }


The correct logic here is:
If the batch has records, read them into bootstrapMetadata (this means 0-0.checkpoint has bootstrap metadata records).
If the batch doesn't have records, try to read the bootstrapMetadata from bootstrap.checkpoint.

kevin-wu24

Thanks for the changes @mannoopj. Review of the metadata layer implementation:

kevin-wu24 · 2025-10-24T17:07:38Z

metadata/src/main/java/org/apache/kafka/metadata/bootstrap/BootstrapMetadata.java

-            if (level > 0) {
-                records.add(new ApiMessageAndVersion(new FeatureLevelRecord().
-                    setName(featureName).
-                    setFeatureLevel(level), (short) 0));
-            }
+            // Include all feature levels, including level 0 which may disable features
+            records.add(new ApiMessageAndVersion(new FeatureLevelRecord().
+                setName(featureName).
+                setFeatureLevel(level), (short) 0));


This should not change. The default level of features is 0, and that is why we don't add a record for them when the level is 0.

kevin-wu24 · 2025-10-24T17:10:39Z

metadata/src/main/java/org/apache/kafka/metadata/bootstrap/BootstrapDirectory.java

+                CLUSTER_METADATA_TOPIC_PARTITION.partition()),
+                BINARY_BOOTSTRAP_CHECKPOINT_FILENAME);
        if (!Files.exists(binaryBootstrapPath)) {
            return readFromConfiguration();


If we are at L76, that means we do 0-0.checkpoint doesn't exist. This is where we should read from bootstrap.checkpoint. If that doesn't exist too, then we call readFromConfiguration().

Actually, if you look at how we're reading stuff in, we probably don't even need to change this file. We don't need to call BootstrapDirectory#read for the 0-0.checkpoint because we are using handleLoadSnapshot, which already puts the checkpoint in memory for us.

kevin-wu24 · 2025-10-24T17:12:32Z

core/src/main/scala/kafka/server/KafkaRaftServer.scala

-    (metaPropsEnsemble, bootstrapMetadata)
+    // val bootstrapDirectory = new BootstrapDirectory(config.metadataLogDir)
+    // val bootstrapMetadata = bootstrapDirectory.read()
+    (metaPropsEnsemble, null)


We should remove bootstrapMetadata here and in ControllerServer since it is just being passed down to QuorumController eventually. We initialize it in QuorumController.

kevin-wu24 · 2025-10-24T17:24:23Z

...mmon/test-common-runtime/src/main/java/org/apache/kafka/common/test/KafkaClusterTestKit.java

+
+            // Copy feature levels from TestKitNodes bootstrap metadata to ensure test annotations are respected
+            for (var record : nodes.bootstrapMetadata().records()) {
+                if (record.message() instanceof FeatureLevelRecord featureLevelRecord) {
+                    String featureName = featureLevelRecord.name();
+                    short level = featureLevelRecord.featureLevel();
+                    // Don't override MetadataVersion as it's handled by setReleaseVersion()
+                    if (!featureName.equals("metadata.version")) {
+                        formatter.setFeatureLevel(featureName, level);
+                    }
+                }
+            }


Some background on how the KafkaClusterTestKit works:

Basically, tests that use this class are "integration tests" in the sense that we're trying to replicate a real cluster, just all within the same JVM. That means multiple brokerServers and controllerServers. This file shouldn't change outside of removing nodes.bootstrapMetadata() from the ControllerServer constructor.

github-actions bot added triage PRs from the community kraft small Small PRs labels Oct 15, 2025

mannoopj changed the title ~~KIP-1170: Formatter changes~~ KAFKA-19648: Formatter refactoring Oct 15, 2025

kevin-wu24 reviewed Oct 15, 2025

View reviewed changes

github-actions bot removed the triage PRs from the community label Oct 16, 2025

redo implementation to use RecordsSnapshotWriter

a9c79a3

mannoopj force-pushed the kip-1170-format branch from 42e105a to a9c79a3 Compare October 20, 2025 13:28

github-actions bot added the core Kafka Broker label Oct 20, 2025

mannoopj added 3 commits October 20, 2025 09:33

NIT fix spacing

2a6f663

NIT remove unused import

752a79e

NIT remove extra line

026da2b

kevin-wu24 reviewed Oct 20, 2025

View reviewed changes

mannoopj added 3 commits October 20, 2025 11:42

add bootstrap records in formatter

7e303b9

revert RaftClientTestContext

584fe3d

WIP kip-1170 refactoring test infra

462e072

github-actions bot added clients and removed small Small PRs labels Oct 24, 2025

kevin-wu24 reviewed Oct 24, 2025

View reviewed changes

		try (RecordsSnapshotWriter<ApiMessageAndVersion> writer = builder.build(new MetadataRecordSerde(), Optional.of(bootstrapMetadata.records()))) {
		writer.freeze();

Uh oh!

KAFKA-19648: Formatter refactoring #20707

Are you sure you want to change the base?

KAFKA-19648: Formatter refactoring #20707

Uh oh!

Conversation

mannoopj commented Oct 15, 2025

Uh oh!

kevin-wu24 left a comment

Choose a reason for hiding this comment

Uh oh!

kevin-wu24 Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin-wu24 Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin-wu24 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mannoopj Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin-wu24 Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin-wu24 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin-wu24 Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kevin-wu24 Oct 15, 2025 •

edited

Loading

kevin-wu24 Oct 15, 2025 •

edited

Loading

mannoopj Oct 20, 2025 •

edited

Loading

kevin-wu24 Oct 20, 2025 •

edited

Loading

kevin-wu24 Oct 24, 2025 •

edited

Loading