KTools - Kafka Topic Truncate

Truncating an Apache Kafka Topic is not an easy task. Typically, I have done this by deleting and recreating a topic, which isn’t always easy or possible. While truncating a topic is something that should never be done in production, it is quite useful in development. With this in mind, kafka-topic-truncate CLI was created.

Introduction

Command-line tools are the cornerstone to Kafka development. Here is a tool added to support our development process; and hoping it can help you. This project is licensed with the Apache License Version 2.0.

Why?

This tool is more a convenience than a requirement for developer success. The goal here is to make it easier for a developer to reset their environment as easily as possible. Publishing Avro to a JSON topic will lead to serialization exceptions; while offsets can be reset, those messages are still there and could impact them again and again as they work through their consumer development; this allows for quick removal and restart.

Cluster Prevents Topic Deletion

Not all clusters have (or give) developers the ability to delete topics. This tool allows them to remove contents from a topic, which would be otherwise be impossible.

Truncating an Active Topic

Another advantage of this tool, is that it can be performed on a topic that has active producer and consumed; something that cannot be done when a topic is deleted and recreated.

Important Features

Requires --execute to do the truncation

This mimics the approach used by the Apache Kafka CLI, such as kafka-consumer-groups. When an action is to be performed where no rollback available, the use of --execute gives additional (although minor) “are you sure” feature.

Requires --force if the topic does not have a delete cleanup policy.

There are 2 cleanup policies for a topic, delete and compact. The Admin API cannot perform deleteRecords on a topic that does not have a delete cleanup policy. This tool will temporarily add delete cleanup policy to the topic so the delete operation can be performed. Now, because of this additional action needed, the use of --force is required for a topic w/out a delete cleanup policy.

If there was an exception performed during this operation, please validate that the cleanup.policy was restored. The reset is in a finally block, but it could fail too.

• This is only required for a topic that does not have a delete cleanup policy. In other words, a cleanup policy of compact,delete does not require the --force flag.

Project

This CLI tool is the first tool to be added to the KTools opened-source project. The project will have a .tar release for each version. The project is written in Java, and leverages the Kafka Admin API for performing the delete operation. As time allows, additional CLI tools I have found to be beneficial in developing Kafka applications will be added to this project.

The ktools-cli submodule leverages picocli library for providing CLI operations to this project; all future CLI tools will just become a new subcommand, and scripts will be added to the bin directory to call the java jar command.

The project, as stated earlier, is licensed under the Apache License Version 2.0.

Finally, this project is brand new; hence the 0.0.x release. As will all code you download, please verify it is developed to your liking.

Reach out

Please contact us if you have ideas or suggestions. Use the project’s GitHub Issues page for specifics for the project.