Setting up multiple kafka cluster configurations to explore the nuances of various tools that monitor Apache Kafka.
Insights
The published Apache Druid container at Docker is a linux/amd64 only image. Running this on your Apple Silicon (M1 or M2 chipset) is slow.
Setting up Change Data Capture with Databases, Apache Kafka, Kafka Connect, and Debezium takes time; with tricky configurations along the way. Here we will walk through a setup of all components to showcase what is possible and give the complete picture to give you the pieces to bring this into your project.
Allowing developers, data engineers, or any one interested in real-time technologies the ability to explore tools locally is a great advantage. I have heard many stories of developers leaving containers running over the weekend, only to use to many of their free developer credits with a cloud provider. I’ve also seen developers not fully explore various configurations, nuances, or tools fearing they would have to expense unexpected charges.
Every distribution inspected to extract out specific library versions and key dependencies.
This is helpful when you are tracking an open source security issue, want to know a version of a library to know if a feature is available, or to minimize dependency graph with your own java software that leverages kafka-client or kafka-streams libraries.