This document was translated from polish original with GPT-3.5 and slightly corrected manually.

How to run Kafka in a very simple scenario

I recently learned about Kafka. To do that, I wanted to set up several Kafka brokers on my computer, send a message, and receive it. Here are my notes on how I did it.

The plan is as follows: I will run ZooKeeper and six brokers under Docker (I already have it installed). I will configure a topic with two partitions and triple replication. I will send a message using the command-line tool kafkacat, and then I will receive it using the command-line tool kafkacat as well.

I'm installing kafkacat:
$ sudo apt-get install kafkacat

If necessary, clean up Docker, especially if I've done these experiments before:
$ docker rm -f $(docker ps -aq)
$ docker network prune

I'm pulling the Docker image with ZooKeeper and the Kafka broker image:
$ docker pull confluentinc/cp-zookeeper:latest
$ docker pull confluentinc/cp-kafka:latest

I'm creating a network in Docker for the brokers and ZooKeeper to communicate:
$ docker network create kafka

I'm starting ZooKeeper:
$ docker run -d --network=kafka --name=zookeeper -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest

I'm starting six brokers:
$ docker run -d --network=kafka --name=broker1 -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_BROKER_ID=1 -e KAFKA_LISTENERS=PLAINTEXT://broker1:29092,PLAINTEXT_HOST://:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker1:29092,PLAINTEXT_HOST://localhost:9092 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT confluentinc/cp-kafka:latest
$ docker run -d --network=kafka --name=broker2 -p 9093:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_BROKER_ID=2 -e KAFKA_LISTENERS=PLAINTEXT://broker2:29092,PLAINTEXT_HOST://:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker2:29092,PLAINTEXT_HOST://localhost:9093 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT confluentinc/cp-kafka:latest
$ docker run -d --network=kafka --name=broker3 -p 9094:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_BROKER_ID=3 -e KAFKA_LISTENERS=PLAINTEXT://broker3:29092,PLAINTEXT_HOST://:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker3:29092,PLAINTEXT_HOST://localhost:9094 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT confluentinc/cp-kafka:latest
$ docker run -d --network=kafka --name=broker4 -p 9095:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_BROKER_ID=4 -e KAFKA_LISTENERS=PLAINTEXT://broker4:29092,PLAINTEXT_HOST://:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker4:29092,PLAINTEXT_HOST://localhost:9095 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT confluentinc/cp-kafka:latest
$ docker run -d --network=kafka --name=broker5 -p 9096:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_BROKER_ID=5 -e KAFKA_LISTENERS=PLAINTEXT://broker5:29092,PLAINTEXT_HOST://:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker5:29092,PLAINTEXT_HOST://localhost:9096 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT confluentinc/cp-kafka:latest
$ docker run -d --network=kafka --name=broker6 -p 9097:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_BROKER_ID=6 -e KAFKA_LISTENERS=PLAINTEXT://broker6:29092,PLAINTEXT_HOST://:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker6:29092,PLAINTEXT_HOST://localhost:9097 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT confluentinc/cp-kafka:latest

I'm configuring the topic:
$ docker exec -it broker1 kafka-topics --create --topic viewed_movies --partitions 2 --replication-factor 3 --bootstrap-server broker1:9092

Out of curiosity, I'm examining the topic configuration to see which broker is in which partition, which brokers are leaders, etc.:
$ docker exec -it broker1 kafka-topics --describe --topic viewed_movies --bootstrap-server broker1:9092

I'm sending a message, using any broker as the bootstrap:
$ echo '12345:{"userId": 12345, "movie": "Nad Niemnem", "date": "2023-07-06"}' | kafkacat -P -b localhost:9093 -t viewed_movies -K:

I'm receiving a message, using any broker as the bootstrap:
$ kafkacat -C -b localhost:9092 -t viewed_movies

I'm inspecting which messages are on which partitions:
$ kafkacat -C -b localhost:9092 -t viewed_movies -p 0
$ kafkacat -C -b localhost:9092 -t viewed_movies -p 1

Out of curiosity, I'm viewing the logs of the brokers:
$ docker logs broker1 | less