Guide to Setting Up Apache Kafka Using Docker

This post explains how to setup Kafka cluster locally using Docker.

In my previous blog post (linked below) , I explained about setting up Kafka cluster using Kafka binary and briefly explained about Kafka architecture. In this blog post I will explain how to setup kafka cluster using docker compose

Installing Kafka Using Docker

You can start the Kafka cluster locally using Docker in case if you do not want to use Kafka binary

1. Create docker compose file

First Copy and paste below text into a file named docker-compose.yml in your local filesystem

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.0.1
    container_name: kafka
    ports:    
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

2. Start Kafka Cluster

Open command prompt and navigate to the folder where the docker-compose file was created in first step and run below command. Below command will start the kafka cluster in the background.

docker-compose up -d

3. Create a topic

Run following command to create a new topic into which we’ll write and read some test messages.

 docker exec kafka kafka-topics --bootstrap-server kafka:9092 --create --topic order-events

4. Write messages to the topic

docker exec --interactive --tty kafka kafka-console-producer --bootstrap-server kafka:9092 --topic order-events

Now enter few order messages on the command prompt

>This is sample order event 1
>This is sample order event 2

5.Read messages from the topic

docker exec --interactive --tty kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic order-events --from-beginning

The above command will read messages entered in step 4.

This is sample order event 1
This is sample order event 2

6.Stop Kafka Cluster

docker-compose down

References

https://kafka.apache.org/documentation/

https://docs.confluent.io/