Skip to content

Docker With Kafka#

What Is The Kafka?#

  • Apache Kafka is a distributed data streaming platform that can publish, subscribe to, store, and process streams of records in real time. It is designed to handle data streams from multiple sources and deliver them to multiple consumers. In short, it moves massive amounts of data—not just from point A to B, but from points A to Z and anywhere else you need, all at the same time.
  • More information

Create Kafka With Docker#

  • Let's create a file name docker-compose.yml, then add scripts as below.
docker-compose.yml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
version: '3.4'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.2
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.3.2
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_HOST_NAME: broker
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LISTENERS: INSIDE://:19092,OUTSIDE://:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://broker:19092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  • Okay, we can see there are two service zookeeper and broker.
  • Firstly, we have the details for the zookeeper service as in the image below.
Field Value Description
Image confluentinc/cp-zookeeper:7.3.2 The Docker image used for the ZooKeeper service. It is pulled from the Confluent repository with version 7.3.2.
Container name zookeeper The name assigned to the ZooKeeper container.
Environment Variables
ZOOKEEPER_CLIENT_PORT 2181 Specifies the client port used to connect to ZooKeeper.
ZOOKEEPER_TICK_TIME 2000 Specifies the tick time in milliseconds used by ZooKeeper for session timeouts.
Networks elk Specifies the network(s) that the ZooKeeper container is connected to.
  • Next, for borker service or kafka we have the configurations as below.
Field Value Description
Image confluentinc/cp-kafka:7.3.2 The Docker image used for the Kafka broker service. It is pulled from the Confluent repository with version 7.3.2.
Container name broker The name assigned to the Kafka broker container.
Ports "9092:9092" Maps the host's port 9092 to the container's port 9092. This allows external access to Kafka on port 9092.
Depends On zookeeper Specifies that this service depends on the ZooKeeper service.
Environment Variables
KAFKA_BROKER_ID 1 Specifies the ID of the Kafka broker.
KAFKA_ZOOKEEPER_CONNECT 'zookeeper:2181' Specifies the connection string to ZooKeeper.
KAFKA_ADVERTISED_HOST_NAME broker Specifies the advertised host name for the Kafka broker.
KAFKA_INTER_BROKER_LISTENER_NAME INSIDE Specifies the name of the inter-broker listener.
KAFKA_LISTENERS INSIDE://:19092,OUTSIDE://:9092 Specifies the listeners for Kafka.
KAFKA_ADVERTISED_LISTENERS INSIDE://broker:19092,OUTSIDE://localhost:9092 Specifies the advertised listeners for Kafka.
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT Specifies the listener security protocol mapping.
created: 2023-07-01 11:05
modified: 2023-09-03 09:55
## Testing
- Now you can use command docker-compose up to start our Kafka service.
- Then we can use the command below to create a kafka topic. 
1
docker exec broker kafka-topics --bootstrap-server broker:9092 --create --replication-factor 1 --partitions 1 --topic sampleTopic
  • Now, to verify the topic, let's access to the kafka conteiner.
1
docker exec -it broker /bin/bash
  • Then use the command below to check list out all created kafka topics.
1
kafka-topics --list --bootstrap-server localhost:9092

1
2
[appuser@a7c57b707699 ~]$ kafka-topics --list --bootstrap-server localhost:9092
sampleTopic