In the world of modern distributed systems and data processing, Apache Kafka (abbreviated as Kafka) has emerged as a powerful and widely adopted platform for real-time data streaming and event processing. In particular, Apache Kafka is widely used in microservices systems.
1. Concept :)
Refer Video: Kafka on Youtube
2. Prerequisites
1
2
sudo apt install default-jre curl && java -version
mkdir -p /etc/kafka /etc/zookeeper /tmp/zookeeper
2.1 Download Kafka
1
2
3
4
curl https://dlcdn.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz -o /etc/kafka/kafka.tgz
cd /etc/kafka
tar -xvzf /etc/kafka/kafka.tgz --strip 1
rm /etc/kafka/kafka.tgz
2.2 Download Zookeeper
1
2
3
4
curl https://dlcdn.apache.org/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz -o /etc/zookeeper/zookeeper.tgz
cd /etc/zookeeper
tar -xvzf /etc/zookeeper/zookeeper.tgz --strip 1
rm /etc/zookeeper/zookeeper.tgz
3. Kafka Standalone
Create Systemd Kafka, modify env KAFKA_HEAP_OPTS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cat << EOF > /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target
After=network.target
[Service]
Type=simple
User=root
Group=root
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
Environment=KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
ExecStart=/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties
ExecStop=/etc/kafka/bin/kafka-server-stop.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
Create Systemd Zookeeper. And remember copy file config templates :)
1
cp /etc/zookeeper/conf/zoo_sample.cfg /etc/zookeeper/conf/zoo.cfg
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
cat << EOF > /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/etc/zookeeper/bin/zkServer.sh start-foreground
ExecStop=/etc/zookeeper/bin/zkServer.sh stop
ExecReload=/etc/zookeeper/bin/zkServer.sh restart
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
EOF
Start Service
1
2
3
4
5
sudo systemctl daemon-reload
sudo systemctl enable zookeeper
sudo systemctl enable kafka
sudo systemctl start zookeeper
sudo systemctl start kafka
So. How it’s work? –> Click here
4. Kafka Distribute
4.1 Config Zookeeper on All Nodes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
cp /etc/zookeeper/conf/zoo_sample.cfg /etc/zookeeper/conf/zoo.cfg
cat << EOF > /etc/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=60
4lw.commands.whitelist=*
server.1=192.168.101.136:2888:3888
server.2=192.168.101.137:2889:3889
server.3=192.168.101.138:2890:3890
EOF
And myid for each node
1
echo 1 > /tmp/zookeeper/myid
4.2 Config Kafka on All Nodes
We modify like as
1
2
3
broker.id=1 # for each of nodes
listeners=PLAINTEXT://$HOSTNAME:9092 # IP Node
zookeeper.connect=192.168.101.136:2181,192.168.101.137:2181,192.168.101.138:2181
4.3 Reload service
1
2
3
sudo systemctl daemon-reload
sudo systemctl restart zookeeper
sudo systemctl restart kafka
5. Action
To continue reading the article please refer to Notion Kafka
5.1. Action For Standalone
5.2. Action For Distribute (Zookeeper or Kraft)
5.3. Manage kafka Cluster with Redpanda
5.4. Setup Monitoring Kafka Cluster
5.5. Observability Kafka Cluster
5.6. Migrate PLAINTEXT to SCRAM
Copy ID Json: 18276
Conclude
Follow Post i hope guys can become master kafka :D
Understand how Kafka works, how to deploy and use Kafka to transfer data between applications: Create and manage topics, send and receive messages through producers and consumers.
Test the stability and connectivity of the Kafka cluster.
Develop applications using Kafka: Creating a Kafka topic and deploying producers and consumers is an important step in developing applications using Kafka.
Kafka cluster monitoring deployment and management.
Comments powered by Disqus.