Deploying Kafka on Kubernetes
Apache Kafka is an open source event streaming platform first developed by LinkedIn, then donated to the Apache software foundation. It is written in Java and Scala. Kafka aims to provide the user with a unified, high latency, high throughput platform for handling real-time (streaming) data. Kafka runs as a cluster of nodes called Kafka Brokers. It was developed first as a messaging queue and works as a publish-subscribe (pub-sub) model. It is used as a popular message queue for distributed systems, and is commonly used to stream data in the Internet of Things (IoT) use cases.
How Kafka Works in a Nutshell:
- Producers create messages and publish them to topics
- Kafka categorizes the messages into topics and stores them so that they are immutable
- Consumers subscribe to a specific topic and absorb the messages provided by the producers.
Zookeper in Kafka
Zookeeper is used in Kafka for choosing the controller, and is used for service discovery for a Kafka broker that deploys in a Kubernetes cluster. Zookeeper sends changes of the topology to Kafka, so nodes in the cluster know when something is new, broken, joined, finished, or the same topic is removed or a new topic is added. Therefore, Zookeeper must be deployed within the Kafka cluster for high availability.
Kafka on Kubernetes
There can be a variety of reasons why Kafka's architecture in Kubernetes is attractive. If your organization is standardizing the use of Kubernetes as an application/data platform, then this is a great reason to consider running Kafka there as well. Running Kafka on Kubernetes enables organizations to simplify operations such as updates, restarts, and monitoring that are more or less integrated into the Kubernetes platform.
As mentioned, Kafka depends on Zookeeper to track its configuration of what topics are added ot removed to served subscribers. So at first, we have to deploy the Zookeeper into the cluster (Note: This post assumes that you alreay have a cluster up and running).
Step 1: Deploying Zookeeper
kind: Deployment
apiVersion: apps/v1
metadata:
name: zookeeper-deploy
spec:
replicas: 2
selector:
matchLabels:
app: zookeeper-1
template:
metadata:
labels:
app: zookeeper-1
spec:
containers:
- name: zoo1
image: digitalwonderland/zookeeper
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_ID
value: "1"
- name: ZOOKEEPER_SERVER_1
value: zoo1
This will create a deployment object for Zookeper with two replicas. Containers created with digitalwonderland/zookeeper
exposing on Port 2181
with two environment variables:
ZOOKEEPER_ID
with value of1
and,ZOOKEPER_SERVER_1
with value ofzoo1
To create this resource, run the following command
$ kubectl create -f .\zooservice.yaml
Step 2: Exposing Zookeeper Service
apiVersion: v1
kind: Service
metadata:
name: zoo1
labels:
app: zookeeper-1
spec:
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper-1
This will create the service object for Zookeeper and route the traffic to the Zookeeper pod. This service has the name of zoo1
which we will use later when we deploy the Kafka brokers. To create this resource, run the following command
$ kubectl create -f .\zooservice.yaml
Step 3: Deploying Kafka
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
name: kafka
spec:
ports:
- port: 9092
name: kafka-port
protocol: TCP
selector:
app: kafka
id: "0"
type: LoadBalancer
To create this resource, run the following command
$ kubectl create -f .\zooservice.yaml
Now we have the service object, we can start the Kafka broker
Step 4: Deploying Kafka Broker
kind: Deployment
apiVersion: apps/v1
metadata:
name: kafka-broker0
spec:
replicas: 2
selector:
matchLabels:
app: kafka
id: "0"
template:
metadata:
labels:
app: kafka
id: "0"
spec:
containers:
- name: kafka
image: wurstmeister/kafka
ports:
- containerPort: 9092
env:
- name: KAFKA_ADVERTISED_PORT
value: "30718"
- name: KAFKA_ADVERTISED_HOST_NAME
value: 192.168.1.240
- name: KAFKA_ZOOKEEPER_CONNECT
value: zoo1:2181
- name: KAFKA_BROKER_ID
value: "0"
- name: KAFKA_CREATE_TOPICS
value: admintome-test:1:1
KAFKA_ADVERTISED_HOST_NAME
is set to the IP address we noted earlier. Also, note thar ew tell Kafka Broker to automatically create an admintome-test
theme with 1 partition and 1 replica. To crete this resouce, run the following command.
$ kubectl create -f .\zooservice.yaml
Tips and Practices for Running Kafka on Kubernetes
1. Low Latency Network and Storage
Kafka demands low latency for network and storage which means it must have low-cotnention, high-throughput, and low noising access storage. To provide this we have to use high-performance disks suchas SSD, and consider the location where data access for broker is local, and where the Pod will increase overall system performance.
Kafka performance depends upon low latency network and high bandwitdh. As with Kafka broker, don't attempt to put all brokers on a single node, as this would reduce availability. Different availability zones are an acceptable trade-off. If the storage in the container is not persistent, data will be lost after restart. EmptyDir
is used for Kafka data, and will persist if the container restarts. So if the container starts, the failing broker firt has to replicate all teh data which is a time-consuming process. That's why you should use a persistance volume, and storage has to be on non-local so that Kubernetes will be more flexible in choosing another node after restart or relocation.
2. High Availability for Kafka Brokers
Kafka runs as a cluster of brokers that can ben deployed across the Kubernetes system on different nodes. As Kubernetes can automatically recover nodes or containers by restarting, Kafka brokers can do the same. Kafka can rebuild brokers after node failure, but these rebuilds have a lower I/O cost to the application during the rebuild. Consider a data replication strategy that allows you to take advantage of multiple brokers for better performance, but also allows for faster failover in the event of a node failure.
3. Data Protection
Kafka replicates the topic across the cluster. With replication, you can achieve fault tolerance. Also, with data mirroring, you can have data available across data centers. Consider your performance requirements and use only as many brokers as you need to achieve that level of performance, while leveraging technologies that replicate data to other hosts in the cluster in order to maximize availability, performance and protection.
4. Data Security
Encryption of Data in-flight using SSL / TLS: This allows your data to be encrypted between your producers and Kafka, and also with your consumers and Kafka. This is a very common pattern that everyone has used when connecting to the web. However, keep in mund that, although Kafka brokers are CPU-friendly, TLS might introduce some overhead. And Kafka clients need more CPU if they are using encryption, but this does not impact the brokers.
Authentication Using SSL or SASL: This allows its producers and consumers to authenticate themselves to their Kafka cluster, which verifies their identitiy - it's also a secure way to allow your customers to expose an identity.
Authorization using ACLs: Once their clients are authenticated, their Kafka agents can run them on access control lists (ACLs) to determine whether or not a particular client would be authorized to write or read a topic.
5. Operator
The operator is a method of packaging, deploying, and managing a Kubernetes application. There are many operators in the market for Kafka. One of the most known one is Strimzi. This operator makes it easy to spin up the Kafka cluster in a minute. Using a Strimzi operator, you almost don't need any configuration for spinning up the cluster. It also adds a feature for point to point TLS encryption.
Confluent has also announced a Kubernetes operator for Kafka. It can be deployed almost any major cloud service provider.
6. Monitoring
Monitoring is a way to handle errors or management of clusters easily. Prometheus can collect metrics from all Java, Kafka, Zookeeper processes with JMX export as a built-in. Strimzi also has a Grafana dashboard for Kafka to visualize key metrics and supplement those metrics with resource usage and performance, as well as stability indicators. This way you get Kafka monitoring basic for free.
7. Health Check
Kubernetes uses liveness and readiness probes to find out if Pods are healthy or not. If the liveness prone fails, Kubernetes will kill the container and automatically restarts it. If the restart policy is configured accordingly. If the readiness probe fails, Kubernetes will remove the Pod from the serving requests. This means that manual intervention is no longer needed in such cases.
8. Logging
Logging is what makes it easier to debug an issue. All containers in your Kafka installation will log to stdout
and stderr
and aggregate all logs in a central logging infrastructure, such as ElasticSearch.
9. Memory
Kafka brokers are memory-eaters. The JVM heap can be usually limited to 4-5 GB, but you also need enough memory system because Kafka makes heavy use of the page cache. In Kubernetes, set the container resource limits and requests accordingly.
10. Performance
Running peformance tests to benchmark your Kafka installation is very important. It gives you insights about possible bottlenecks before you run into trouble. Luckily, Kafka already ships with two performance test tools: kafka-producer-perf-test.sh
and kafka-consumer-perf-test.sh
. Use them intensively. For reference you can see the results from this blog by Jaj Kreps, or this reviewhttps://medium.com/@stephane.maarek/an-honest-review-of-aws-managed-apache-kafka-amazon-msk-94b1ff9459d8 of AWS MSK by Stéphane Maarek
11. Rolling Updates
StatefulSets support automated updates: The RollingUpdate strategy will update each Kafka Pod one at a time. With this you can reach zero-downtime.
Conclusion
For small to medium sized Kafka clusters, Kubernetes provides flexibility and (relative) simplicity for operations. Since Kubernetes is becoming more and more dominant, it will not be surprising to see that it will become the defacto platform for Kafka also.