Deploying Kafka on Kubernetes

Apache Kafka is an open source event streaming platform first developed by LinkedIn, then donated to the Apache software foundation. It is written in Java and Scala. Kafka aims to provide the user with a unified, high latency, high throughput platform for handling real-time (streaming) data. Kafka runs as a cluster of nodes called Kafka Brokers. It was developed first as a messaging queue and works as a publish-subscribe (pub-sub) model. It is used as a popular message queue for distributed systems, and is commonly used to stream data in the Internet of Things (IoT) use cases.

image

How Kafka Works in a Nutshell:

  • Producers create messages and publish them to topics
  • Kafka categorizes the messages into topics and stores them so that they are immutable
  • Consumers subscribe to a specific topic and absorb the messages provided by the producers.

Zookeper in Kafka

Zookeeper is used in Kafka for choosing the controller, and is used for service discovery for a Kafka broker that deploys in a Kubernetes cluster. Zookeeper sends changes of the topology to Kafka, so nodes in the cluster know when something is new, broken, joined, finished, or the same topic is removed or a new topic is added. Therefore, Zookeeper must be deployed within the Kafka cluster for high availability.

Kafka on Kubernetes

There can be a variety of reasons why Kafka's architecture in Kubernetes is attractive. If your organization is standardizing the use of Kubernetes as an application/data platform, then this is a great reason to consider running Kafka there as well. Running Kafka on Kubernetes enables organizations to simplify operations such as updates, restarts, and monitoring that are more or less integrated into the Kubernetes platform.

As mentioned, Kafka depends on Zookeeper to track its configuration of what topics are added ot removed to served subscribers. So at first, we have to deploy the Zookeeper into the cluster (Note: This post assumes that you alreay have a cluster up and running).

Step 1: Deploying Zookeeper

kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper-deploy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: zookeeper-1
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1

This will create a deployment object for Zookeper with two replicas. Containers created with digitalwonderland/zookeeper exposing on Port 2181 with two environment variables:

  • ZOOKEEPER_ID with value of 1 and,
  • ZOOKEPER_SERVER_1 with value of zoo1

To create this resource, run the following command

$ kubectl create -f .\zooservice.yaml

Step 2: Exposing Zookeeper Service

apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

This will create the service object for Zookeeper and route the traffic to the Zookeeper pod. This service has the name of zoo1 which we will use later when we deploy the Kafka brokers. To create this resource, run the following command

$ kubectl create -f .\zooservice.yaml

Step 3: Deploying Kafka

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "0"
  type: LoadBalancer

To create this resource, run the following command

$ kubectl create -f .\zooservice.yaml

Now we have the service object, we can start the Kafka broker

Step 4: Deploying Kafka Broker

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kafka-broker0
spec:
  replicas: 2
  selector:
    matchLabels:
        app: kafka
        id: "0"
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.1.240
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_CREATE_TOPICS
          value: admintome-test:1:1

KAFKA_ADVERTISED_HOST_NAME is set to the IP address we noted earlier. Also, note thar ew tell Kafka Broker to automatically create an admintome-test theme with 1 partition and 1 replica. To crete this resouce, run the following command.

$ kubectl create -f .\zooservice.yaml

image

Tips and Practices for Running Kafka on Kubernetes

1. Low Latency Network and Storage

Kafka demands low latency for network and storage which means it must have low-cotnention, high-throughput, and low noising access storage. To provide this we have to use high-performance disks suchas SSD, and consider the location where data access for broker is local, and where the Pod will increase overall system performance.

Kafka performance depends upon low latency network and high bandwitdh. As with Kafka broker, don't attempt to put all brokers on a single node, as this would reduce availability. Different availability zones are an acceptable trade-off. If the storage in the container is not persistent, data will be lost after restart. EmptyDir is used for Kafka data, and will persist if the container restarts. So if the container starts, the failing broker firt has to replicate all teh data which is a time-consuming process. That's why you should use a persistance volume, and storage has to be on non-local so that Kubernetes will be more flexible in choosing another node after restart or relocation.

2. High Availability for Kafka Brokers

Kafka runs as a cluster of brokers that can ben deployed across the Kubernetes system on different nodes. As Kubernetes can automatically recover nodes or containers by restarting, Kafka brokers can do the same. Kafka can rebuild brokers after node failure, but these rebuilds have a lower I/O cost to the application during the rebuild. Consider a data replication strategy that allows you to take advantage of multiple brokers for better performance, but also allows for faster failover in the event of a node failure.

3. Data Protection

Kafka replicates the topic across the cluster. With replication, you can achieve fault tolerance. Also, with data mirroring, you can have data available across data centers. Consider your performance requirements and use only as many brokers as you need to achieve that level of performance, while leveraging technologies that replicate data to other hosts in the cluster in order to maximize availability, performance and protection.

4. Data Security

Encryption of Data in-flight using SSL / TLS: This allows your data to be encrypted between your producers and Kafka, and also with your consumers and Kafka. This is a very common pattern that everyone has used when connecting to the web. However, keep in mund that, although Kafka brokers are CPU-friendly, TLS might introduce some overhead. And Kafka clients need more CPU if they are using encryption, but this does not impact the brokers.

Authentication Using SSL or SASL: This allows its producers and consumers to authenticate themselves to their Kafka cluster, which verifies their identitiy - it's also a secure way to allow your customers to expose an identity.

Authorization using ACLs: Once their clients are authenticated, their Kafka agents can run them on access control lists (ACLs) to determine whether or not a particular client would be authorized to write or read a topic.

5. Operator

The operator is a method of packaging, deploying, and managing a Kubernetes application. There are many operators in the market for Kafka. One of the most known one is Strimzi. This operator makes it easy to spin up the Kafka cluster in a minute. Using a Strimzi operator, you almost don't need any configuration for spinning up the cluster. It also adds a feature for point to point TLS encryption.

Confluent has also announced a Kubernetes operator for Kafka. It can be deployed almost any major cloud service provider.

6. Monitoring

Monitoring is a way to handle errors or management of clusters easily. Prometheus can collect metrics from all Java, Kafka, Zookeeper processes with JMX export as a built-in. Strimzi also has a Grafana dashboard for Kafka to visualize key metrics and supplement those metrics with resource usage and performance, as well as stability indicators. This way you get Kafka monitoring basic for free.

7. Health Check

Kubernetes uses liveness and readiness probes to find out if Pods are healthy or not. If the liveness prone fails, Kubernetes will kill the container and automatically restarts it. If the restart policy is configured accordingly. If the readiness probe fails, Kubernetes will remove the Pod from the serving requests. This means that manual intervention is no longer needed in such cases.

8. Logging

Logging is what makes it easier to debug an issue. All containers in your Kafka installation will log to stdout and stderr and aggregate all logs in a central logging infrastructure, such as ElasticSearch.

9. Memory

Kafka brokers are memory-eaters. The JVM heap can be usually limited to 4-5 GB, but you also need enough memory system because Kafka makes heavy use of the page cache. In Kubernetes, set the container resource limits and requests accordingly.

10. Performance

Running peformance tests to benchmark your Kafka installation is very important. It gives you insights about possible bottlenecks before you run into trouble. Luckily, Kafka already ships with two performance test tools: kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh. Use them intensively. For reference you can see the results from this blog by Jaj Kreps, or this reviewhttps://medium.com/@stephane.maarek/an-honest-review-of-aws-managed-apache-kafka-amazon-msk-94b1ff9459d8 of AWS MSK by Stéphane Maarek

11. Rolling Updates

StatefulSets support automated updates: The RollingUpdate strategy will update each Kafka Pod one at a time. With this you can reach zero-downtime.

Conclusion

For small to medium sized Kafka clusters, Kubernetes provides flexibility and (relative) simplicity for operations. Since Kubernetes is becoming more and more dominant, it will not be surprising to see that it will become the defacto platform for Kafka also.

blog

copyright©2021 ylcnky all rights reserved