Kubernetes Patterns - Application Process Management

Many programming language frameworks implement the concept of lifecycle management. The term refers to how the platform can interact with the component it creates right after it starts or before it stops. The implementation is important because sometimes we may need to perform some actions on the Pod such as testing for connectivity to one or more of its dependencies. Similarly, the Pod may need to undergo cleanup activities before the Pod is destroyed. In the application process management pattern, we ensure that our containerized application is aware of its environment and correctly responds to the different signals that the platform (Kubernetes) sends to it.

image

How Kubernetes Terminates it Pods

Through its lifecycle, a container may get terminated. Perhaps because the Pod it belongs to is being shut down or failing one or both of the liveness and readiness proes. In all cases, Kubernetes follows a standard way of destroying a running container: by sending kill signals. In the end, a container is just a process running on the machine. First Kubernetes sends the SIGTERM signal. The SIGTERM signal is sent by default when you issue the kill command against a process running on your Linux system.

SIGTERM allows the running process to perform any required cleanup activities before shutdown such as releasing file locks, closing database and network connections, and so on. Even so, sometimes the process (the container in our case) does not respond to the SIGTERM signal. Either because of a code bug that put the process in an infinite loop or for other reasons. Because of this, Kubernetes waits for a grace period of thirty seconds (which is an override metric) before it sends the more aggressive signal SIGKILL.

SIGKILL is the same signal sent to a running process when you issue the popular kill -9 command and hand it the process id. The process does not receive a SIGKILL signal but by the underlying operating system. Once the kernel detects this signal, it stops provisioning any resources to the process in question. The kernel also stops any CPU threads currently in use by the dangling process. In other words, it cuts the power off the process, forcing to die.

Communicating with Containers When They Start and Before They Terminate

So far, Kubernetes treats containers the same way any Linux system administrator deals with the running process: sending signals to the process or the kernel. But because containers are part of larger application with complex functions and tasks, signals are not enough. For that reason, Kubernetes offers postStart and preStop hooks.

Executing Commands when the Container Starts with the postStart Hook

You can think of a hook as a placeholder for executing code at a specific stage. You may or may not use the hook depending on you needs. The postStart hook is a placeholder for any logic that you need to execute as soon as the container starts. Example use cases are many:

  • If the container is a client to an external API, you have to make sure that the API is up and running and capable of responding to requests before your container comes to service.
  • When the container needs to execute some activities before the primary process launches such as resetting users' passwords or logging events to a log or database.
  • When there is a need to fulfill a specific condition before the container comes into service, for example you may probe an external API for several seconds; if the check fails after that number of seconds passes, the prober retuns a non-zero exit code. When the non-zero exit code is returned, Kubernetes automatically kills the container's main process.

Let's have a quick example: the following yml definition will start a Pod hosting one container. The container needs to ensure that a dependency service is available. Otherwise, the whole container should get killed:

apiVersion: v1
kind: Pod
metadata:
 name: client
spec:
 containers:
   - image: nginx
     name: client
     lifecycle:
       postStart:
         exec:
           command:
             - sh
             - -c
             - sleep 10 && exit 1

When you apply the above definition to the cluster kubectl apply -f portstart.yml and have a look at the Pods status, you will find out that the client Pod is always in the ContainerCreating status:

$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    6s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    9s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    11s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    14s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    19s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    22s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    27s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    29s
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    43s

This is what happened in sequence:

  1. Kubernetes pulled the nginx image
  2. It created the container and prepared to start it
  3. Because we have a lifecycle stanza within the definition, Kubernetes executes the postStart hook and scheduled bringing the container up till the hook script is finished.
  4. The postStart script pauses the thread for ten seconds before it returns a non-zero exit status
  5. When Kubernetes detects the non-zero exit status, it kills and restarts the container again, and the whole cycle repeats indefinitely.

We can make nginx start after ten seconds (which simulates any precheck activities) by altering the postStart script so that the definition looks as follows

apiVersion: v1
kind: Pod
metadata:
name: client
spec:
containers:
 - image: nginx
  name: client
  lifecycle:
   postStart:
    exec:
     command:
      - sh
      - -c
      - sleep 10

When we apply the new configuration

$ kubectl apply -f poststart.yml
pod/client created
$ kubectl get pods
NAME    READY STATUS       RESTARTS AGE
client 0/1    ContainerCreating 0    4s
$ kubectl get pods
NAME    READY STATUS    RESTARTS AGE
client 1/1    Running    0      22s

As you can see from the above, Kubernetes executed the portStart script then started the contaier's main ENTRYPOINT which is the nginx daemon. image

postStart Script Methods

postStart script uses the following methods for running the checks:

  • exec: Used in the preceding example, the exec method executes one or more arbitrary commands against the container. The exit status specifies whether or not the check has passed.
  • httpGet: Opens an HTTP connection to a local port on the container. You can optionally supply a path. For example, if we can modify the preceding example to check whether or not port 8080 is open and the /status endpoint path returns a valid response:
apiVersion: v1
kind: Pod
metadata:
name: client
spec:
containers:
 - image: mynginx
  name: client
  lifecycle:
   postStart:
    httpGet:
    port: 8080
    path: /status

Why Not Use an init Container Instead?

init container is a Kubernetes feature that allows a container to start and do one or more tasks, then it gets terminated. The init container starts and stops before other containers do, making it the right candidate for performing any pre-launch tasks. However, with postStart hooks and init containers appear to do similar jobs, the implementation is most different. Let's have a quick comparison between both methods and demonstrate possible use cases for each:

  • postStart scripts are executed using the same image as of the main container. Init containers can use the same or a different image than the one used by subsequent containers. So, if the tasks that you need to perfrom require a different base image, then you would better use init containers
  • postStart scripts are executed inside the same container. So, if the script you need to run is tightly coupled with the container, for example, you need to make configuration changes to the container itself, you should use postStart scripts.
  • All init containers must finish before the primary containers start. On the other hand, postStart scripts are specific to each container. So, container A uses its postStart scripts and launches, container B executes its postStart scripts and starts and so on. So, if you are hosting more than one container on the same Pod and you need to run one or more container-agnostic tasks, you should use init containers. However, if each init scripts is specific to its container, then a postStart hook is the suitable choice.
  • Init containers start and stop before any other container starts. postStart scripts get executed in parallel with their containers. This means that the script may or may not run the ENTRYPOINT of the containers kicks in. If you need to be confident that pre-launch logic is always executed before the main container does, then use init continers.
  • Because of how they are designed, postStart scripts may run many times. The application logi should be able to deal with this numerous execution possibility. For example, if the postStart script adds a new temporary user account before the container runs, then it should first check whether the user has already been created so that it does not return an incorrect non-zero exit status.

image

Executing Commands Against the Container Before It Terminates Using the preStop Hook

Earlier in this post, we learned about the different signals that Kubernetes sends to the running containers inside the Pods when it wants to bring them down. However, although the container receives the SIGTERM and it allows the container to shut down for up to thirty seconds, this may not be sufficient for complex scenarios.

Let's continue with out preceding example, and let's day that the RESTful API service that is running in parallel with out nginx service needs to perform several steps before it shuts down. The API designers were smart enough to expose an endpoint specifically for this purpose: executing the shutdown procedure. Kubernetes provide the preStop hook, which gets called right before SIGTERM signal is sent to the container. The preStop hook also provides the same check methods as the postStart hook: httpGet and exec.

However, unlike the portStart hook, if Kubernetes detects a non-zero exist status or a non-success HTTP code, it will continue the shutdown procedure and send the SIGTERM signal.

The main difference between a traditional and a cloud-native application is that the latter does not run on an infrastructure that you own or under your control. Orchestration platforms like Kubernetes were designed to ensure that you get the highest level of application performance and availability given an unpredictable infrastructure. Accordingly, cloud-native applications should be written in a way that honors the contracts and constraints imposed by Kubernetes to enjoy the features it provides.

blog

copyright©2021 ylcnky all rights reserved