Kubernetes Patterns - Application Process Management
Many programming language frameworks implement the concept of lifecycle management. The term refers to how the platform can interact with the component it creates right after it starts or before it stops. The implementation is important because sometimes we may need to perform some actions on the Pod such as testing for connectivity to one or more of its dependencies. Similarly, the Pod may need to undergo cleanup activities before the Pod is destroyed. In the application process management pattern, we ensure that our containerized application is aware of its environment and correctly responds to the different signals that the platform (Kubernetes) sends to it.
How Kubernetes Terminates it Pods
Through its lifecycle, a container may get terminated. Perhaps because the Pod it belongs to is being shut down or failing one or both of the liveness and readiness proes. In all cases, Kubernetes follows a standard way of destroying a running container: by sending kill signals. In the end, a container is just a process running on the machine. First Kubernetes sends the
SIGTERM signal. The
SIGTERM signal is sent by default when you issue the kill command against a process running on your Linux system.
SIGTERM allows the running process to perform any required cleanup activities before shutdown such as releasing file locks, closing database and network connections, and so on. Even so, sometimes the process (the container in our case) does not respond to the
SIGTERM signal. Either because of a code bug that put the process in an infinite loop or for other reasons. Because of this, Kubernetes waits for a grace period of thirty seconds (which is an override metric) before it sends the more aggressive signal
SIGKILL is the same signal sent to a running process when you issue the popular
kill -9 command and hand it the process id. The process does not receive a
SIGKILL signal but by the underlying operating system. Once the kernel detects this signal, it stops provisioning any resources to the process in question. The kernel also stops any CPU threads currently in use by the dangling process. In other words, it cuts the power off the process, forcing to die.
Communicating with Containers When They Start and Before They Terminate
So far, Kubernetes treats containers the same way any Linux system administrator deals with the running process: sending signals to the process or the kernel. But because containers are part of larger application with complex functions and tasks, signals are not enough. For that reason, Kubernetes offers
Executing Commands when the Container Starts with the
You can think of a hook as a placeholder for executing code at a specific stage. You may or may not use the hook depending on you needs. The
postStart hook is a placeholder for any logic that you need to execute as soon as the container starts. Example use cases are many:
- If the container is a client to an external API, you have to make sure that the API is up and running and capable of responding to requests before your container comes to service.
- When the container needs to execute some activities before the primary process launches such as resetting users' passwords or logging events to a log or database.
- When there is a need to fulfill a specific condition before the container comes into service, for example you may probe an external API for several seconds; if the check fails after that number of seconds passes, the prober retuns a non-zero exit code. When the non-zero exit code is returned, Kubernetes automatically kills the container's main process.
Let's have a quick example: the following yml definition will start a Pod hosting one container. The container needs to ensure that a dependency service is available. Otherwise, the whole container should get killed:
apiVersion: v1 kind: Pod metadata: name: client spec: containers: - image: nginx name: client lifecycle: postStart: exec: command: - sh - -c - sleep 10 && exit 1
When you apply the above definition to the cluster
kubectl apply -f portstart.yml and have a look at the Pods status, you will find out that the client Pod is always in the ContainerCreating status:
$ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 6s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 9s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 11s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 14s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 19s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 22s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 27s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 29s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 43s
This is what happened in sequence:
- Kubernetes pulled the nginx image
- It created the container and prepared to start it
- Because we have a lifecycle stanza within the definition, Kubernetes executes the
postStarthook and scheduled bringing the container up till the hook script is finished.
postStartscript pauses the thread for ten seconds before it returns a non-zero exit status
- When Kubernetes detects the non-zero exit status, it kills and restarts the container again, and the whole cycle repeats indefinitely.
We can make nginx start after ten seconds (which simulates any precheck activities) by altering the
postStart script so that the definition looks as follows
apiVersion: v1 kind: Pod metadata: name: client spec: containers: - image: nginx name: client lifecycle: postStart: exec: command: - sh - -c - sleep 10
When we apply the new configuration
$ kubectl apply -f poststart.yml pod/client created $ kubectl get pods NAME READY STATUS RESTARTS AGE client 0/1 ContainerCreating 0 4s $ kubectl get pods NAME READY STATUS RESTARTS AGE client 1/1 Running 0 22s
As you can see from the above, Kubernetes executed the portStart script then started the contaier's main
ENTRYPOINT which is the nginx daemon.
postStart Script Methods
postStart script uses the following methods for running the checks:
- exec: Used in the preceding example, the exec method executes one or more arbitrary commands against the container. The exit status specifies whether or not the check has passed.
- httpGet: Opens an HTTP connection to a local port on the container. You can optionally supply a path. For example, if we can modify the preceding example to check whether or not port 8080 is open and the
/statusendpoint path returns a valid response:
apiVersion: v1 kind: Pod metadata: name: client spec: containers: - image: mynginx name: client lifecycle: postStart: httpGet: port: 8080 path: /status
Why Not Use an
init Container Instead?
init container is a Kubernetes feature that allows a container to start and do one or more tasks, then it gets terminated. The init container starts and stops before other containers do, making it the right candidate for performing any pre-launch tasks. However, with
postStart hooks and init containers appear to do similar jobs, the implementation is most different. Let's have a quick comparison between both methods and demonstrate possible use cases for each:
postStartscripts are executed using the same image as of the main container. Init containers can use the same or a different image than the one used by subsequent containers. So, if the tasks that you need to perfrom require a different base image, then you would better use init containers
postStartscripts are executed inside the same container. So, if the script you need to run is tightly coupled with the container, for example, you need to make configuration changes to the container itself, you should use
- All init containers must finish before the primary containers start. On the other hand,
postStartscripts are specific to each container. So, container A uses its
postStartscripts and launches, container B executes its
postStartscripts and starts and so on. So, if you are hosting more than one container on the same Pod and you need to run one or more container-agnostic tasks, you should use init containers. However, if each init scripts is specific to its container, then a
postStarthook is the suitable choice.
- Init containers start and stop before any other container starts.
postStartscripts get executed in parallel with their containers. This means that the script may or may not run the
ENTRYPOINTof the containers kicks in. If you need to be confident that pre-launch logic is always executed before the main container does, then use init continers.
- Because of how they are designed,
postStartscripts may run many times. The application logi should be able to deal with this numerous execution possibility. For example, if the
postStartscript adds a new temporary user account before the container runs, then it should first check whether the user has already been created so that it does not return an incorrect non-zero exit status.
Executing Commands Against the Container Before It Terminates Using the
Earlier in this post, we learned about the different signals that Kubernetes sends to the running containers inside the Pods when it wants to bring them down. However, although the container receives the
SIGTERM and it allows the container to shut down for up to thirty seconds, this may not be sufficient for complex scenarios.
Let's continue with out preceding example, and let's day that the RESTful API service that is running in parallel with out nginx service needs to perform several steps before it shuts down. The API designers were smart enough to expose an endpoint specifically for this purpose: executing the shutdown procedure. Kubernetes provide the
preStop hook, which gets called right before
SIGTERM signal is sent to the container. The
preStop hook also provides the same check methods as the
However, unlike the
portStart hook, if Kubernetes detects a non-zero exist status or a non-success HTTP code, it will continue the shutdown procedure and send the
The main difference between a traditional and a cloud-native application is that the latter does not run on an infrastructure that you own or under your control. Orchestration platforms like Kubernetes were designed to ensure that you get the highest level of application performance and availability given an unpredictable infrastructure. Accordingly, cloud-native applications should be written in a way that honors the contracts and constraints imposed by Kubernetes to enjoy the features it provides.