Kubernetes Patterns: Declarative Deployment Types

Deployments are higher level controllers that use ReplicaSets to ensure that no downtime is incurred when you need to update your Pods. Deployments help you to orchestrate your upgrades in different ways for best reliability and availability of your application.

In this article, we investigate how you can do zero-downtime releases using Deployments. We will also dig into more advanced deployment scenarios such Blue/Green and Canary workflows that give you more control on your release process.

First things first: Let's Prepare Our Images.

We will use a custom web server image. We use Nginx as the base image. To test different versions of the image, we create three different images. The only difference is the text displayed in the default page the web server displays. We create three Dockerfile as follows:

Dockerfile_1

FROM nginx:latest
COPY v1.html /usr/share/nginx/html/index.html

Dockerfile_2

FROM nginx:latest
COPY v2.html /usr/share/nginx/html/index.html

Dockerfile_2

FROM nginx:latest
COPY v3.html /usr/share/nginx/html/index.html

And the HTML files are as follows:

v1.html

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta http-equiv="X-UA-Compatible" content="ie=edge">
	<title>Release 1</title>
</head>
<body>
	<h1>This is release #1 of the application</h1>
</body>
</html>

v2.html

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta http-equiv="X-UA-Compatible" content="ie=edge">
	<title>Release 2</title>
</head>
<body>
	<h1>This is release #2 of the application</h1>
</body>
</html>

v3.html

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta http-equiv="X-UA-Compatible" content="ie=edge">
	<title>Release 3</title>
</head>
<body>
	<h1>This is release #3 of the application</h1>
</body>
</html>

Finally we will push the images. You can use your own DockerHub account for this.

$ docker build -t ylcnky/mywebserver:1 -f Dockerfile_1 .
$ docker build -t ylcnky/mywebserver:2 -f Dockerfile_2 .
$ docker build -t ylcnky/mywebserver:3 -f Dockerfile_3 .
$ docker push ylcnky/mywebserver:1
$ docker push ylcnky/mywebserver:2
$ docker push ylcnky/mywebserver:3

Zero Downtime with Rolling Updates

Let's create a Deployment for running v1 for our webserver. Following yml file defines the services and deployment objects.

---
apiVersion: v1
kind: Service
metadata:
  name: mywebservice
spec:
  selector:
	app: nginx
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 80
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebserver
spec:
  replicas: 4
  strategy:
	type: RollingUpdate
	rollingUpdate:
  	maxSurge: 1
  	maxUnavailable: 1
  selector:
	matchLabels:
  	app: nginx
  template:
	metadata:
  	labels:
    	app: nginx
	spec:
  	containers:
  	- image: ylcnky/mywebserver:1
    	name: nginx
    	readinessProbe:
      	httpGet:
        	path: /
        	port: 80
        	httpHeaders:
        	- name: Host
          	value: K8sProbe

The above yml file defines two entities: (1) The service that enables external access to the Pods, and (2) the deployment controller. We are using here the RollingUpdate strategy, which is the default update strategy if you don't explicitly define a Deployment parameter. This deployment uses the Pod template to schedule 4 Pods, each hosting one container running ylcnky/mywebserver:1 image instance. Let's deploy the Service and the Deployment to see that in action.

$ kubectl apply -f nginx_deployment.yml

Above command will craete 4 Pods in the Running state.

$ kubectl get pods
NAME                       	READY   STATUS	RESTARTS   AGE
mywebserver-68cd66868f-78jgt   1/1 	Running   0      	5m29s
mywebserver-68cd66868f-kdxx9   1/1 	Running   0      	29m
mywebserver-68cd66868f-lh6wz   1/1 	Running   0      	29m
mywebserver-68cd66868f-vvqrh   1/1 	Running   0      	5m29s

If we want to actually see the content of the web page nginx is serving, we need to know the port that the Service is listening at:

$ kubectl get svc
NAME       	TYPE    	CLUSTER-IP 	EXTERNAL-IP   PORT(S)    	AGE
kubernetes 	ClusterIP   10.96.0.1  	    	443/TCP    	2d10h
mywebservice   NodePort	10.107.1.198       	80:32288/TCP   32m

Our mywebservice Service is using the port 32288 to route traffic to the Pod on port 89. If you navigate http://node_ip:32288 you should see something similar to the following:

image2

Upgrading Your Deployments

There is more than one way to update a running Deployment, one of them is modifying the definition file to reflect the new changes and applying it using kubectl. Change the .spec.template.spec.container[].image in the definition file to look as follows

spec:
  	containers:
  	- image: ylcnky/mywebserver:2

Clearly, the only thing that changed is the image tag: we need to deploy the second version of our application. Apply the file using kubectl

$ kubectl apply -f nginx_deployment.yaml
service/mywebservice unchanged
deployment.apps/mywebserver configured

Testing the Deployment Strategy

Now, quickly run kubectl get pods to see what the Deployment is doing to the Pods it manages:

$ kubectl get pods
NAME                       	READY   STATUS          	RESTARTS   AGE
mywebserver-68cd66868f-7w4fc   1/1 	Terminating     	0      	83s
mywebserver-68cd66868f-dwknx   1/1 	Running         	0      	94s
mywebserver-68cd66868f-mv9dg   1/1 	Terminating     	0      	94s
mywebserver-68cd66868f-rpr5f   0/1 	Terminating     	0      	84s
mywebserver-77d979dbfb-qt58n   1/1 	Running         	0      	4s
mywebserver-77d979dbfb-sb9s5   1/1 	Running         	0      	4s
mywebserver-77d979dbfb-wxqfj   0/1 	ContainerCreating   0      	0s
mywebserver-77d979dbfb-ztpc8   0/1 	ContainerCreating   0      	0s
$ kubectl get pods
NAME                       	READY   STATUS    	RESTARTS   AGE
mywebserver-68cd66868f-dwknx   0/1 	Terminating   0      	100s
mywebserver-77d979dbfb-qt58n   1/1 	Running   	0      	10s
mywebserver-77d979dbfb-sb9s5   1/1 	Running   	0      	10s
mywebserver-77d979dbfb-wxqfj   1/1 	Running   	0      	6s
mywebserver-77d979dbfb-ztpc8   0/1 	Running   	0      	6s
$ kubectl get pods
NAME                       	READY   STATUS	RESTARTS   AGE
mywebserver-77d979dbfb-qt58n   1/1 	Running   0      	25s
mywebserver-77d979dbfb-sb9s5   1/1 	Running   0      	25s
mywebserver-77d979dbfb-wxqfj   1/1 	Running   0      	21s
mywebserver-77d979dbfb-ztpc8   1/1 	Running   0      	21s

Running the command quickly several times shows us how the Deployment is creating new Pods that use the new Pod template while at the same time terminating the old Pods. However, it is evident that at no point we have zero running Pods. The Deployment is ensuring that replacing old Pods with new ones gradually while always keeping a portion of the running. This portion can be contolled by changing maxSurge and maxUnavailable parameters.

maxSurge: the number of Pods that can be deployed temporarily in addition to the new replicas. Setting this to 1 means that we can have a maximum total of five running Pods during the update process (the 4 replicas + 1) maxUnavailable: the number of Pods that can be killed simulatenously during the update process. In our example, we can have at least three Pods running while the update is in progress (4 replicas - 1) image While the update is in progress, you can try refreshing the web page several times, you will notice the webserver is always responsive. You don't get encountered an unanswered HTTP request. When you navigate the page, you will see like the following: image

A note for high-traffic environments

If we are in a high-traffic environment, the service may not be instantly aware that a Pod was down so that it does not route traffic to it. This causes some connection to drop while the update is in progress. To address this potential problem, we can add a lifecycle step that pauses the thread for a few seconds to ensure that no connections get dropped.

lifecycle:
  preStop:
    exec:
      command: ["/bin/bash","-c","sleep 20"]

Fixed Deployment Strategy

Mos of the time, you need to have zero downtime when deploying a new version of your software. However, is some cases, you require to deny access to the old application version totally, even if that entails displaying an error message or an under mainteanance message to your clients for a brieg period. The most common use case for such a scenario is when you have a severe bug or security vulnerability that was not patched yet. When you encounter that sort of situation, the rolling pdate strategy may work against you. Client will still use the old, vulnerable application version while the patched version deployment is in progress. Compromising your users' security poses a much more significant threat than giving them a come back later message.

In this case, the best option is to use a Fixed Deployment Strategy. In this deployment workflow, Deployment will not gradually replace the Pods. Instead, it kills all the Pods running the old application version; then it recreates them using the new Pod template. To use this strategy, you set the strategy tyoe to recrate. image

Deployment using the Recreate Strategy

Assuming that version 2 of our web server contained severe security bugs, now we need to make necesarry patching and deploy the new version. Change the deployment part of the definition file to look as follows:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebserver
spec:
  replicas: 4
  strategy:
	type: Recreate
  selector:
	matchLabels:
  	app: nginx
  template:
	metadata:
  	labels:
    	app: nginx
	spec:
  	containers:
  	- image: ylcnky/mywebserver:3
    	name: nginx
    	readinessProbe:
      	httpGet:
        	path: /
        	port: 80
        	httpHeaders:
        	- name: Host
          	value: K8sProbe

Now let's apply the new definiition and immediately check the status of the Pods to view what's happening:

$ kubectl apply -f nginx_deployment.yaml
service/mywebservice unchanged
deployment.apps/mywebserver configured
$ kubectl get pods
NAME                       	READY   STATUS    	RESTARTS   AGE
mywebserver-77d979dbfb-ztpc8   0/1 	Terminating   0      	60m
$ kubectl get pods
NAME                       	READY   STATUS          	RESTARTS   AGE
mywebserver-5f6bbd8587-4qpzt   0/1 	ContainerCreating   0      	4s
mywebserver-5f6bbd8587-76c4v   0/1 	ContainerCreating   0      	4s
mywebserver-5f6bbd8587-s6v64   0/1 	ContainerCreating   0      	4s
mywebserver-5f6bbd8587-tt6nc   0/1 	ContainerCreating   0      	4s
$ kubectl get pods
NAME                       	READY   STATUS	RESTARTS   AGE
mywebserver-5f6bbd8587-4qpzt   0/1 	Running   0      	15s
mywebserver-5f6bbd8587-76c4v   0/1 	Running   0      	15s
mywebserver-5f6bbd8587-s6v64   0/1 	Running   0      	15s
mywebserver-5f6bbd8587-tt6nc   0/1 	Running   0      	15s

As you can see, the Deployment started by terminating all the running Pods, then it created the new ones all at ones until the desired number of Pod replicas was reached. If you tried refreshing the web page during the deployment, you might have found that the web page was unresponsive. Perhaps you received an Website unreachable or a similar message depending on the browser you are using. Again, this is the desired behavior as we don't need someone using our v2 of the hypothetical app until v3 is deployed. The web page should like following when all the pod are generated: image

Blue/Green Release Strategy

Also referred to as A/B deployment, the Blue/Green deployment involves having two sets of identical hardware. The software application is deployed to both environment at the same time. However, only one of the environments receives a live traffic while the other remains idle. When a new version of the application is ready, it gets deployed to the blue environment. The network is directed to the blue environment through a router or a similar mechanism. If problems are detected on the new release and a rollback is needed, the only action that should be done is redirecting back to the green environment.

In the next software release iteration, the new code gets deployed to the green environment. The blue environment now can be used as a staging environment or for disaster recovery purposes. The advantages of this strategy are that -unlike rolling update- there is zero downtime during deployment process, although there is never more than one version of the application running at the same time. The drawback, however, is that you need to double the resources hosting the application, which may increase your costs. image

Zero Downtime and No Concurrent Versions with Blue/Green Deployments

Let's have a quick revisit to the Recreate strategy that was used in an earlier example. While we were able to protect our clients from our application version that was compromised, we still incurred a downtime. In mission-critical environments, this is not acceptable. However, using the Deployment and Service resource controllers, we can quickly achieve both targets. The idea is tho create a second Deployment with the new application version (blue) while the original one is still running (green). Once all the Pods in the Blue deployment are ready, we instruct the Service to switch to the Blue deployment (by changing the Pod Selector appropriately). If a rollback is required, we shift the selector back to the green Pods. Let's see how this can be done in our example.

First, let's destroy the current Deployment:

$ kubectl delete deployment mywebserver

First, let's split our definition file so that each: the Service and the Deployment lives in its file. So, you may have a file called nginx_deployment.yml and nginx_service.yml. Copy nginx_deployment.yml to nginx_deployment_blue.yml and rename the source file to be nginx_deployment_green.yml So, to wrap up, you should have the following three files in your directory:

  • nginx_deployment_green.yml
  • nginx_deployment_blue.yml
  • nginx_service.yml

So far, the first two files are the same. Let's change nginx_deployment_green.yml to look as follows

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebserver-green
spec:
  replicas: 4
  strategy:
	type: Recreate
  selector:
	matchLabels:
  	app: nginx_green
  template:
	metadata:
  	labels:
    	app: nginx_green
	spec:
  	containers:
  	- image: ylcnky/mywebserver:1
    	name: nginx

The deployment is using v1 of our application We also appended "-gree" to the Pod tags and their selectors denoting that this is the green deployment. Additionally, the deployment name is mywebserver_green indicating that is the green deployment. Notice that we are using the Recreate deployment strategy. Using Recreate or RollingUpdate is of no significance here as we are not relying on the Deployment controller to perform the update.

Although we are doing this exercise manually since the only supported deployment strategies -currently- are RollingUpdate and Recreate, it is worth noting that there are tools that automate this process like Istio and Knative. Let's now change the nginx_deployment_blue.yml to look as follows

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebserver-blue
spec:
  replicas: 4
  strategy:
	type: Recreate
  selector:
	matchLabels:
  	app: nginx_blue
  template:
	metadata:
  	labels:
    	app: nginx_blue
	spec:
  	containers:
  	- image: ylcnky/mywebserver:2
    	name: nginx

The changes are similar to what we did the green deployment file except that we are labelling this one as the blue version. Our Service should be defined as follows:

---
apiVersion: v1
kind: Service
metadata:
  name: mywebservice
spec:
  selector:
	app: nginx_blue
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 80
  type: NodePort

The only difference between this Service definition file and the one used earlier is that we changed the Pod Selector to match the blue Pods that are part of our blue deployment.

Testing the Blue/Green Deployment

Let's create te blue and green deployments and the service

$ kubectl apply -f nginx_deployment_blue.yaml
deployment.apps/mywebserver-blue created

$ kubectl apply -f nginx_deployment_green.yaml
deployment.apps/mywebserver-green created

$ kubectl apply -f nginx_service.yaml
service/mywebservice configured

Navigating to http://node_ip:32288 shows that we are using version 2 of our application. If we need to quickly rollback to version 1, we just change the Service definition in nginx_service.yml to look as follows:

---
apiVersion: v1
kind: Service
metadata:
  name: mywebservice
spec:
  selector:
	app: nginx_green
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 80
  type: NodePort

Now refreshing the web page shows that we have reverted to version 1 of our application. there was no downtime during this process. Additionally, we had only one version of our application running at any particular point in time.

Canary Deployment Strategy

Canary Deployment is a popular release strategy that focuses more on testing the air before going with the full deployment.

The name of Canary Deployment Strategy has its origins rooted back to coal miners. When a new mine is discovered, workers used to carry a cage with some Canary birds. The placed the cage at the mine entrance. If the birts die, that was an indication of toxic Carbon Monoxide gas emission

While the implementation is different (way to go, Cararies!), the concept remains the same. When software is released using a Canary deployment, a small subset of the incoming traffic is directed to the new application version while the majority remains routed to the old, stable version. The main advantage of this method is that you get customer feedback quickly on any new feature your application offers. If things go wrong, you can easily route all the traffic to the stable version. If enough positive feedbacks is received, you can gradually increase the portion of traffic going to the new version until it reaches 100%. image

Canary Testing using Kubernetes Deployment and Services

Assuming that we are currently running version 1 of our application, and we need to deploy version 2. We want to test some metrics like latency, CPU consumption under different load levels. We are also collecting feedback from the users. If everything looks good, we do a full deployment. We are doing this old-fashioned way for demostration purposes, yet some tools such as Istio can automate this process.

The first thing we need to do is to create 2 Deployment definition files; one of them uses version 1 of the image, and the other one uses version 2. Both Deployments use the same Pod labels and selectors. The files should look something like following: nginx_deployment_stable.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebserver-stable
spec:
  replicas: 6
  strategy:
	type: Recreate
  selector:
	matchLabels:
  	app: nginx
  template:
	metadata:
  	labels:
    	app: nginx
	spec:
  	containers:
  	- image: ylcnky/mywebserver:1
    	name: nginx

nginx_deployment_canary.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebserver-canary
spec:
  replicas: 2
  strategy:
	type: Recreate
  selector:
	matchLabels:
  	app: nginx
  template:
	metadata:
  	labels:
    	app: nginx
	spec:
  	containers:
  	- image: ylcnky/mywebserver:2
    	name: nginx

Both files look identical except for the Deployment name and Pod image. Notice that we set the number of replicas on the stable deployment to 6 while we are deploying only 2 Pods on the Canary one. This is intentional, we need 25% only of our application to serve version 2 while the remainin 75% continues to serve version 1. Unlike the Blue/Green strategy, we don't make the changes to the Service definition

apiVersion: v1
kind: Service
metadata:
  name: mywebservice
spec:
  selector:
	app: nginx
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 80
  type: NodePort

The Service routes traffic to any Pod labelled app=nginx. The Deployment controls which Pods get more traffic by increasing/decreasing the number of replicas.

Testing the Canary Deployment

Let's apply our Deployments and Service

$ kubectl apply -f nginx_deployment_stable.yaml
deployment.apps/mywebserver-stable created
$ kubectl apply -f nginx_deployment_canary.yaml
deployment.apps/mywebserver-canary created
$ kubectl apply -f nginx_service.yaml
service/mywebservice configured

If you refresh the web page http://node_port:32288 several times, you may occasionally see version 2 of the application shows.

Increasing the percentage of users going to version 2 is a simple as increasing the replicas count of the Canary deployment and decreasing the replicas count of the stable one. If you need to rollback, you just set the number of replicas to be 8 (100%) on the stable Deployment and deleting Canary Deployment. Alternatively, you can go ahead with the Canary Deployment be reversing this operation

Conclusion:

In this post, we tried to cover the most applied or referred Deploument strategies. Indeed, there are more than what we presented here. I strongly suggest this Github page to see other deployment types. I find the following image (taken from link) quite useful for a comparison image

blog

copyright©2021 ylcnky all rights reserved