DevOps Infrastructure with K3s and Gitlab: A Simple yet Powerful Pair

Kubernetes has seen an incredible rise over the past years as organizations leverage containers for complex applications, micro-services and even cloud-native applications. And with the rise of Kubernetes, DevOps has gained more traction. While they may seem very different - as one is a tool and the other is a methodology - they work together to help deliver fast. Continuous integration and deployments (CI/CD) to Kubernetes is essential for productive and efficient platform development, and stable production releases. There are number of commercial CI/CD offerings, and most of them work well with Kubernetes. The open source application GitLab has a stable and mature CI/CD component, able to perform build operations and testing directly in a Kubernetes cluster. The following illustrates a GitLab runner building and testing code from within a development Kubernetes cluster.

IMAGE HERE (Figure 2-3)

K3s is a 40MB binary that runs a fully compliant production-grade Kubernetes distribution and requires only 512 MB of RAM. It is a gread way to wrap applications that you may not want to run in a full production cluster but would like to achieve greater uniformity in systems deployment, monitoring, and management across all development operations. Using K3s to host GitLab is great way to become familiar with single-node clusters and with the added benefit of a management plane unified under the Kubernetes API.

This blog post covers how to leverage Kubernetes in the management of the entire development lifecycle. To avoid the higher amount of cost a cloud-based production level cluster, it outlines the installation of GitLab on a single-node K3s cluster. This post does not detail how to set a virutal machine (VM) on cloud. Any cloud service can be utilized based on your preferences. GitLab on K3s requires at least two CPU cores and 4086MB of memory to run efficiently, supporting a small development team.

Host Domain and DNS

This post uses gitlab.ylcnky.com domain for the setup, so please choose your own domain name and replace it in the following steps whereever needed. Besides, add at least two DNS A records your the domain.

| Type  | Host                      | Answer                 | TTL |
|-------|---------------------------|------------------------|-----|
|  A    | *.gitlab.yourdomain.com   |  VM Public IP Adress   | 300 |
|  A    | gitlab.yourdomain.com     |  VM Public IP Adress   | 300 |

K3s Installation

Log in to your VM via ssh. The first thing to do is upgrading outdated packages (if any) to ensure the server is equipped with the ltest security, bug fixes and performance improvements.

$ ssh root@PUBLIC.IP.ADDRESS
$ apt update && apt upgrade -y

Download the K3s installer with curl and pipe it to the shell for execution. It will download the K3s with the lates stable release. Installation takes about 30 seconds.

$ curl -sfL https://get.k3s.io | sh -

To be sure installation went well, use kubectl to get the node.

In this blog, `kubectl` is aliased with `k` so we don't need to write longer `kubectl` command everytime.

$ alias k=kubectl
$ k get no

NAME                    STATUS   ROLES    AGE   VERSION
gitlab.ylcnky.com       Ready    master   45s   v1.18.9+k3s1

Cert Manager / Let's Encrypt

K3s is a fully functional Kubernetes cluster and comes with Traefik Ingress controller to handle HTTP requests. Cert Manager is a Kubernetes add-on to automate the management and issueance of TLS certificates from various sources. We will install and configure Cert Manager to use Let's Encrypt for generating free TLS certificates used to secure the GitLab instance over HTTPS. GitLab has Let's Encrypt capabilities. However, since we will run GitLab through K3s Ingree (with Traefik), we need to generate certificates and provide TLS from the cluster.

First create a namespace named as cert-manager. Then, get Cert Manager's configuration file (there are newer versions available, so increase version number according to your setup requirements)

$ k create namespace cert-manager
$ k apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml

The be sure the resources were generated and Cert Manager is running, check the pods in the cert-manager namespace

$ k get po -n cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-77d8f4d85f-8xhnn              1/1     Running   1          2m
cert-manager-cainjector-75f88c9f56-dkhls   1/1     Running   1          2m
cert-manager-webhook-56669d7fcb-rkz4x      1/1     Running   1          2m

Next, a ClusterIssuer that is configured to retrieve TLS certificates from Let's Encrypt is needed. Later, we will define a Certificate resource which utilizes this ClusterIssuer. Create a yaml file cluster-issuer.yml with the following configuration. Change YOURMEAIL@EMAIL.COM with your e-mail address. Finally apply the resource to cluster.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: YOURMEAIL@EMAIL.COM
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: traefik

$ k apply -f cluster-issuer.yml

Installation of GitLab

For the installation of GitLab, there are multiple resource are needed to be defined in K3s cluster including namespaces, TLS certificate, services, configmap and deployment.

Namespace `gitlab`

At the first step, we will define the gitlab namespace as it is needed in the cluster.

$ k create namespace gitlab

Certificate `gitlab-ylcnky`

Next, we need to define the Certificate resource describing the domains we defined above. cert-manager use this Certificate resources to generate a single TLS key pair and populate a Secret with it. Any Ingress with the domains listed in this Certificate can use the generated Secret as a valid TLS certificate. Create a certificate.yml file with the following configuration. Remember to change gitlab.ylcnky.com domain with your own domain name. Finally apply the Certificate resource to K3s.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: gitlab-ylcnky
  namespace: gitlab
spec:
  secretName: gitlab-ylcnky-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  commonName: gitlab.ylcnky.com
  dnsNames:
    - gitlab.ylcnky.com
    - reg.gitlab.ylcnky.com
  acme:
    config:
      - http01:
          ingressClass: traefik
        domains:
          - gitlab.ylcnky.com
          - reg.gitlab.ylcnky.com

$ k apply -f certificate.yml

To check whether the certificate issued successfully, you can check the status of it within the corresponding namespace.

$ k get certificate -n gitlab

NAME            READY   SECRET              AGE
gitlab-ylcnky   True    gitlab-ylcnky-tls   2m

Services `gitlab` and `gitlab-ssh`

Next, we will define two service resources named gitlab and gitlab-ssh. The gitlab service provides a backend service for Ingress by exposing port 80 for the GitLab website and port 5050 for container registry. These ports later connects the pods matching the label app: gitlab in the deployment resource. The gitlab-ssh exposes the NodePort 32222 used by the git ssh protocol for operations handled in git.

IMAGE 2-13 HERE

To define the services, create the services.yml file with the following configuration, and apply the resource to K3s

apiVersion: v1
kind: Service
metadata:
  name: gitlab
  namespace: gitlab
  labels:
    app: gitlab
spec:
  selector:
    app: gitlab
  ports:
    - name: http-web
      protocol: "TCP"
      port: 80
      targetPort: 80
    - name: http-reg
      protocol: "TCP"
      port: 5050
      targetPort: 5050
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: gitlab-ssh
  namespace: gitlab
  labels:
    app: gitlab-ssh
spec:
  selector:
    app: gitlab
  ports:
    - name: tcp-git
      protocol: "TCP"
      targetPort: 22
      port: 32222
      nodePort: 32222
  type: NodePort

$ k apply -f services.yml

ConfigMap `gitlab-config`

Next we will define the ConfigMap resource to manage the main GitLab config file gitlab.rb. There are an extensive list of configurations is available on GitLab documents. In this post, we will cover a minimal operating functionalities of Gitlab such as port numbers, URL, root password etc. Create a configmap.yml file with the following configuration (enrich the configs based on your needs) and apply the resource to K3s.

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-config
  namespace: gitlab
data:
  gitlab.rb: |-
    gitlab_rails['gitlab_shell_ssh_port'] = 32222
    prometheus['monitor_kubernetes'] = false
    gitlab_rails['initial_root_password'] = "YOUR_STRONG_PASSWORD"
    external_url 'https://gitlab.ylcnky.com'
    nginx['listen_port'] = 80
    nginx['listen_https'] = false
    nginx['proxy_set_headers'] = {
      'X-Forwarded-Proto' => 'https',
      'X-Forwarded-Ssl' => 'on'
    }
    registry_external_url 'https://reg.gitlab.ylcnky.com'
    gitlab_rails['registry_enabled'] = true
    registry_nginx['listen_port'] = 5050
    registry_nginx['listen_https'] = false
    registry_nginx['proxy_set_headers'] = {
      'X-Forwarded-Proto' => 'https',
      'X-Forwarded-Ssl' => 'on'
    }

$ k apply -f configmap.yml

Deployment `gitlab`

Next, we will create the deployment resource which defines the GitLab application as a single Pod, and because of that we can use hostPath to mount directories on the server, providing a persistent storage for the Pod. The directory /srv/gitlab is created automatically on the server. All configuration and data persist as files on the server, while gitlab-configmap-volume mounts the ConfigMap created above with the contents of gitlab.rb.

Create the deployment.yaml file with the following configuration and apply it to K3s.

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: gitlab
  name: gitlab
  labels:
    app: gitlab
spec:
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: gitlab
  template:
    metadata:
      labels:
        app: gitlab
    spec:
      containers:
        - name: gitlab
          image: gitlab/gitlab-ce:11.10.4-ce.0
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: config-volume
              mountPath: /etc/gitlab
            - name: logs-volume
              mountPath: /var/log/gitlab
            - name: data-volume
              mountPath: /var/opt/gitlab
            - name: reg-volume
              mountPath: /var/opt/gitlab/gitlab-rails/shared/registry
            - name: uploads-volume
              mountPath: /var/opt/gitlab/gitlab-rails/uploads
            - name: gitlab-configmap-volume
              mountPath: /etc/gitlab/gitlab.rb
              subPath: gitlab.rb
          ports:
            - name: http-web
              containerPort: 80
            - name: tcp-ssh
              containerPort: 22
            - name: http-reg
              containerPort: 5050
      volumes:
        - name: gitlab-configmap-volume
          configMap:
            name: gitlab-config
        - name: config-volume
          hostPath:
            path: /srv/gitlab/config
        - name: logs-volume
          hostPath:
            path: /srv/gitlab/logs
        - name: data-volume
          hostPath:
            path: /srv/gitlab/data
        - name: reg-volume
          hostPath:
            path: /srv/gitlab/reg
        - name: uploads-volume
          hostPath:
            path: /srv/gitlab/uploads

$ k apply -f deployment.yml

Ingress `gitlab`

K3s is preconfigured with a Traefik Ingress controller managing requests to HTTP port 80 and HTTPS port 443. A basic Ingress configuration is needed along with an added annotation requesting that Traefik route any HTTP request to HTTPS. Create an ingress.yml file with the following configuration and apply the resource to K3s.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gitlab
  namespace: gitlab
  labels:
    app: gitlab
  annotations:
    traefik.ingress.kubernetes.io/redirect-entry-point: https
spec:
  rules:
    - host: gitlab.ylcnky.com
      http:
        paths:
          - backend:
              serviceName: gitlab
              servicePort: 80
            path: /
    - host: reg.gitlab.ylcnky.com
      http:
        paths:
          - backend:
              serviceName: gitlab
              servicePort: 5050
            path: /
  tls:
    - hosts:
        - reg.gitlab.ylcnky.com
        - gitlab.ylcnky.com
      secretName: gitlab-ylcnky-tls

$ k apply -f ingress.yml

Go to Your GitLab

We deployed all resources the GitLab. To be sure, you can check the generated resource under the gitlab namespace.

$ k get all -n gitlab

NAME                          READY   STATUS    RESTARTS   AGE
pod/gitlab-6d68ffc7f9-c2rvq   1/1     Running   1          10m

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
service/gitlab       ClusterIP   xx.xx.xx.xx    <none>        80/TCP,5050/TCP   10m
service/gitlab-ssh   NodePort    xx.xx.xxx.xx   <none>        32222:32222/TCP   10m

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gitlab   1/1     1            1           10m

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/gitlab-6d68ffc7f9   1         1         1       10m

Now you can go to your domain to access your GitLab platform. Remember to use root as username and YOUR_STRONG_PASSWORD as password. ylcnky-gitlab

DevOps Infrastructure with K3s and Gitlab: A Simple yet Powerful Pair

IMAGE HERE (Figure 2-3)

Host Domain and DNS

K3s Installation

In this blog, kubectl is aliased with k so we don't need to write longer kubectl command everytime.

Cert Manager / Let's Encrypt

Installation of GitLab

Namespace gitlab

Certificate gitlab-ylcnky

Services gitlab and gitlab-ssh