Skip to content

GitLab CI template for Kubernetes (k8s)

This project implements a generic Gitlab CI template for Kubernetes environments.

Overview

This template implements continuous delivery/continuous deployment for projects hosted on Kubernetes platforms.

It provides several features, usable in different modes.

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master by default):

  • a staging environment (an iso-prod environment meant for testing and validation purpose),
  • the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

  • continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
  • continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Usage

Include

In order to include this template in your project, add the following to your gitlab-ci.yml:

include:
  - project: 'to-be-continuous/kubernetes'
    ref: '2.0.3'
    file: '/templates/gitlab-ci-k8s.yml'

Supported cluster access methods

The Kubernetes template supports 3 ways of login/accessing your Kubernetes cluster(s):

  1. The GitLab Kubernetes integration: when enabled, the template automatically retrieves and uses your Kubernetes cluster configuration (KUBECONFIG env),
  2. By defining an explicit kubeconfig yaml content from env,
  3. By defining explicit kubeconfig exploded parameters from env: server url, server certificate authority and user token.

Global configuration

The Kubernetes template uses some global configuration used throughout all jobs.

Name description default value
K8S_KUBECTL_IMAGE the Docker image used to run Kubernetes kubectl commands
⚠️ set the version required by your Kubernetes server
bitnami/kubectl:latest
K8S_BASE_APP_NAME default application name $CI_PROJECT_NAME (see GitLab doc)
πŸ”’ K8S_DEFAULT_KUBE_CONFIG default kubeconfig content used by kubectl. required if not using exploded kubeconfig parameters
K8S_URL the Kubernetes API url required if using exploded kubeconfig parameters
K8S_CA_CERT the default Kubernetes server certificate authority optional if using exploded kubeconfig parameters
πŸ”’ K8S_TOKEN default service account token required if using exploded kubeconfig parameters
K8S_SCRIPTS_DIR directory where k8s scripts (hook scripts) are located . (root project dir)
DOCKER_CONTAINER_STABLE_IMAGE Docker image name to use for staging/prod has to be defined when not chaining execution from Docker template
DOCKER_CONTAINER_UNSTABLE_IMAGE Docker image name to use for review has to be defined when not chaining execution from Docker template

Secrets management

Here are some advices about your secrets (variables marked with a πŸ”’):

  1. Manage them as project or group CI/CD variables:
    • masked to prevent them from being inadvertently displayed in your job logs,
    • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
  2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
  3. Don't forget to escape special characters (ex: $ -> $$).

Environments configuration

As seen above, the Kubernetes template may support up to 4 environments (review, integration, staging and production).

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

  • environment_type: set to the type of environment (review, integration, staging or production),
  • environment_name: the application name (see below),
  • environment_url: set to $CI_ENVIRONMENT_URL.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

Here are configuration details for each environment.

Review environments

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the K8S_REVIEW_SPACE variable (see below).

Here are variables supported to configure review environments:

Name description default value
K8S_REVIEW_SPACE k8s namespace for review env none (disabled)
K8S_REVIEW_APP_NAME application name for review env "${K8S_BASE_APP_NAME}-${CI_COMMIT_REF_SLUG}"
πŸ”’ K8S_REVIEW_KUBE_CONFIG kubeconfig content used by kubectl for review env (only define if not using exploded kubeconfig parameters and if different from default) $K8S_DEFAULT_KUBE_CONFIG
K8S_REVIEW_URL Kubernetes API url for review env (only define if using exploded kubeconfig parameters and if different from default) $K8S_URL
K8S_REVIEW_CA_CERT the Kubernetes server certificate authority for review env (only define if using exploded kubeconfig parameters and if different from default) $K8S_CA_CERT
πŸ”’ K8S_REVIEW_TOKEN service account token for review env (only define if using exploded kubeconfig parameters and if different from default) $K8S_TOKEN
K8S_REVIEW_ENVIRONMENT_SCHEME The review environment protocol scheme https
K8S_REVIEW_ENVIRONMENT_DOMAIN The review environment domain none

Note: By default review environment.url will be built as ${K8S_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${K8S_REVIEW_ENVIRONMENT_DOMAIN}

Integration environment

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the K8S_INTEG_SPACE variable (see below).

Here are variables supported to configure the integration environment:

Name description default value
K8S_INTEG_SPACE k8s namespace for integration env none (disabled)
K8S_INTEG_APP_NAME application name for integration env $K8S_BASE_APP_NAME-integration
πŸ”’ K8S_INTEG_KUBE_CONFIG kubeconfig content used by kubectl for integration env (only define if not using exploded kubeconfig parameters and if different from default) $K8S_DEFAULT_KUBE_CONFIG
K8S_INTEG_URL Kubernetes API url for integration env (only define if using exploded kubeconfig parameters and if different from default) $K8S_URL
K8S_INTEG_CA_CERT the Kubernetes server certificate authority for integration env (only define if using exploded kubeconfig parameters and if different from default) $K8S_CA_CERT
πŸ”’ K8S_INTEG_TOKEN service account token for integration env (only define if using exploded kubeconfig parameters and if different from default) $K8S_TOKEN
Β K8S_INTEG_ENVIRONMENT_URL The integration environment url including scheme (ex: https://my-application-integration.nonpublic.k8s.domain.com). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. none

Staging environment

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (master by default).

It is disabled by default and can be enabled by setting the K8S_STAGING_SPACE variable (see below).

Here are variables supported to configure the staging environment:

Name description default value
K8S_STAGING_SPACE k8s namespace for staging env none (disabled)
K8S_STAGING_APP_NAME application name for staging env $K8S_BASE_APP_NAME-staging
πŸ”’ K8S_STAGING_KUBE_CONFIG kubeconfig content used by kubectl for staging env (only define if not using exploded kubeconfig parameters and if different from default) $K8S_DEFAULT_KUBE_CONFIG
K8S_STAGING_URL Kubernetes API url for staging env (only define if using exploded kubeconfig parameters and if different from default) $K8S_URL
K8S_STAGING_CA_CERT the Kubernetes server certificate authority for staging env (only define if using exploded kubeconfig parameters and if different from default) $K8S_CA_CERT
πŸ”’ K8S_STAGING_TOKEN service account token for staging env (only define if using exploded kubeconfig parameters and if different from default) $K8S_TOKEN
Β K8S_STAGING_ENVIRONMENT_URL The staging environment url including scheme (ex: https://my-application-staging.nonpublic.k8s.domain.com). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. none

Production environment

The production environment is the final deployment environment associated with your production branch (master by default).

It is disabled by default and can be enabled by setting the K8S_PROD_SPACE variable (see below).

Here are variables supported to configure the production environment:

Name description default value
K8S_PROD_SPACE k8s namespace for production env none (disabled)
K8S_PROD_APP_NAME application name for production env $K8S_BASE_APP_NAME
πŸ”’ K8S_PROD_KUBE_CONFIG kubeconfig content used by kubectl for production env (only define if not using exploded kubeconfig parameters and if different from default) $K8S_DEFAULT_KUBE_CONFIG
K8S_PROD_URL Kubernetes API url for production env (only define if using exploded kubeconfig parameters and if different from default) $K8S_URL
K8S_PROD_CA_CERT the Kubernetes server certificate authority for production env (only define if using exploded kubeconfig parameters and if different from default) $K8S_CA_CERT
πŸ”’ K8S_PROD_TOKEN service account token for production env (only define if using exploded kubeconfig parameters and if different from default) $K8S_TOKEN
AUTODEPLOY_TO_PROD Set this variable to auto-deploy to production. If not set deployment to production will be manual (default behaviour). none (disabled)
K8S_PROD_ENVIRONMENT_URL Β The production environment url including scheme (ex: https://my-application.public.k8s.domain.com) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. none

Deployment jobs

The GitLab CI template for Kubernetes supports two policies for deploying your code:

  1. script-based deployment
  2. template-based deployment

1: script-based deployment

In this mode, you only have to provide a shell script that fully implements the deployment using the kubectl CLI.

The deployment script is searched as follows:

  1. look for a specific k8s-deploy-$env.sh in the $K8S_SCRIPTS_DIR directory in your project (e.g. k8s-deploy-staging.sh for staging environment),
  2. if not found: look for a default k8s-deploy.sh in the $K8S_SCRIPTS_DIR directory in your project,
  3. if not found: the GitLab CI template assumes you're using the template-based deployment policy.

Your script(s) may use dynamic variables.

2: template-based deployment

In this mode, you have to provide a Kubernetes deployment file in your project structure, and let the GitLab CI template kubectl apply it.

The template processes the following steps:

  1. optionally executes the k8s-pre-apply.sh script in your project to perform specific environment pre-initialization (for e.g. create required services),
  2. looks for your Kubernetes deployment file, substitutes dynamic variables and kubectl apply it,
    1. look for a specific deployment-$env.yml in your project (e.g. deployment-staging.yml for staging environment),
    2. fallbacks to default deployment.yml.
  3. optionally executes the k8s-post-apply.sh script in your project to perform specific environment post-initialization stuff,
  4. optionally executes the k8s-readiness-check to wait & check for the application to be ready (if not found, the template assumes the application was successfully started).

All scripts and Kubernetes deployment files may use dynamic variables.

Cleanup jobs

The GitLab CI template for Kubernetes supports two policies for destroying an environment (actually only review environments):

  1. script-based cleanup
  2. template-based cleanup

1: script-based cleanup

In this mode, you only have to provide a shell script that fully implements the environment cleanup using the kubectl CLI.

The a deployment script is searched as follows:

  1. look for a specific k8s-cleanup-$env.sh in the $K8S_SCRIPTS_DIR directory in your project (e.g. k8s-cleanup-staging.sh for staging environment),
  2. if not found: look for a default k8s-cleanup.sh in the $K8S_SCRIPTS_DIR directory in your project,
  3. if not found: the GitLab CI template assumes you're using the template-based cleanup policy.

Your script(s) may use dynamic variables.

TIP: a nice way to implement environment cleanup is to declare the label app=${appname} on every Kubernetes object associated to your environment. Then environment cleanup can be implemented very easily with command kubectl delete all,pvc,secret,ingress -l "app=${appname}"

2: template-based cleanup

In this mode, you mainly let Kubernetes delete all objects from your Kubernetes deployment file.

The template processes the following steps:

  1. optionally executes the k8s-pre-cleanup.sh script in your project to perform specific environment pre-cleanup stuff,
  2. looks for your Kubernetes deployment file, substitutes dynamic variables and kubectl delete it,
    1. look for a specific deployment-$env.yml in your project (e.g. deployment-staging.yml for staging environment),
    2. fallbacks to default deployment.yml.
  3. optionally executes the k8s-post-cleanup.sh script in your project to perform specific environment post-cleanup (for e.g. delete bound services).

All scripts and Kubernetes deployment files may use dynamic variables.

Cleanup job limitations

When using this template, you have to be aware of one limitation (bug) with the cleanup job.

By default, the cleanup job triggered automatically on branch deletion will fail due to not being able to fetch the Git branch prior to executing the job (obviously because the branch was just deleted). This is pretty annoying, but as you may see above, deleting an env may require scripts from the project...

In a future version, the template will rely on Kustomize and will be able to delete an entire environment using the app label. In the meantime we're just sorry about this bug, but for now there is not much we can do:

  • remind to delete your review env manually before deleting the branch
  • otherwise you'll have to do it afterwards from your computer (using kubectl CLI) or from the some k8s web console.

Dynamic Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments (review, integration, staging and production), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some genericity in your scripts and templates, you should use available environment variables:

  1. any GitLab CI variable (ex: ${CI_ENVIRONMENT_URL} to retrieve the actual environment exposed route)
  2. any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)
  3. dynamic variables set by the template:
    • ${appname}: the application target name to use in this environment (ex: myproject-review-fix-bug-12 or myproject-staging)
    • ${appname_ssc}: the application target name in SCREAMING_SNAKE_CASE format (ex: MYPROJECT_REVIEW_FIX_BUG_12 or MYPROJECT_STAGING)
    • ${env}: the environment type (review, integration, staging or production)
    • ${hostname}: the environment hostname, extracted from ${CI_ENVIRONMENT_URL} (has to be explicitly declared as environment:url in your .gitlab-ci.yml file)
    • any GitLab CI variable (ex: ${CI_ENVIRONMENT_URL} to retrieve the actual environment exposed route)

⚠️

In order to be properly replaced, variables in your YAML descriptors shall be written with curly braces (ex: ${MYVAR} and not $MYVAR).

Multiline variables must be surrounded by double quotes ("). Example:

[...]
containers:
- name: restaurant-app
  env:
  # multiline variable
  - name: MENU
    value: "${APP_MENU}"

kube-score job

The Kubernetes template enables kube-score analysis of your Kubernetes object definitions. This job is mapped to the test stage and is active by default.

Here are its parameters:

Name description default value
K8S_KUBE_SCORE_IMAGE Docker image to run kube-score zegl/kube-score:latest-helm it is recommended to set a tool version compatible with your Kubernetes cluster
K8S_SCORE_DISABLED Set to trueto disable the kube-score analysis none (enabled)
K8S_SCORE_EXTRA_OPTS Additional options to kube-score command line none

Variants

Vault variant

This variant allows delegating your secrets management to a Vault server.

Configuration

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

Name description default value
VAULT_BASE_URL The Vault server base API url none
πŸ”’ VAULT_ROLE_ID The AppRole RoleID must be defined
πŸ”’ VAULT_SECRET_ID The AppRole SecretID must be defined

Usage

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Name description
secret_path (path parameter) this is your secret location in the Vault server
field (query parameter) parameter to access a single basic field from the secret JSON payload

Example

include:
  # main template
  - project: 'to-be-continuous/kubernetes'
    ref: '2.0.3'
    file: '/templates/gitlab-ci-k8s.yml'
  # Vault variant
  - project: 'to-be-continuous/kubernetes'
    ref: '2.0.3'
    file: '/templates/gitlab-ci-k8s-vault.yml'

variables:
    # Secrets managed by Vault
    K8S_DEFAULT_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/kubernetes/noprod?field=kube_config"
    K8S_PROD_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/kubernetes/prod?field=kube_config"
    VAULT_BASE_URL: "https://vault.acme.host/v1"
    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

GitLab compatibility

ℹ️ This template is actually tested and validated on GitLab Community Edition instance version 13.12.11