Skip to content

Using To Be Continuous

Include a template

As previously said, each template may be used by including it to your .gitlab-ci.yml file.

For example:

include:
  - project: 'to-be-continuous/maven' # this is the template project
    file: '/templates/gitlab-ci-maven.yml' # template file within the project
    ref: '1.0.0' # template version

Our templates are versioned:

  • each version is exposed through a Git tag such as 0.1.0, 1.1.2, ...
  • our recommendation is to use a fixed version of each template, and upgrade when a new valuable feature is rolled out.
  • you may also chose to use the latest released version (discouraged as a new version with breaking changes would break your pipeline). For this, simply include the template from master.

Configure a template

Each template comes with a predefined configuration (whenever possible), but is always overridable via variables.

Some template features are also enabled by defining the right variable(s).

Here is an example of a Maven project that:

  1. overrides the Maven version used (with MAVEN_IMAGE variable),
  2. overrides the build arguments (with MAVEN_BUILD_ARGS),
  3. enables SonarQube analysis (by defining SONAR_URL and SONAR_AUTH_TOKEN),
include:
  - project: 'to-be-continuous/maven'
    file: '/templates/gitlab-ci-maven.yml'
    ref: '1.0.0'

variables:
  # use Maven 3.6 with JDK 8
  MAVEN_IMAGE: "maven:3.6-jdk-8"
  # use 'cicd' Maven profile
  MAVEN_BUILD_ARGS: 'verify -Pcicd'
  # enable SonarQube analysis
  SONAR_URL: "https://mysonar.domain.my"
  # SONAR_AUTH_TOKEN defined as a secret CI/CD variable

This is the basic pattern for configuring the templates!

You'll find configuration details in each template reference documentation.

Secrets managements

Most of our templates manage 🔒 secrets (access tokens, user/passwords, ...).

Our general recommendation for those secrets is to manage them as project or group CI/CD variables:

  • masked to prevent them from being inadvertently displayed in your job logs,
  • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).

What if a secret can't be masked?

It may happen that a secret contains characters that prevent it from being masked.

In that case there is a simple solution: simply encode it in Base64 and declare the variable value as the Base64 string prefixed with @b64@. This value can be masked, and it will be automatically decoded by our templates (make sure you're using a version of the template that supports this syntax).

Example

CAVE_PASSPHRASE={"open":"$€5@me"} can't be masked, but the Base64 encoded secret can.

Then just declare instead:

CAVE_PASSPHRASE=@b64@eyJvcGVuIjoiJOKCrDVAbWUifQ==

Scoped variables

All our templates support a generic and powerful way of limiting/overriding some of your environment variables, depending on the execution context.

This feature is comparable to GitLab Scoping environments with specs feature, but covers a broader usage:

  • can be used with non-secret variables (defined in your .gitlab-ci.yml file),
  • variables can be scoped by any other criteria than deployment environment.

The feature is based on a specific variable naming syntax:

# syntax 1: using a unary test operator
scoped__<target var>__<condition>__<cond var>__<unary op>=<target val>

# syntax 2: using a comparison operator
scoped__<target var>__<condition>__<cond var>__<cmp op>__<cmp val>=<target val>

⚠️ mind the double underscore that separates each part.

Where:

Name Description Possible values / examples
<target var> Scoped variable name any
example: MY_SECRET, MAVEN_BUILD_ARGS, ...
<condition> The test condition one of: if or ifnot
<cond var> The variable on which relies the condition any
example: CI_ENVIRONMENT_NAME, CI_COMMIT_REF_NAME, ...
<unary op> Unary test operator to use only: defined
<cmp op> Comparison operator to use one of: equals, startswith, endswith, contains, in
or their ignore case version: equals_ic, startswith_ic, endswith_ic,contains_ic or in_ic
<cmp val> Sluggified value to compare <cond var> against any
With in or in_ic operators, matching values shall be separated with double underscores
<target val> The value <target var> takes when condition matches any (can even use other variables that will be expanded)

Which variables support this?

The scoped variables feature has a strong limitation: it may only be used for variables used in the script and/or before_script parts; not elsewhere in the .gitlab-ci.yml file.

🔴 They don't support scoped variables:

  • variables used to parameterize the jobs Docker image(s) (ex: MAVEN_IMAGE or K8S_KUBECTL_IMAGE),
  • variables that enable/disable some jobs behavior (ex: MAVEN_DEPLOY_ENABLED, NODE_AUDIT_DISABLED or AUTODEPLOY_TO_PROD).

✅ They do support scoped variables:

  • credentials (logins, passwords, tokens, ...),
  • configuration URLs,
  • tool CLI options and arguments (ex: MAVEN_BUILD_ARGS or PHP_CODESNIFFER_ARGS)

If you have any doubt: have a look at the template implementation.

How variable values are sluggified?

Each character that is not a letter, a digit or underscore is replaced by an underscore (_).

Examples:

  • Wh@t*tH€!h3¢k becomes: Wh_t_tH__h3_k
  • feat/add-welcome-page becomes: feat_add_welcome_page

Example 1: scope by environment

variables:
  # default configuration
  K8S_URL: "https://my-nonprod-k8s.domain"
  MY_DATABASE_PASSWORD: "admin"

  # overridden for prodution environment
  scoped__K8S_URL__if__CI_ENVIRONMENT_NAME__equals__production: "https://my-prod-k8s.domain"
  # MY_DATABASE_PASSWORD is overridden for prod in my project CI/CD variables using
  # scoped__MY_DATABASE_PASSWORD__if__CI_ENVIRONMENT_NAME__equals__production

Example 2: scope by branch

variables:
  # default Angular build arguments (default configuration)
  NG_BUILD_ARGS: "build"

  # use 'staging' configuration on develop branch
  scoped__NG_BUILD_ARGS__if__CI_COMMIT_REF_NAME__equals__develop: "build --configuration=staging"

  # use 'production' configuration and optimization on master branch
  scoped__NG_BUILD_ARGS__if__CI_COMMIT_REF_NAME__equals__master: "build --configuration=production --optimization=true"

Example 3: scope on tag

variables:
  # default Docker build configuration
  DOCKER_BUILD_ARGS: "--build-arg IMAGE_TYPE=snapshot"

  # overridden when building image on tag (release)
  scoped__DOCKER_BUILD_ARGS__if__CI_COMMIT_TAG__defined: "--build-arg IMAGE_TYPE=release"

Proxy configuration

Our templates don't have any proxy configuration set by default, but they all support standard Linux variables:

  • http_proxy
  • https_proxy
  • ftp_proxy
  • no_proxy

As a result, you may perfectly define those variables in your project:

  • either globally as group or project variables or in the top variables block definition of your .gitlab-ci.yml file,
  • either locally in specific jobs,
  • or for all jobs from one single template (see below).

Certificate Authority configuration

Our templates all come configured with the Default Trusted Certificate Authorities, but they all support the CUSTOM_CA_CERTS variable to configure additional certificate authorities.

When set, this variable shall contain one or several certificates in PEM format, then the template will assume those are trusted certificates, and add them accordingly to the right trust store.

Again, you may perfectly set CUSTOM_CA_CERTS in your project:

  • either globally as group or project variables or in the top variables block definition of your .gitlab-ci.yml file,
  • either locally in specific jobs,
  • or for all jobs from one single template (see below).

Advanced usage - Override YAML

Sometimes, configuration via variables is not enough to tweak an existing template to fit to your needs.

Fortunately, GitLab CI include feature is implemented in a way that allows you to override the included YAML code.

from GitLab documentation

The files defined in include are:

  • Deep merged with those in .gitlab-ci.yml.
  • Always evaluated first and merged with the content of .gitlab-ci.yml, regardless of the position of the include keyword.

In order to override the included templates YAML code, you'll probably have to deep dive into it and understand how it is designed.

The templates base job

A very important thing you should be aware of is that every template defines a (hidden) base job, extended by all other jobs. That might not be the case for templates that declare one single job.

For example the Maven template defines the .mvn-base base job.

Thus, if you wish to override something for all the jobs from a specific template, this is the right place to do the magic.

Example 1: add Docker services

In this example, let's consider my Java project needs a MySQL database to run its unit tests.

According to the Maven template implementation, that can be done by overriding the mvn-build job as follows:

mvn-build:
  services:
    - name: mysql:latest
      alias: mysql_host
  variables:
    MYSQL_DATABASE: "acme"
    MYSQL_ROOT_PASSWORD: "root"

Those changes will gracefully be merged with the mvn-build job, the rest of it (defined by the Maven template) will remain unchanged.

Example 2: run on private runners with proxy

In this example, let's consider my project needs to deploy on a Kubernetes cluster that is only accessible from my private runner (with tags kubernetes, private), and that requires an http proxy.

According to the Kubernetes template implementation, that can be done by overriding the base .k8s-base job as follows:

.k8s-base:
  # set my runner tags
  tags:
    - kubernetes
    - private
  # set my proxy configuration
  variables:
    http_proxy: "http://my.proxy:8080"
    https_proxy: "http://my.proxy:8080"

This way, all Kubernetes jobs will inherit this configuration.