Skip to content

GitLab CI template for Terraform

This project implements a generic GitLab CI template for managing infrastructure with Terraform.

Overview

This template implements continuous delivery/continuous deployment for projects hosted on Terraform platforms.

It provides several features, usable in different modes.

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master by default):

  • a staging environment (an iso-prod environment meant for testing and validation purpose),
  • the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

  • continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
  • continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Usage

Include

In order to include this template in your project, add the following to your gitlab-ci.yml:

include:
  - project: 'to-be-continuous/terraform'
    ref: '2.3.1'
    file: '/templates/gitlab-ci-terraform.yml'

Template jobs

The Terraform template implements - for each environment presented above - the following jobs:

  • environment creation plan: this job is optional and computes the environment changes before applying them. When enabled, the env creation has to be applied manually. By default, this job is enabled only for the production environment.
  • environment creation apply: applies the environment changes, whether manually by applying the upstream plan (when enabled), or automatically.
  • environment destruction: can be executed manually on non-production envs only.

Global configuration

The Terraform template uses some global configuration used throughout all jobs.

Name description default value
TF_IMAGE the Docker image used to run Terraform CLI commands
โš ๏ธ set the version required by your project
hashicorp/terraform:light
TF_GITLAB_BACKEND_DISABLED Set to true to disable GitLab managed Terraform State none (enabled)
TF_PROJECT_DIR Terraform project root directory .
TF_SCRIPTS_DIR Terraform (hook) scripts base directory (relative to $TF_PROJECT_DIR) .
TF_OUTPUT_DIR Terraform output directory (relative to $TF_PROJECT_DIR). Everything generated in this directory will be kept as job artifacts. tf-output
TF_EXTRA_OPTS Default Terraform extra options (applies to all Terraform commands) none
TF_INIT_OPTS Default Terraform extra init options none
TF_APPLY_OPTS Default Terraform extra apply options none
TF_DESTROY_OPTS Default Terraform extra destroy options none

Secrets management

Here are some advices about your secrets (variables marked with a ๐Ÿ”’):

  1. Manage them as project or group CI/CD variables:
    • masked to prevent them from being inadvertently displayed in your job logs,
    • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
  2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
  3. Don't forget to escape special characters (ex: $ -> $$).

Terraform integration in Merge Requests

This template enables Terraform integration in Merge Requests.

As a result if you enabled your production environment, every merge request will compute and display infrastructure changes compared to master branch.

GitLab managed Terraform State

By default, this template enables GitLab managed Terraform State (set $TF_GITLAB_BACKEND_DISABLED to disable).

Error acquiring the state lock workaround

The template takes care of configuring the http backend, including with authentication credentials (using GitLab job token).

Anyway - depending on the Terraform version you are using - you may face this error when applying a plan that was computed in an upstream job:

Error locking state: Error acquiring the state lock: HTTP remote state endpoint requires auth

This is a known issue. A simple workaround is to create a Project Access Token with API rights, then declare it as a masked secret variable with name TF_PASSWORD in your Terraform project.

How to use GitLab backend in your development environment ?

First create a Project Access Token or Personal Access Token.

In your shell terminal, execute the following script:

#!/bin/bash

# TODO: replace 3 next variables
MY_PROJECT_PATH="path/to/my-project"
MY_ENV_NAME="dev"
TF_HTTP_PASSWORD="YOUR-ACCESS-TOKEN"

CI_API_V4_URL=https://gitlab.com/api/v4
CI_PROJECT_ID=${MY_PROJECT_PATH//\//%2f}

TF_HTTP_ADDRESS="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/$MY_ENV_NAME"
TF_HTTP_LOCK_ADDRESS="${TF_HTTP_ADDRESS}/lock"
TF_HTTP_LOCK_METHOD="POST"
TF_HTTP_UNLOCK_ADDRESS="${TF_HTTP_ADDRESS}/lock"
TF_HTTP_UNLOCK_METHOD="DELETE"
TF_HTTP_USERNAME="gitlab-token"
TF_HTTP_RETRY_WAIT_MIN="5"

terraform -v

terraform init \
    -reconfigure \
    -backend-config=address="${TF_HTTP_ADDRESS}" \
    -backend-config=lock_address="${TF_HTTP_LOCK_ADDRESS}" \
    -backend-config=unlock_address="${TF_HTTP_UNLOCK_ADDRESS}" \
    -backend-config=username="${TF_HTTP_USERNAME}" \
    -backend-config=password="${TF_HTTP_PASSWORD}" \
    -backend-config=lock_method="${TF_HTTP_LOCK_METHOD}" \
    -backend-config=unlock_method="${TF_HTTP_UNLOCK_METHOD}" \
    -backend-config=retry_wait_min="${TF_HTTP_RETRY_WAIT_MIN}"

Environments configuration

As seen above, the Terraform template may support up to 4 environments (review, integration, staging and production).

Here are configuration details for each environment.

Review environments

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the TF_REVIEW_ENABLED variable (see below).

Here are variables supported to configure review environments:

Name description default value
TF_REVIEW_ENABLED Set to true to enable your review environments none (disabled)
TF_REVIEW_EXTRA_OPTS Terraform extra options for review env (applies to all Terraform commands) $TF_EXTRA_OPTS
TF_REVIEW_INIT_OPTS Terraform extra init options for review env $TF_INIT_OPTS
TF_REVIEW_PLAN_ENABLED Set to true to enable separate Terraform plan job for review env. none (disabled)
TF_REVIEW_PLAN_OPTS Terraform extra plan options for review env none
TF_REVIEW_APPLY_OPTS Terraform extra apply options for review env $TF_APPLY_OPTS
TF_REVIEW_DESTROY_OPTS Terraform extra destroy options for review env $TF_DESTROY_OPTS

Integration environment

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the TF_INTEG_ENABLED variable (see below).

Here are variables supported to configure the integration environment:

Name description default value
TF_INTEG_ENABLED Set to true to enable your integration env none (disabled)
TF_INTEG_EXTRA_OPTS Terraform extra options for integration env (applies to all Terraform commands) $TF_EXTRA_OPTS
TF_INTEG_INIT_OPTS Terraform extra init options for integration env $TF_INIT_OPTS
TF_INTEG_PLAN_ENABLED Set to true to enable separate Terraform plan job for integration env. none (disabled)
TF_INTEG_PLAN_OPTS Terraform extra plan options for integration env none
TF_INTEG_APPLY_OPTS Terraform extra apply options for integration env $TF_APPLY_OPTS
TF_INTEG_DESTROY_OPTS Terraform extra destroy options for integration env $TF_DESTROY_OPTS

Staging environment

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (master by default).

It is disabled by default and can be enabled by setting the TF_STAGING_ENABLED variable (see below).

Here are variables supported to configure the staging environment:

Name description default value
TF_STAGING_ENABLED Set to true to enable your staging env none (disabled)
TF_STAGING_EXTRA_OPTS Terraform extra options for staging env (applies to all Terraform commands) $TF_EXTRA_OPTS
TF_STAGING_INIT_OPTS Terraform extra init options for staging env $TF_INIT_OPTS
TF_STAGING_PLAN_ENABLED Set to true to enable separate Terraform plan job for staging env. none (disabled)
TF_STAGING_PLAN_OPTS Terraform extra plan options for staging env none
TF_STAGING_APPLY_OPTS Terraform extra apply options for staging env $TF_APPLY_OPTS
TF_STAGING_DESTROY_OPTS Terraform extra destroy options for staging env $TF_DESTROY_OPTS

Production environment

The production environment is the final deployment environment associated with your production branch (master by default).

It is disabled by default and can be enabled by setting the TF_PROD_ENABLED variable (see below).

Here are variables supported to configure the production environment:

Name description default value
TF_PROD_ENABLED Set to true to enable your production env none (disabled)
TF_PROD_EXTRA_OPTS Terraform extra options for production env (applies to all Terraform commands) $TF_EXTRA_OPTS
TF_PROD_INIT_OPTS Terraform extra init options for production env $TF_INIT_OPTS
TF_PROD_PLAN_ENABLED Set to true to enable separate Terraform plan job for production env. true (enabled)
TF_PROD_PLAN_OPTS Terraform extra plan options for production env none
TF_PROD_APPLY_OPTS Terraform extra apply options for production env $TF_APPLY_OPTS

Hook scripts

Terraform jobs also support optional hook scripts from your project, located in the $TF_SCRIPTS_DIR directory (root project dir by default, but may be overridden).

  • tf-pre-init.sh is executed before running terraform init
  • tf-pre-apply.sh is executed before running terraform apply
  • tf-post-apply.sh is executed after running terraform apply
  • tf-pre-destroy.sh is executed before running terraform destroy
  • tf-post-destroy.sh is executed after running terraform destroy

Terraform commands overrides

Instead of creating hook scripts, you can also override and/or decorate the Terraform commands using predefined .tf-commands template block, referenced by the !reference directive.

By default, the .tf-commands, block is composed as below:

.tf-commands:
  init: 
    - !reference [ .tf-commands, default, init ]
  plan: 
    - !reference [ .tf-commands, default, plan ]
  apply: 
    - !reference [ .tf-commands, default, apply ]
  destroy: 
    - !reference [ .tf-commands, default, destroy ]

You can override it for example in the following way:

.tf-commands:
  init: 
    - source sandbox.env
    - !reference [ .tf-commands, default, init ]
    - echo "I'm executed after the terraform init command"

You can use this mechanism to source to the current shell your own environmental variables.

Environment & Terraform variables support

You have to be aware that your Terraform code has to be able to cope with various environments (review, integration, staging and production), each with different application names, exposed routes, settings, ...

In order to be able to implement some genericity in your code, you should use Terraform variables (in your Terraform files), and environment variables (in your hook scripts):

  1. any predefined GitLab CI variable may be freedly used in your hook scripts or extra options variables (ex: TF_EXTRA_OPTS: "-var project_name=$CI_PROJECT_NAME")
  2. you may also use custom GitLab variables to pass values to your hook script or even directly as Terraform variables using the right syntax (ex: env variable $TF_VAR_ssh_private_key_file will be visible as ssh_private_key_file Terraform variable in your code)
  3. dynamic variables provided by the template:
    • environment_type: the environment type (review, integration, staging or production)
    • environment_name (set as $CI_ENVIRONMENT_NAME): the full environment name (ex: review/fix-prometheus-configuration, integration, staging or production)
    • environment_slug (set as $CI_ENVIRONMENT_SLUG): the slugified environment name (ex: review-fix-promet-r13zmu, integration, staging or production)

When managing multiple environments, it is a good practice to prefix your Terraform resource names with environment_slug variable.

Example:

resource "aws_instance" "web_server" {
  name = "myproj_${var.environment_slug}_web_server"
  ...
}

How to manage separate values per environment?

It may happen that you need to use different configuration variables depending on the environment you are deploying. For instance separate GOOGLE_CREDENTIALS if you're using Google Provider.

For this, you shall either use GitLab scoped variables or our scoped variables syntax to limit/override some variables values, using $CI_ENVIRONMENT_NAME as the conditional variable.

Example: different OpenStack provider configuration for production

variables:
  # global OpenStack provider configuration
  OS_AUTH_URL: "https://openstack-nonprod.api.domain/v3"
  OS_TENANT_ID: "my-nonprod-tenant"
  OS_TENANT_NAME: "my-project-nonprod"
  OS_INSECURE: "true"
  # OS_USERNAME & OS_PASSWORD are defined as secret GitLab CI variables

  # overridden configuration for production
  scoped__OS_AUTH_URL__if__CI_ENVIRONMENT_NAME__equals__production: "https://openstack-prod.api.domain/v3"
  scoped__OS_TENANT_ID__if__CI_ENVIRONMENT_NAME__equals__production: "my-prod-tenant"
  scoped__OS_TENANT_NAME__if__CI_ENVIRONMENT_NAME__equals__production: "my-project-prod"
  scoped__OS_INSECURE__if__CI_ENVIRONMENT_NAME__equals__production: "false"
  # OS_USERNAME & OS_PASSWORD are overridden as secret GitLab CI variables

Supported job artifacts

The Terraform template supports job artifacts that your Terraform code may generate and need to propagate to downstream jobs:

  • $TF_OUTPUT_DIR directory and all contained files are stored as job artifacts. Allows to propagate files.
  • If present, the terraform.env file is stored as a dotenv artifact. Allows to propagate environment variables.

Examples:

  • When used in conjuction with Ansible template, your Terraform script may generate the Ansible inventory file into the $TF_OUTPUT_DIR directory.
  • When dynamically obtaining a floating IP address, your Terraform script may generate the terraform.env file to propated it as an environment variables.

tflint job

tflint is a Terraform Linter and uses the following variables:

Name description default value
TF_TFLINT_IMAGE the Docker image used to run tflint ghcr.io/terraform-linters/tflint-bundle:latest
TF_TFLINT_DISABLED Set to true to disable tflint none (enabled)
TF_TFLINT_ARGS tflint extra options and args --enable-plugin=google --enable-plugin=azurerm --enable-plugin=aws

tfsec job

tfsec uses static analysis of your terraform templates to spot potential security issues and uses the following variables:

Name description default value
TF_TFSEC_IMAGE the Docker image used to run tfsec tfsec/tfsec-ci
TF_TFSEC_ENABLED Set to true to enable tfsec none (disabled)
TF_TFSEC_ARGS tfsec options and args .

checkov job

checkov is a static code analysis tool for infrastructure-as-code and uses the following variables:

Name description default value
TF_CHECKOV_IMAGE the Docker image used to run checkov bridgecrew/checkov
TF_CHECKOV_ENABLED Set to true to enable checkov none (disabled)
TF_CHECKOV_ARGS checkov options and args --directory .

You can skip checkov specific check adding following comment in code :

resource "aws_s3_bucket" "foo-bucket" {
  region        = var.region
    #checkov:skip=CKV_AWS_20:The bucket is a public static content host
  bucket        = local.bucket_name
  force_destroy = true
  acl           = "public-read"
}

infracost job

Infracost shows cloud cost estimates for infrastructure-as-code projects and uses the following variables:

Name description default value
TF_INFRACOST_ENABLED Set to true to enable infracost none (disabled)
TF_INFRACOST_IMAGE the infracost container image infracost/infracost
TF_INFRACOST_ARGS infracost CLI options and args breakdown
TF_INFACOST_USAGE_FILE infracost usage file infracost-usage.yml
๐Ÿ”’ INFRACOST_API_KEY the infracost API key required

To use infracost, an api key is needed. To obtain it run :

docker run -it --name infracost infracost/infracost register
Please enter your name and email address to get an API key.
See our FAQ (https://www.infracost.io/docs/faq) for more details.
Name: Your Name
โœ” Email: you_email@domainโ–ˆ

Thank you !
Your API key is: api_key

Save the API key as ๐Ÿ”’ INFRACOST_API_KEY GitLab secret variable.

Set INFRACOST_CURRENCY variable to set currency ISO 4217 prices should be converted to. Defaults to USD.

GitLab compatibility

โ„น๏ธ This template is actually tested and validated on GitLab Community Edition instance version 13.12.11