Skip to content

GitLab CI template for S3 (Simple Storage Service)

This project implements a generic GitLab CI template for any S3 (Simple Storage Service) compatible object storage service.

This is a basic and very cheap solution to host static pages websites as well as progressive web applications.

Overview

This template implements continuous delivery/continuous deployment for projects hosted on S3 platforms.

It provides several features, usable in different modes.

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master by default):

  • a staging environment (an iso-prod environment meant for testing and validation purpose),
  • the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

  • continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
  • continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Usage

Include

In order to include this template in your project, add the following to your gitlab-ci.yml:

include:
  - project: 'to-be-continuous/s3'
    ref: '2.1.2'
    file: '/templates/gitlab-ci-s3.yml'

Global configuration

The S3 template uses some global configuration used throughout all jobs.

Name description default value
S3_CMD_IMAGE The Docker image used to run s3cmd commands d3fk/s3cmd:latest
S3_ENDPOINT_HOST Default S3 endpoint hostname (and port) has to be defined
S3_HOST_BUCKET Default DNS-style bucket+hostname:port template for accessing a bucket %(bucket)s.$S3_ENDPOINT_HOST
πŸ”’ S3_ACCESS_KEY Default S3 service Access Key has to be defined
πŸ”’ S3_SECRET_KEY Default S3 service Secret Key has to be defined
S3_BASE_BUCKET_NAME Base bucket name $CI_PROJECT_NAME (see GitLab doc)
S3_ROOT_PATH Default root path where files will be uploaded in the S3 bucket (⚠️ don't forget the starting /) none

Secrets management

Here are some advices about your secrets (variables marked with a πŸ”’):

  1. Manage them as project or group CI/CD variables:
    • masked to prevent them from being inadvertently displayed in your job logs,
    • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
  2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
  3. Don't forget to escape special characters (ex: $ -> $$).

Environments configuration

As seen above, the S3 template may support up to 4 environments (review, integration, staging and production).

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

  • environment_type: set to the type of environment (review, integration, staging or production),
  • environment_name: the application name (see below),
  • environment_url: set to $CI_ENVIRONMENT_URL.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

Here are configuration details for each environment.

Review environments

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are enabled by default and can be disabled by setting the S3_REVIEW_DISABLED variable (see below).

Here are variables supported to configure review environments:

Name description default value
S3_REVIEW_DISABLED Set to true to disable review environments none (enabled)
S3_REVIEW_ENDPOINT_HOST S3 endpoint hostname (and port) for review env (only define if different from default) $S3_ENDPOINT_HOST
πŸ”’ S3_REVIEW_ACCESS_KEY S3 service Access Key for review env (only define if different from default) $S3_ACCESS_KEY
πŸ”’ S3_REVIEW_SECRET_KEY S3 service Secret Key for review env (only define if different from default) $S3_SECRET_KEY
S3_REVIEW_BUCKET_NAME Bucket name for review env "${S3_BASE_BUCKET_NAME}-${CI_ENVIRONMENT_SLUG}" (ex: myproject-review-fix-bug-12)
S3_REVIEW_ENVIRONMENT_SCHEME The review environment protocol scheme https
S3_REVIEW_ENVIRONMENT_DOMAIN The review environment domain. none
S3_REVIEW_ROOT_PATH S3 bucket root path (prefix) for review env (only define if different from default) S3_ROOT_PATH

Note: By default, review environment.url will be built as ${S3_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${S3_REVIEW_ENVIRONMENT_DOMAIN}

Integration environment

The integration environment is the environment associated to your integration branch (develop by default).

It is enabled by default and can be disabled by setting the S3_INTEG_DISABLED variable (see below).

Here are variables supported to configure the integration environment:

Name description default value
S3_INTEG_DISABLED Set to true to disable the integration environment none (enabled)
S3_INTEG_ENDPOINT_HOST S3 endpoint hostname (and port) for integration env (only define if different from default) $S3_ENDPOINT_HOST
πŸ”’ S3_INTEG_ACCESS_KEY S3 service Access Key for integration env (only define if different from default) $S3_ACCESS_KEY
πŸ”’ S3_INTEG_SECRET_KEY S3 service Secret Key for integration env (only define if different from default) $S3_SECRET_KEY
S3_INTEG_BUCKET_NAME Bucket name for integration env ${S3_BASE_BUCKET_NAME}-integration
Β S3_INTEG_ENVIRONMENT_URL The integration environment url including scheme (ex: https://my-project-integration.s3-website.nonpublic.domain.com). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. none
S3_INTEG_ROOT_PATH S3 bucket root path (prefix) for integration env (only define if different from default) S3_ROOT_PATH

Staging environment

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (master by default).

It is enabled by default and can be disabled by setting the S3_STAGING_DISABLED variable (see below).

Here are variables supported to configure the staging environment:

Name description default value
S3_STAGING_DISABLED Set to true to disable the staging environment none (enabled)
S3_STAGING_ENDPOINT_HOST S3 endpoint hostname (and port) for staging env (only define if different from default) $S3_ENDPOINT_HOST
πŸ”’ S3_STAGING_ACCESS_KEY S3 service Access Key for staging env (only define if different from default) $S3_ACCESS_KEY
πŸ”’ S3_STAGING_SECRET_KEY S3 service Secret Key for staging env (only define if different from default) $S3_SECRET_KEY
S3_STAGING_BUCKET_NAME Bucket name for staging env ${S3_BASE_BUCKET_NAME}-staging
Β S3_STAGING_ENVIRONMENT_URL The staging environment url including scheme (ex: https://my-project-staging.s3-website.nonpublic.domain). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. none
S3_STAGING_ROOT_PATH S3 bucket root path (prefix) for staging env (only define if different from default) S3_ROOT_PATH

Production environment

The production environment is the final deployment environment associated with your production branch (master by default).

It is enabled by default and can be disabled by setting the S3_PROD_DISABLED variable (see below).

Here are variables supported to configure the production environment:

Name description default value
S3_PROD_DISABLED Set to true to disable the production environment none (enabled)
S3_PROD_ENDPOINT_HOST S3 endpoint hostname (and port) for production env (only define if different from default) $S3_ENDPOINT_HOST
πŸ”’ S3_PROD_ACCESS_KEY S3 service Access Key for production env (only define if different from default) $S3_ACCESS_KEY
πŸ”’ S3_PROD_SECRET_KEY S3 service Secret Key for production env (only define if different from default) $S3_SECRET_KEY
S3_PROD_BUCKET_NAME Bucket name for production env $S3_BASE_BUCKET_NAME
S3_PROD_ENVIRONMENT_URL Β The production environment url including scheme (ex: https://my-project.s3-website.public.domain.com) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. none
AUTODEPLOY_TO_PROD Set this variable to auto-deploy to production. If not set deployment to production will be manual (default behaviour). none (disabled)
S3_PROD_ROOT_PATH S3 bucket root path (prefix) for production env (only define if different from default) S3_ROOT_PATH

Deployment jobs

Each environment has its own deployment job (associated with the right branch).

It uses the following variables:

Name description default value
S3_DEPLOY_ARGS s3cmd command and options to deploy files to the bucket put --acl-public --no-mime-magic --guess-mime-type --recursive
S3_DEPLOY_FILES Pattern(s) of files to deploy to the S3 bucket public/ (all files from public directory)
S3_WEBSITE_DISABLED Set to true to disable WebSite hosting by your S3 bucket none (enabled by default)
S3_WEBSITE_ARGS s3cmd command and options to enable WebSite hosting on the bucket ws-create --ws-index=index.html

If need be you could add your own hook script s3-pre-deploy.sh that will be triggered right before deploying files to the S3 bucket.

s3-cleanup-review job

This job allows destroying each review environment. Simply deletes the associated bucket.

s3-cleanup-all-review job

This job allows destroying all review environments at once (in order to save cloud resources).

It is disabled by default and can be controlled using the $CLEANUP_ALL_REVIEW variable:

  1. automatically executed if $CLEANUP_ALL_REVIEW set to force,
  2. manual job enabled from any master branch pipeline if $CLEANUP_ALL_REVIEW set to true (or any other value),

The first value force can be used in conjunction with a scheduled pipeline to cleanup cloud resources for instance everyday at 6pm or on friday evening.

The second one simply enables the (manual) cleanup job on the master branch pipeline.

Anyway destroyed review environments will be automatically re-created the next time a developer pushes a new commit on a feature branch.

⚠️ in case of scheduling the cleanup, you'll probably have to create an almost empty branch without any other template (no need to build/test/analyse your code if your only goal is to cleanup environments).

Variants

Vault variant

This variant allows delegating your secrets management to a Vault server.

Configuration

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

Name description default value
VAULT_BASE_URL The Vault server base API url none
πŸ”’ VAULT_ROLE_ID The AppRole RoleID must be defined
πŸ”’ VAULT_SECRET_ID The AppRole SecretID must be defined

Usage

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Name description
secret_path (path parameter) this is your secret location in the Vault server
field (query parameter) parameter to access a single basic field from the secret JSON payload

Example

include:
  # main template
  - project: 'to-be-continuous/s3'
    ref: '2.1.2'
    file: '/templates/gitlab-ci-s3.yml'
  # Vault variant
  - project: 'to-be-continuous/s3'
    ref: '2.1.2'
    file: '/templates/gitlab-ci-s3-vault.yml'

variables:
    # Secrets managed by Vault
    S3_ACCESS_KEY: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-backend/s3?field=access_key"
    S3_SECRET_KEY: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-backend/s3?field=secret_key"
    VAULT_BASE_URL: "https://vault.acme.host/v1"
    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

Gitlab compatibility

ℹ️ This template is actually tested and validated on GitLab Community Edition instance version 13.12.11