Continuous integration

Let’s take a breath. In this lecture, you learned about how to perform automatic rule checks, how to document your code, and how to write and execute tests.

There is an adage that we should not write more than 10 lines of code per day. It does not seem much, but if at each step you have to manually perform all quality checks, write and deploy the corresponding documentation, and design and run new tests to make sure the newly introduced lines does not break the code, you would probably write less than 10 lines a day.

GitLab CI/CD

What if at each modification, your code were shipped somewhere that has the correct environment to check and execute it, reporting any errors? This would mean that instead of manually performing all checks, you would outsource most of the work to a remote server that will do the boring work on your behalf, leaving you the luxury to focus on more interesting tasks. Crazy? Possible!

In this section, we will learn how to automate a workflow using the GitLab CI/CD (continuous integration/continuous deployment). A workflow can be: checking the code is correctly written (linting), running a test suite, deploying a package on a registry, deploying an online documentation, …

Before starting

Connect to the IN2P3 GitLab (https://gitlab.in2p3.fr) and create a new repository. If you have not yet done so, generate a ssh key, and add it to your account.

Then clone your newly created repository, and create a new empty file called .gitlab-ci.yml at the root of your repository. Also add the robust-programming folder. Your repository should look like:

(robprog) -bash-5.1$ ls -al
drwxr-xr-x.  4 student users   109 Mar 27 12:18 .
drwx------. 10 student users  4096 Mar 27 20:09 ..
drwxr-xr-x.  8 student users  4096 Mar 27 16:33 .git
-rw-r--r--.  1 student users   894 Mar 27 12:18 .gitlab-ci.yml
drwxr-xr-x.  8 student users   154 Mar 27 16:33 robust-programming

Your first pipeline

A CI pipeline is composed of a set of jobs that must all successfully run to completion. Jobs may be independent or depend on each other. The easiest form of dependency management in GitLab CI are pipeline stages. Jobs in a pipeline stage can run in parallel, but each pipeline stage must run to completion before the next stage starts. Let’s implement a first job to apply rule checking to our code. Add the following lines to the file:

default:
  image: almalinux:9

stages:
  - check
  - render
  - deploy

The first block specifies the operating system that we want to use in the remote server. You can use any name known from e.g. the DockerHub, and more. The second block defines the different stages of our pipeline (that we will have to define).

Let’s implement the check stage. Define a new block that contains:

Lint Python:
  image: python:3.9
  stage: check
  before_script:
    - |
      python -m venv .venv
      source .venv/bin/activate
      pip install hypothesis flake8
  script:
    - flake8 practicals/code/tests/sum.py

Lint C++:
  stage: check
  before_script:
    - |
      dnf update -y
      dnf install -y gcc-c++
      dnf install -y clang libubsan libasan clang-tools-extra valgrind
  script:
    - cd robust-programming/analyzers
    - make
    - clang-format -i mdbook_snippets.cpp

Lint Python (or Lint C++) is a (arbitrary) job name related to the check stage. In the case of the Python job, we specify another image than the default one (you can override any defaults within a job). The section before_script is where we install any dependencies. The section script is the actual execution of checks. More information on available keywords can be found online.

Save your work, commit, and push to your repository. Then go to the GitLab interface, select on the left bar: Build > Pipelines and observe the progression of your pipeline!

Exercise

Implement the other jobs and stages:

• check: add a job to run tests and report any failures.
• render: build your documentation written with mdbook in a previous section.
• deploy: deploy your documentation online using GitLab pages.