Concept

We decided to automate our workflow with the continuous integration (CI) service included with the GitLab platform, GitLab CI. Continuous integration refers to a development practice where software engineers merge their latest changes into the main code branch in short intervals. A connected continuous integration service like GitLab CI will pick up the new version of the code and then compile and run tests against it. Therefore, continuous integration serves as a feedback mechanism to ensure the correct functionality of the software.

The continuous integration service will execute a so-called pipeline, a sequence of tasks or jobs with several individual steps, forming an automated workflow.

The whole workflow is divided into three successive stages. In the first stage, we build a container using Singularity, that includes all the necessary software and dependencies to run the computation. Containers provide a way to package and distribute software in a portable and reproducible way, making it easier to run the computation on different machines and environments. Once the container is built, it can be deployed to a High Performance Computing (HPC) cluster using the tool HPC-Rocket that will submit the computation as a job to the Slurm scheduling system and monitor its progress. Upon finishing the computation, HPC-Rocket collects the produced data and copies it back to the CI pipeline where it will be compared to a trusted reference data set or experimental measurements using the Fieldcompare tool.