SURESOFT HPC Workflow

Introduction

In this guide we will demonstrate an automated CI workflow in the HPC context that utilizes Singularity, HPC-Rocket and Fieldcompare to make resutlts of computations easily repeatable and reproducible.

Code

The code examples in this guide are shortened for readability. The entire code for the example workflow can be found on GitLab.

Motivation

Automating scientific workflows can increase the chance of detecting human-introduced programming or execution errors by performing repetitive and time-consuming tasks with greater accuracy and consistency than humans. This can reduce the time and effort required for testing and deployment, and help to ensure that the software meets the required quality standards. Moreover, automation enhances the reproducibility of scientific results by capturing all the necessary steps to generate the results within a script, enabling easy repeatability of workflows.

In the context of scientific research projects that demand significant computational resources, such as high-performance clusters, the need for automation becomes even more crucial. However, the utilization of high-performance clusters introduces an additional layer of complexity, posing challenges for researchers in terms of performing computations and identifying software defects. Despite the availability of several workflow automation tools for HPC systems, launching them in response to changes in the application code can be difficult due to user permission limitations that restrict connections to a CI pipeline.