Regression Testing with Fieldcompare

Testing is a fundamental part of software quality assurance, and a comprehensive test suite is crucial in order to verify a software’s correctness. By testing all parts of a system and their interoperability, the developers make sure the software behaves in agreement with their intentions. Research software, and especially such that performs numerical simulations, is often employed as a tool to analyse and learn about physical systems. This makes testing more difficult due to the lack of a known expected behaviour to test against. In other words, it is not clear what the correct behaviour of the software should be. However, with regression-testing one may ensure that changes to the code do not introduce any undetected and unintentional changes to the software’s behaviour. In this technique, the results produced by the software are compared against stored reference results, for instance, from computations with an earlier version of the software. If significant deviations are detected between the results, the test suite is considered to have failed, thereby notifying the developers that a particular code modification lead to a change in behaviour. What a significant deviation is has to be decided for each test individually, and tolerances have to be chosen such that physically relevant deviations are detected while avoiding false positives from machine precision issues. In this workflow, we employ Fieldcompare to detect deviations between simulation re- sults produced in the CI and stored reference results. Fieldcompare has several mechanisms to control relative and absolute tolerances below which differences in the results are considered negligible. The default behaviour of Fieldcompare with respect to allowed deviations is very strict and suffices for the purposes of our workflow. The CI configuration for the regression-test job is given in the following listing:

regression-test:
  image: python:3.10

  stage: test

  needs: ["run-hpc-cluster-mpich-bind"]

  before_script:
    - pip install "fieldcompare[all]"

  script:
    - |
      fieldcompare file results/TemperatureField.avs \ 
                        reference_data/TemperatureField.avs