Continuous Integration in Common Lisp with Github Actions, Part 2: pulling code from github, Ci-utils, and test frameworks.

In part 1, I described the basics of how I set up CI on github actions. This time, I'll use CI-Utils and show some examples of its scripts for using various testing frameworks.

repo for part 2

Manual git checkouts for dependencies

Sometimes we need to update multiple projects at once, and tests in one won't pass if we don't have the current version of the other. Or maybe we need things that aren't in Quicklisp to start with.

In that case, we can manually check out the other project until QL picks up the changes. (Adding an action to remind you to undo it in a month is left as an exercise for the reader).

Roswell's install-for-ci.sh script adds ~/lisp/ to the ASDF search path, so we just need to create that and check out other projects there.

See the readme for the checkout action for full details on how to configure it.


    - run: mkdir -p ~/lisp/

    - name: ci-utils fork
      uses: actions/checkout@v2
      with:
        # check out my fork of CI-Utils
        repository: 3b/ci-utils
        # on branch test2
        ref: test2
        # into a subdir of ~/lisp/
        path: ~/lisp/ci-utils

CI-Utils

Ci-utils adds various things useful for CI, for example if your tests or build scripts need to distinguish whether it is running in CI etc. It also adds some convenient scripts for running tests using various test frameworks. We install it using roswell so it will install the scripts, and also tell github to add the script path to PATH.


    - name: install ci-utils
      run: |
        ros install ci-utils
        echo "::add-path::$HOME/.roswell/bin"

In my fork (will send PR soon) CI-Utils adds a wrapper for hand-made tests with no framework where you just want evaluate a form and see if it returns true or not, which simplifies the "load and run" step.



    - name: load code and run tests
      shell: bash
      run: |
        run-test-forms -l ci-examples2/test "(ci-example2.test:run-tests-for-ci)"
        run-test-forms -l ci-example2 "(= (ci-example2:run 3) 4)"

CI status

Test frameworks

Prove

Usually we want more structure to our tests, so use a test framework like Parachute, Prove, Rove, or FiveAM.

All of those have similar scripts, except FiveAM which is supported by a script in CI-Utils.

an example using Prove: repo on branch prove

First, install prove instead of CI-Utils


    - name: install prove
      run: |
        ros install prove
        echo "::add-path::$HOME/.roswell/bin"

then run the tests (run-prove seems to want tests in a separate .asd file?)

Prove's default output color scheme doesn't seem to go well with github actions log display, so disable colors.


    - name: load code and run tests
      shell: bash
      run: |
        run-prove --without-colors ci-example2-test.asd

Prove CI status

Parachute

With run-parachute, we use -l to load a test system, then pass a list of test names to run.

(example from 3b-hdr )


    - name: install parachute
      run: |
        ros install parachute
        echo "::add-path::$HOME/.roswell/bin"

For the list of tests, we can just pass the name of the package containing the tests, or actual names of tests (package qualified, i think?)

Here we load the 3b-hdr/test system, and run all tests from 3b-hdr/test package.

    - name: load code and run tests
      shell: bash
      run: |
        run-parachute --quickload "3b-hdr/test" "3b-hdr/test"

3b-hdr CI status