Continuous Integration in Common Lisp with Github Actions, Part 3: portability testing, msys, misc hacks

edit 2020/09/27: Updated msys info

For pngload, we wanted it to run on as many implementations as possible. To test this, we run tests on every combinations of implementation and OS easily supported by the combination of github actions and roswell.

  • sbcl32 and clisp32 are supported as nicknames of sbcl and clisp respectively, but still install 64-bit binaries on 64-bit OS.

  • Roswell can build clasp, but that takes too long to run on github CI servers (and pngload doesn't run on clasp yet anyway), so that is skipped for now. Eventually, it could be modified to pull a pre-built binary from another repo.

  • The available MacOS version doesn't seem to run 32 bit binaries, so we can't test ccl32, allegro, or cmucl there.

  • The clisp binary on MacOS doesn't support FFI, so is skipped.

  • sbcl-bin on MacOS is too old, so skip that as well

  • Some implementations don't install properly on the windows VM, so allegro, cmucl, abcl, ecl, and clisp are skipped there.

As far as I could tell, we need to explicitly list each combination we don't want, which is a bit verbose.

jobs:
  test:
    name: ${{ matrix.lisp }} on ${{ matrix.os }}
    strategy:
      matrix:
        lisp: [sbcl-bin,sbcl,ccl,ccl32,ecl,clisp,allegro,cmucl,abcl]
        os: [ubuntu-latest, macos-latest, windows-latest]
        exclude:
          # skip 32bit lisps on osx
            - os: macos-latest
              lisp: ccl32
            - os: macos-latest
              lisp: allegro
            - os: macos-latest
              lisp: cmucl
          # CFFI requires CLISP compiled with dynamic FFI support.
            - os: macos-latest
              lisp: clisp
          # sbcl-bin is too old on macos
            - os: macos-latest
              lisp: sbcl-bin
          # some implementations don't install properly on windows?
            - os: windows-latest
              lisp: allegro
            - os: windows-latest
              lisp: cmucl
            - os: windows-latest
              lisp: abcl
            - os: windows-latest
              lisp: ecl
            - os: windows-latest
              lisp: clisp
      fail-fast: false

To build latest sbcl release on windows, we need to configure the windows runner to use msys2 instead of the default git-bash normally used for shell: bash. (Actually we could probably build sbcl manually using whatever random combination of tools are in the PATH already, but Roswell will waste time trying to install msys anyway if it doesn't find it)

We will use the setup-msys2 action from msys2 to configure it, since manually installing things on top of the default installation is either slow or unreliable. (Status of msys2 on github actions is currently being worked on, so some of this will probably change in the near future. See

for more info on what is changing, and what the problems are.)

First we set the default shell on windows to msys2, which will be configured below.

# under jobs: test:
    defaults:
      run:
        # set "msys2" as default shell on windows
        # couldn't find any better way to do this, but seems to work?
        shell: ${{ fromJSON('[ "bash", "msys2 {0}" ]') [ matrix.os == 'windows-latest' ] }}

Then we use the msys2/setup-msys2@v2 action to configure msys2 on windows.

We want to build sbcl with mingw64, so set msystem to indicate that, and we want to see the $PATH entries added by ::add-path:: so set path-type to inherit.

If we need to install anything not included in the base image, or need to be sure it is the very latest (images are rebuilt every few weeks, so generally not very old anyway), we need to pass update: true. Usually we don't actually need everything installed by default, and updating everything would be slow, so usually we also want release: true when using update: true. Once it is updated, we can add install: 'whatever packages we need', but since we used release: true we need to specify everything we want that isn't in the base msysy2 install.

Installing without update: will usually work, but msys2 doesn't officially support installing things without updating the system first, so you will probably eventually get some random failures from that.


    - uses: msys2/setup-msys2@v2
      with:
        path-type: inherit
        msystem: MINGW64
        # set these to true if we want to install things from pacman
        release: false
        update: false
        # list all packages we want installed if using release&update true

        # for example the following would be enough for us to build sbcl
        # from git:
        # install: 'git base-devel unzip mingw-w64-x86_64-gcc mingw64/mingw-w64-x86_64-zlib'

In the windows specific config, we need to add the roswell bin dir under pwsh's $HOME to the $PATH, since it ends up there instead of under msys' home dir.

We also need to set MSYSCON so Roswell doesn't try to install its own copy of msys2.

    - name: windows specific settings
      if: matrix.os == 'windows-latest'
      shell: pwsh
      run: |
        git config --global core.autocrlf false
        echo "::add-path::$HOME/.roswell/bin"
        echo "::set-env name=MSYSCON::defterm"

Clisp doesn't currently build on ubuntu-latest ( https://sourceforge.net/p/clisp/bugs/688/ ), so manually install it


    - name: install clisp from apt
      if: matrix.lisp == 'clisp' && matrix.os == 'ubuntu-latest'
      run: |
        sudo apt install clisp
        ros use clisp/system
        ros install asdf

Unfortunately, while that gives us a runnable clisp, it still never finishes the CI run. It also doesn't give any indication of what is wrong, so clisp is disabled on linux too for now.

For pngload, there were some bugs that only showed up when compiled+loaded in one image, or when loaded from precompiled .fasl files, due to things like differences in handling of DEFCONSTANT or other compile-time side effects.

To test that, tests are run twice, once from a clean fasl cache, and again from cached fasls from previous run.


    - name: clear fasl cache
      run: |
        rm -rf ~/.cache/common-lisp/
        mkdir -p ~/.cache/common-lisp/


    - name: load code from clean fasl cache and run tests
      run: |
        run-test-forms -l pngload.test '(pngload.test:run-tests-for-ci)'

    - name: load code from fasls and run tests
      run: |
        run-test-forms -l pngload.test '(pngload.test:run-tests-for-ci)'

Finally, to hack around some libraries that work on cmucl but don't compile cleanly, compile it separately while ignoring errors so the compile+load during testing works.


    - name: cmucl hax
      # cmucl gets some build errors on deps we don't actually need, so try a few extra loads to get past that
      continue-on-error: true
      if: matrix.lisp == 'cmucl'
      run: |
        ros -e '(ql:quickload :skippy)'
        ros -e '(ql:quickload :skippy)'

(Had time to try a fix for that and send a PR, so actual pngload repo pulls a fork instead of the above workaround)