|
| 1 | +# Workflow templates |
| 2 | + |
| 3 | +This document explains how to set up Continuous Integration (CI) workflows for an Exercism Language Track using GitHub Actions (GHA). It provides best practices and examples for you to use to make your own fast, reliable and robust CI workflows. The GHA workflows in this folder can be adapted to work with any CI, because the base structure will remain the same. |
| 4 | + |
| 5 | +It will: |
| 6 | + |
| 7 | +- Outline the ideal CI workflow |
| 8 | +- Discuss considerations and recommendations |
| 9 | +- Provide you with some templates to use |
| 10 | +- Leave you with a guide to migrating from Travis |
| 11 | + |
| 12 | +Example implementation of these workflow files [can be found in `exercism/javascript`][git-javascript]. |
| 13 | + |
| 14 | +## **HELP**: this looks like _a lot_ of work :sweat: |
| 15 | + |
| 16 | +The rest of the document is designed to explain how the workflows work. If you're in a hurry, and you just want to switch from Travis or Circle to GHA without optimizing the PR scripts, scroll down to the **~10 minute guide** on [**Migrating from Travis**](#migrating-from-travis). |
| 17 | + |
| 18 | +## Track CI actions |
| 19 | + |
| 20 | +The recommended actions for checking the content of your repository has integrity are as follows: |
| 21 | + |
| 22 | +1. [`configlet` linting][git-configlet] in order to check `config.json` |
| 23 | +2. check for stubs |
| 24 | +3. check for documentation (`v3` requires new files; this might move to `configlet`) |
| 25 | +4. lint the exercises using a "maintainers" configuration |
| 26 | +5. test the exercises using the example/examplar files (can include build step) |
| 27 | + |
| 28 | +There can also be track-specific actions. For example: |
| 29 | + |
| 30 | +1. check the [integrity][wiki-integrity] of the exercise configurations |
| 31 | +2. check the formatting of the exercise files |
| 32 | + |
| 33 | +And perhaps you'd want more Quality Of Life checks, such as: |
| 34 | + |
| 35 | +1. ensure CONTRIBUTING exists |
| 36 | +2. ensure there is a sane lockfile for dependencies |
| 37 | +3. ensure links inside markdown files are valid |
| 38 | +4. ... |
| 39 | + |
| 40 | +## Recommendations |
| 41 | + |
| 42 | +### Frequency of running checks |
| 43 | + |
| 44 | +For each action think about how often it should run. |
| 45 | + |
| 46 | +- The `configlet` linting is something that's so important (because a track can break if the `config.json` breaks) that it should probably always run, **but** only needs to run once per commit. |
| 47 | +- The existence or integrity of files only needs to run once per commit. |
| 48 | +- If a track is supposed to run under multiple runtime-versions or compiler-versions, building/testing exercises should be ran against each supported version |
| 49 | +- PRs _probably_ only need to run actions on files _added_ or _changed_, but since a file can influence an exercise, it's safer to run the actions for the _exercise_, if one of its files changes. |
| 50 | + |
| 51 | +It can be very helpful to make the actions that should run, available locally as well. This means that the scripts that do the actual work are also manually runnable. To achieve this **do not inline** the action inside the workflow files., but create a standalone script. For example, checking for stubs can be completely bashed out inside the workflow file, but the recommendation here is to create a new executable script `scripts/ci-check` instead. |
| 52 | + |
| 53 | +> "But the command is very short, e.g. `eslint . --ext ts --ext tsx`". |
| 54 | +> |
| 55 | +> When this command needs to be updated, it now needs to update in all the places in the documentation, the workflow files, ánd in the _minds of the maintainers_. Extracting this to a script resolves all that. Reading a workflow file can also be **very** daunting. |
| 56 | +
|
| 57 | +### Checks on PRs where exercises change |
| 58 | + |
| 59 | +The `scripts/pr` and `scripts/pr-check` scripts (see [templates](#templates)) are run with multiple arguments, one for each file that has been changed or added in this PR. For example, if `two-fer` has been updated, a call might look like this: |
| 60 | + |
| 61 | +```bash |
| 62 | +scripts/pr exercises/two-fer/README.md exercises/two-fer/.meta/example.ext |
| 63 | +``` |
| 64 | + |
| 65 | +It's recommended to run any actions against the changed _exercise_ and not the changed _file_. This is because changing a file is likely to trigger changes for the entire exercise (think: configuration, packages). |
| 66 | + |
| 67 | +> **Not ready?** / **Complex?** |
| 68 | +> |
| 69 | +> Before implementing this optimization, it may be safely ignored! The migration guide hints at adding at a later stage. If the input arguments are ignored, all the checks will run on all the exercises. **This is perfectly fine**. It will just take longer. |
| 70 | +
|
| 71 | +### Integrity checks |
| 72 | + |
| 73 | +If the track has a single "top-level" dependency file and/or other configuration files, add an [integrity][wiki-integrity] step (that exists alongside a `scripts/sync` or `bin/sync`, which would copy all configuration files to all exercises), which ensures that the top-level/base files are the same as the one copied to the exercise directories. Now dependencies can be updated, synced across the repository, and we can ensure that all exercises have the same configuration. |
| 74 | + |
| 75 | +A common way to accomplish this is to use a checksum. Ubuntu (and various other Linux distributions) comes with a tool called `sha1sum`, but using _whichever_ method to hash or reduce the configuration file (md5, sha1, crc32) to a checksum value, would work: |
| 76 | + |
| 77 | +```bash |
| 78 | +$ sha1sum README.md |
| 79 | +cd58091c5043bf21f00d39ff1740d8b2976deeff *README.md |
| 80 | +``` |
| 81 | + |
| 82 | +### Security checks |
| 83 | + |
| 84 | +If the track uses additional workflows that require access to the GitHub token or other secrets, it's best practice to pin **all** actions used in the workflow to a specific commit. See [GitHub's security hardening guide][github-actions-security] for details. |
| 85 | + |
| 86 | +For example: |
| 87 | + |
| 88 | +```diff |
| 89 | +- uses: julia-actions/setup-julia@v1 |
| 90 | ++ uses: julia-actions/setup-julia@d26d1111976eae5f00db04f0515ab744ec9cd79e # 1.3.1 |
| 91 | +``` |
| 92 | + |
| 93 | +If the tooling has lockfiles for dependency management, consider checking it into the repository and use a "frozen lockfile" inside the workflow files. For example: `npm ci`, `yarn install --frozen-lockfile` and `bundle install --frozen`. This ensures that the lockfile is up-to-date when changing dependencies and prevents malicious packages to come in. |
| 94 | + |
| 95 | +## Templates |
| 96 | + |
| 97 | +In [this directory][workflow-template-dir] at minimum there are the following templates: |
| 98 | + |
| 99 | +- [`configlet.yml`][workflow-template-configlet-yml]: This workflow will do a fetch the latest configlet binary and lint this repository. Run on every commit. For PRs, runs on the actual commit and a "after merge" tree. |
| 100 | +- [`ci.yml`][workflow-template-ci-yml]: This workflow **only runs on the main branch**, once on each commit. |
| 101 | + 1. Run a 'pre-check' command (check for stubs, lint, docs, etc.) for all exercises |
| 102 | + 2. Run a 'ci' command (build and test) for multiple versions, for all exercises |
| 103 | +- [`pr.ci.yml`][workflow-template-pr-ci-yml]: This workflow **only runs on PRs**, once on each commit. |
| 104 | + 1. Run a 'pre-check' command (check for stubs, lint, docs, etc.) for changed files |
| 105 | + 2. Run a 'ci' command (build and test) for multiple versions, for changed exercises |
| 106 | + |
| 107 | +The non-pr workflows can also be triggered via [`workflow_dispatch`][github-workflow-dispatch]. |
| 108 | + |
| 109 | +Each file has listed at the top which "scripts" should be available. If you want these to be binaries, replace `scripts/xxx` with `bin/xxx`. Some tooling will _require_ binaries to be inside a `bin` folder. |
| 110 | + |
| 111 | +- `scripts/ci`: a script that should build and test all exercises using the example solutions against the tests |
| 112 | +- `scripts/ci-check`: a script that should lint all exercises, and optionally check for stubs, configuration integrity, and more |
| 113 | +- `scripts/pr`: same as `scripts/ci`, but should only run exercises resolved from the paths given as input |
| 114 | +- `scripts/pr-check`: same as `scripts/ci-check`, but should only run for files or exercises resolved from the paths given as input |
| 115 | + |
| 116 | +## Migrating from Travis |
| 117 | + |
| 118 | +Here is an example `.travis.yml` (taken from the `elm` track): |
| 119 | + |
| 120 | +```yml |
| 121 | +sudo: false |
| 122 | +language: node_js |
| 123 | +node_js: |
| 124 | + - lts/* |
| 125 | +script: |
| 126 | + - bin/fetch-configlet |
| 127 | + - bin/configlet lint |
| 128 | + - bin/build.sh |
| 129 | +``` |
| 130 | +
|
| 131 | +In order to convert this quickly to GitHub Actions, take the following steps: |
| 132 | +
|
| 133 | +### Determine the template variables |
| 134 | +
|
| 135 | +| variable | value | |
| 136 | +| --------------------------- | ------------------------------------ | |
| 137 | +| `<track>` | `elm` | |
| 138 | +| `<image-name>` | `ubuntu-latest` | |
| 139 | +| `<action to setup tooling>` | `actions/setup-node@v1` | |
| 140 | +| `<install dependencies>` | `npm ci` (happens inside `build.sh`) | |
| 141 | +| `<code-extensions>` | `.elm` | |
| 142 | + |
| 143 | +> Found the setup action via [this search](https://github.com/actions/?q=setup+node&type=&language=). |
| 144 | +> Found the image name by looking at [default distribution for Travis](https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment). |
| 145 | + |
| 146 | +### Determine the steps |
| 147 | + |
| 148 | +- `bin/fetch-configlet`: don't need this anymore when using `configlet.yml` workflow |
| 149 | +- `bin/configlet lint`: don't need this anymore when using `configlet.yml` workflow |
| 150 | +- `bin/build.sh`: single script that does everything |
| 151 | + |
| 152 | +### Prepare the "scripts" |
| 153 | + |
| 154 | +This track uses the `bin` folder, so inside the `bin` folder, create the following files: |
| 155 | + |
| 156 | +```bash |
| 157 | +# bin/pr |
| 158 | +bin/build.sh |
| 159 | +``` |
| 160 | + |
| 161 | +```bash |
| 162 | +# bin/pr-check |
| 163 | +echo "No checks yet" |
| 164 | +``` |
| 165 | + |
| 166 | +```bash |
| 167 | +# bin/ci |
| 168 | +bin/build.sh |
| 169 | +``` |
| 170 | + |
| 171 | +```bash |
| 172 | +# bin/ci-check |
| 173 | +echo "No checks yet" |
| 174 | +``` |
| 175 | + |
| 176 | +Creating these as _separate_ binaries will allow for optimisation later. No need to in-line anything right now. |
| 177 | + |
| 178 | +### Fill in the templates |
| 179 | + |
| 180 | +Here is the diff for `workflows/ci.yml`. |
| 181 | + |
| 182 | +```diff |
| 183 | +# # .github/workflows/ci.yml |
| 184 | +
|
| 185 | + # This workflow will do a clean install of node dependencies and run tests across different versions |
| 186 | + # |
| 187 | +- # Replace <track> with the track name |
| 188 | +- # Replace <image-name> with an image to run the jobs on |
| 189 | +- # Replace <action to setup tooling> with a github action to setup tooling on the image |
| 190 | +- # Replace <install dependencies> with a cli command to install the dependencies |
| 191 | +- # Replace <code-extensions> with file extensions that should trigger the workflow |
| 192 | +- # |
| 193 | +- # Find Github Actions to setup tooling here: |
| 194 | +- # - https://github.com/actions/?q=setup&type=&language= |
| 195 | +- # - https://github.com/actions/starter-workflows/tree/main/ci |
| 196 | +- # - https://github.com/marketplace?type=actions&query=setup |
| 197 | +- # |
| 198 | + # Requires scripts: |
| 199 | +- # - scripts/ci-check |
| 200 | +- # - scripts/ci |
| 201 | ++ # - bin/ci-check |
| 202 | ++ # - bin/ci |
| 203 | +
|
| 204 | +- name: <track> / main |
| 205 | ++ name: elm / main |
| 206 | +
|
| 207 | + on: |
| 208 | + push: |
| 209 | + branches: [main] |
| 210 | + workflow_dispatch: |
| 211 | +
|
| 212 | + jobs: |
| 213 | + precheck: |
| 214 | +- runs-on: <image-name> |
| 215 | ++ runs-on: ubuntu-latest |
| 216 | +
|
| 217 | + steps: |
| 218 | + - uses: actions/checkout@v2 |
| 219 | +- - name: Use <setup tooling> |
| 220 | +- uses: <action to setup tooling> |
| 221 | ++ - name: Use Node LTS |
| 222 | ++ uses: actions/setup-node@v1 |
| 223 | + with: |
| 224 | +- # here, use the LTS/stable version of the track's tooling |
| 225 | +- # node-version: 12.x |
| 226 | ++ node-version: 12.x |
| 227 | +
|
| 228 | + - name: Install project dependencies |
| 229 | +- run: <install dependencies> |
| 230 | ++ run: npm ci |
| 231 | +
|
| 232 | +- - name: Run exercism/<track> ci pre-check (checks config, lint code) for all exercises |
| 233 | +- run: scripts/ci-check |
| 234 | ++ - name: Run exercism/elm ci pre-check (checks config, lint code) for all exercises |
| 235 | ++ run: bin/ci-check |
| 236 | +
|
| 237 | + ci: |
| 238 | +- runs-on: <image-name> |
| 239 | ++ runs-on: ubuntu-latest |
| 240 | +
|
| 241 | + strategy: |
| 242 | + matrix: |
| 243 | +- # here, add all SUPPORTED versions only |
| 244 | +- # version: [10.x, 12.x, 14.x] |
| 245 | ++ version: [10.x, 12.x, 14.x] |
| 246 | +
|
| 247 | + steps: |
| 248 | + - uses: actions/checkout@v2 |
| 249 | +- - name: Use <setup tooling> ${{ matrix.version }} |
| 250 | +- uses: <action to setup tooling> |
| 251 | ++ - name: Use Node ${{ matrix.version }} |
| 252 | ++ uses: actions/setup-node@v1 |
| 253 | + with: |
| 254 | +- # below: see how to inject the version |
| 255 | +- # node-version: ${{ matrix.version }} |
| 256 | ++ node-version: ${{ matrix.version }} |
| 257 | +
|
| 258 | + - name: Install project dependencies |
| 259 | +- run: <install dependencies> |
| 260 | ++ run: npm ci |
| 261 | +
|
| 262 | +- - name: Run exercism/<track> ci (runs tests) for all exercises |
| 263 | +- run: scripts/ci |
| 264 | ++ - name: Run exercism/elm ci (runs tests) for all exercises |
| 265 | ++ run: bin/ci |
| 266 | +``` |
| 267 | + |
| 268 | +`workflows/pr.yml` has the same changes, with the notable exception of the bash-fu that calls the `pr-check` and `pr` scripts with each changed file as argument: |
| 269 | + |
| 270 | +```diff |
| 271 | +# # .github/workflows/pr.yml |
| 272 | +
|
| 273 | +# # ... |
| 274 | +
|
| 275 | +- - name: Run exercism/<track> ci pre-check (stub files, config integrity) for changed exercises |
| 276 | ++ - name: Run exercism/elm ci pre-check (stub files, config integrity) for changed exercises |
| 277 | + run: | |
| 278 | + PULL_REQUEST_URL=$(jq -r ".pull_request.url" "$GITHUB_EVENT_PATH") |
| 279 | + curl --url $"${PULL_REQUEST_URL}/files" --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' | \ |
| 280 | +- jq -c '.[] | select(.status == "added" or .status == "modified") | select(.filename | match("\\.(<code-extensions>|md|json)$")) | .filename' | \ |
| 281 | ++ jq -c '.[] | select(.status == "added" or .status == "modified") | select(.filename | match("\\.(elm|md|json)$")) | .filename' | \ |
| 282 | +- xargs -r scripts/pr-check |
| 283 | ++ xargs -r bin/pr-check |
| 284 | +
|
| 285 | + # ... |
| 286 | +``` |
| 287 | + |
| 288 | +### Now it should work |
| 289 | + |
| 290 | +This is enough to convert to GitHub Actions, with the possibility to optimise your scripts. |
| 291 | + |
| 292 | +1. From `build.sh`, remove steps that should run only once, and extract them to the `ci-check.sh` and `pre-check.sh` files (hint, you can create `lint.sh`, and call that from both "scripts") |
| 293 | +2. To `pr.sh` and `pr-check.sh`, add optimisations that use the input arguments to determine which files or exercises to check. |
| 294 | +3. Add additional checks |
| 295 | +4. Add documentation how to run checks locally, and what each one tries to accomplish. |
| 296 | + |
| 297 | +## Troubleshooting |
| 298 | + |
| 299 | +If you run into any issues or want someone to review your workflows, please ping the `@exercism/github-actions` team. |
| 300 | + |
| 301 | +> **Changed a top-level file that should trigger a CI run on all exercises** |
| 302 | + |
| 303 | +At moment of writing, `pr.ci.yml` only allows for "extension" testing. Ideally this is updated to trigger always when certain files are changed (for example the binary to run the tests). However, these changes are often infrequent and done by maintainers, that the fact that `ci.yml` runs on main, always, for everything, is probably safe enough. |
| 304 | + |
| 305 | +> **Created a scripts/xxx file on Windows and now it doesn't work on {other OS}** |
| 306 | + |
| 307 | +By default, files created on Windows don't have metadata in the [git-index][git-index] embedded about their executability, because the model for permissions on windows is different. Git, by default, will use the git-index metadata to determine if the file should be executable on POSIX-based systems, and thus make the `scripts/xxx` file NOT executable. |
| 308 | + |
| 309 | +```bash |
| 310 | +git update-index --chmod=+x scripts/xxx |
| 311 | +git commit -m "Make scripts/xxx executable" |
| 312 | +``` |
| 313 | + |
| 314 | +[git-configlet]: https://github.com/exercism/configlet#lint |
| 315 | +[git-index]: https://www.git-scm.com/docs/git-update-index |
| 316 | +[git-javascript]: https://github.com/exercism/javascript/tree/f49ac022d3a55cbbb48dadbc6dbf1d407de72187/.github/workflows |
| 317 | +[github-workflow-dispatch]: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/ |
| 318 | +[github-actions-security]: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions |
| 319 | +[wiki-integrity]: https://en.wikipedia.org/wiki/File_verification |
| 320 | +[workflow-template-dir]: https://github.com/exercism/docs/tree/main/building/tracks/ci |
| 321 | +[workflow-template-ci-yml]: https://github.com/exercism/docs/tree/main/reference/templates/ci/ci.yml |
| 322 | +[workflow-template-configlet-yml]: https://github.com/exercism/docs/tree/main/reference/templates/ci/configlet.yml |
| 323 | +[workflow-template-pr-ci-yml]: https://github.com/exercism/docs/tree/main/reference/templates/ci/pr.ci.yml |
0 commit comments