Skip to content

Commit d2b88d7

Browse files
committed
architecture: adjust after the migration to GitHub Actions
As of gitgitgadget/gitgitgadget#609, GitGitGadget no longer runs on a single Azure Pipelines runner, but instead on GitHub-hosted GitHub Actions runners. Let's adjust the part that described the Azure Pipelines' names and purposes, talking about the GitHub workflows and Actions instead. Signed-off-by: Johannes Schindelin <[email protected]>
1 parent d9aeb1e commit d2b88d7

File tree

1 file changed

+14
-65
lines changed

1 file changed

+14
-65
lines changed

content/architecture.md

Lines changed: 14 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -5,94 +5,43 @@ title: The architecture of GitGitGadget
55

66
GitGitGadget is implemented as a GitHub App (with the very imaginative name ["GitGitGadget"](https://github.com/apps/gitgitgadget)), which means that a webhook is called on certain events, such as new PR comments on PRs (e.g. `issue_comment`).
77

8-
In GitGitGadget's case, this webhook is implemented as an Azure Function: https://github.com/gitgitgadget/gitgitgadget/tree/master/azure-function
8+
In GitGitGadget's case, this webhook is implemented as [an Azure Function](https://github.com/gitgitgadget/gitgitgadget/tree/master/azure-function).
99

10-
Apart from validating that the payload really originated from GitHub, the Azure Function performs a rudimentary check whether the comment (if it was triggered by a comment) contains a command that GitGitGadget should act on, and depending on that check triggers the Azure Pipeline [GitGitGadget PR Handler](https://dev.azure.com/gitgitgadget/git/_build?definitionId=3).
10+
Apart from validating that the payload really originated from GitHub, the Azure Function performs a rudimentary check whether the comment (if it was triggered by a comment) contains a command that GitGitGadget should act on, and depending on that check triggers [the `handle-pr-comment` GitHub workflow](https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/handle-pr-comment.yml), which creates a Check Run that is shown in the Checks tab of the PR.
1111

12-
The `push` event (and also the `pull_request` event) is not actually handled by the Azure Function. It is handled via the Azure Pipeline being configured as "Pull request validation". That also allows it to be shown in the Checks tab of the PR.
12+
The `push` event (and also the `pull_request` event) is also handled by the Azure Function, triggering [the `handle-pr-push` GitHub workflow](https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/handle-pr-push.yml), which creates a Check Run that is shown in the Checks tab of the PR.
1313

14-
You can see the difference on [the summary page of the GitGitGadget PR Handler pipeline](https://dev.azure.com/gitgitgadget/git/_build?definitionId=3): the `push`-triggered builds are labeled as "Pull request build", and the `issue_comment` ones as "Manual build" (because they are triggered using a Personal Access Token, also known as "PAT").
15-
16-
Depending how the Azure Pipeline was triggered, it calls [`misc-helper.ts`](https://github.com/gitgitgadget/gitgitgadget/blob/master/script/misc-helper.ts) with the `handle-pr-comment` or with the `handle-pr-push` parameter, respectively. More precisely, the pipeline has 4 steps:
17-
18-
1. Use Node 10.x
19-
2. Pull GitGitGadget, run npm, essentially:
20-
```sh
21-
test -d .git || git init || exit 1
22-
23-
git rev-parse HEAD >.git/PRE_FETCH_HEAD &&
24-
git fetch https://github.com/gitgitgadget/gitgitgadget master ||
25-
exit 1
26-
27-
[... commented-out dirty tricks to use PRs' patches early ...]
28-
git reset --hard FETCH_HEAD ||
29-
exit 1
30-
31-
# If nothing was pulled, we already built it
32-
test "$(git rev-parse HEAD)" != "$(cat .git/PRE_FETCH_HEAD)" ||
33-
exit 0
34-
35-
npm install &&
36-
npm run build ||
37-
exit 1
38-
```
39-
3. Obtain GitHub Token:
40-
```sh
41-
GIT_CONFIG_PARAMETERS="$GIT_CONFIG_PARAMETERS $EXTRA_CONFIG"
42-
node build/script/misc-helper.js set-app-token &&
43-
git config gitgitgadget.publishRemote \
44-
https://x-access-token:$(git config gitgitgadget.githubToken)@github.com/gitgitgadget/git
45-
```
46-
(The `EXTRA_CONFIG` is necessary because it contains a value that is configured via a secret pipeline variable.)
47-
4. Handle PR (Comment or Push):
48-
```sh
49-
GIT_CONFIG_PARAMETERS="$GIT_CONFIG_PARAMETERS $EXTRA_CONFIG"
50-
set -x
51-
if test "$(pr.comment.id)" -lt 0
52-
then
53-
case "$BUILD_SOURCEBRANCH" in
54-
refs/pull/[1-9]*/head|refs/pull/[1-9]*/merge)
55-
branchname=${BUILD_SOURCEBRANCH#refs/pull/}
56-
prnumber=${branchname%/*}
57-
;;
58-
*) echo "Invalid source branch: $BUILD_SOURCEBRANCH">&2; exit 1;;
59-
esac
60-
node build/script/misc-helper.js handle-pr-push "$prnumber"
61-
else
62-
node build/script/misc-helper.js handle-pr-comment "$(pr.comment.id)"
63-
fi
64-
```
65-
(The `pr.comment.id` pipeline variable is set when queuing the build via the Azure Function that is registered as a webhook, that's how we can discern between this mode vs push.)
66-
67-
The pipeline variable `GIT_CONFIG_PARAMETERS` (which is pre-set as an environment variable in the scripts, interpreted by Git in the same way as `git -c <key>=<value>` would be) is defined as `'user.name=GitGitGadget' '[email protected]' 'gitgitgadget.workDir=$(Agent.HomeDirectory)/../git-worktree'`, i.e. it configures GitGitGadget as committer and it asks GitGitGadget to fetch the commits and notes to the directory`git-worktree/` that is located next to the Azure Pipeline build agent's own directory, which assumes that the agent is running in a suitable VM and its files are installed into the home directory of the account running the agent.
68-
69-
Ideally, this definition (even if it is very small compared to other pipeline definitions I maintain) would be tracked in a Git repository, but since we want this to be a CI build (so that it neatly shows those Checks in the PR page), the pipeline is already associated with gitgitgadget/git (even if the pipeline is configured not to "sync the source", i.e. it does not check out the code pushed to the PR), and we cannot simply add GitGitGadget's Azure Pipelines' definitions to `master` because that is mirrored verbatim from git/git.
14+
Either workflow calls one of GitGitGadget's GitHub Actions, which are very thin wrappers around [the `CIHelper` class](https://github.com/gitgitgadget/gitgitgadget/blob/HEAD/lib/ci-helper.ts) that implements the core logic.
7015

7116
# Mirroring replies from the Git mailing list to GitGitGadget's PRs
7217

73-
This is performed by the [Mirror Git List to GitGitGadget's PRs](https://dev.azure.com/gitgitgadget/git/_build?definitionId=5) pipeline. Its tasks look very similar to the ones of the GitGitGadget PR Handler pipeline described [above](#how-does-gitgitgadget-process-user-comments-on-prs), but it is triggered by updates to the `master` branch of the Git mailing list archive at https://public-inbox.org/git and calls `misc-helper.js handle-new-mails`instead of the `handle-pr-comment`/`handle-pr-push` commands.
18+
This is performed by [the `handle-new-mails` GitHub workflow]https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/handle-pr-push.yml). It calls [the `handle-new-mails` GitHub Action](https://github.com/gitgitgadget/gitgitgadget/tree/HEAD/handle-new-mails), which is a very thin wrapper around [the `CIHelper` class](https://github.com/gitgitgadget/gitgitgadget/blob/HEAD/lib/ci-helper.ts) that implements the core logic. The workflow runs are triggered by updates to the main branch of GitGitGadget's [mirror of the Git mailing list archive](https://github.com/gitgitgadget/git-mailing-list-mirror/), thanks to [GitGitGadget's GitHub App](https://github.com/gitgitgadget/gitgitgadget-github-app) reacting to the corresponding `push` event.
7419

7520
# Update PR labels and mentioning when/where the patches were integrated
7621

77-
The PRs labels are updated, and the comments that document how the corresponding branch is called and when the patches were integrated and into which branch(es), and the PRs are closed when the patches hit `master` and/or `maint` by the [Update GitGitGadget's PRs](https://dev.azure.com/gitgitgadget/git/_build/index?definitionId=2) pipeline. Its tasks look very similar to the GitGitGadget PR Handler pipeline described [above](#how-does-gitgitgadget-process-user-comments-on-prs), but it is triggered by updates to the branches `pu`, `next`, `master` and `maint` at https://github.com/git/git, and it calls `misc-helper.ts update-open-prs`, `misc-helper.ts update-commit-mappings` and `misc-helper.ts handle-open-prs` instead of the `handle-pr-comment`/`handle-pr-push` commands.
22+
The PRs labels are updated, and the comments that document how the corresponding branch is called and when the patches were integrated and into which branch(es), and the PRs are closed when the patches hit `master` and/or `maint` by [the `update-prs` GitHub workflow](https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/update-prs.yml). It calls [the `update-prs` GitHub Action](https://github.com/gitgitgadget/gitgitgadget/tree/HEAD/update-prs), which is a very thin wrapper around [the `CIHelper` class](https://github.com/gitgitgadget/gitgitgadget/blob/HEAD/lib/ci-helper.ts) that implements the core logic. It is triggered by updates to the `seen` branch at https://github.com/git/git, thanks to [GitGitGadget's GitHub App](https://github.com/gitgitgadget/gitgitgadget-github-app) reacting to the corresponding `push` event.
7823

7924
# Keeping https://github.com/gitgitgadget/git up to date
8025

81-
The repository gitgitgadget/git is kept up to date by two Azure Pipelines: [Synchronize gitster.git to GitGitGadget](https://dev.azure.com/gitgitgadget/git/_build?definitionId=8) and [Synchronize git.git to GitGitGadget](https://dev.azure.com/gitgitgadget/git/_build?definitionId=7).
26+
The repository gitgitgadget/git is kept up to date by three GitHub workflows:
27+
28+
- [`sync-ref`](https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/sync-ref.yml), which is triggered by branch updates in https://github.com/git/git (via [GitGitGadget's GitHub App](https://github.com/gitgitgadget/gitgitgadget-github-app), because GitHub Actions does not support cross-repository triggers), handling a single branch per workflow run.
29+
- [`sync-gitster-git`](https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/sync-gitster-git.yml), which is a scheduled workflow, bulk-updating all `maint-*` and `<initials>/<topic>` branches in one go.
30+
- [`sync-git-gui-branches`](https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/sync-git-gui.yml), which is a scheduled workflow, bulk-updating all of Git GUI's branches into the `git-gui/` ref namespace in one go.
8231

8332
## Why synchronize both git/git _and_ gitster/git?
8433

8534
### Background
8635

8736
The existence of the `git/git` -> `gitgitgadget/git` pipeline is probably obvious.
8837

89-
The `gitster/git` repository contains the individual branches for code contributions, and the first reason why the corresponding pipeline was added is that `pu` had test failures all the time without any actionable information: it has been always unclear _which_ of those patch series that made it into `pu` was responsible for the test failures. Now with the individual branches being mirrored into `gitgitgadget/git`, and obviously triggering [CI builds](https://dev.azure.com/gitgitgadget/git/_build?definitionId=4), it is a lot easier to find the culprits.
38+
The `gitster/git` repository contains the individual branches for code contributions, and the first reason why the corresponding pipeline was added is that `seen` had test failures all the time without any actionable information: it has been always unclear _which_ of those patch series that made it into `seen` was responsible for the test failures. Now with the individual branches being mirrored into `gitgitgadget/git`, and obviously triggering [CI builds]https://github.com/gitgitgadget/git/actions/workflows/main.yml), it is a lot easier to find the culprits.
9039

91-
Of course, from time to time the `pu` failures are caused by unfortunate interactions of separate patch series' changes, in which case the CI builds of the individual branches may very well succeed but `pu` still causes failures, which is a very useful piece of information in and of itself.
40+
Of course, from time to time the `seen` failures are caused by unfortunate interactions of separate patch series' changes, in which case the CI builds of the individual branches may very well succeed but `seen` still causes failures, which is a very useful piece of information in and of itself.
9241

9342
A secondary benefit of mirroring the `gitster/git` branches is that PRs at `gitgitgadget/git` can target more fine-grained base commits. I use this a lot, e.g. in my six "built-in add -i/-p" patch series which build on top of each other.
9443

95-
While the pipeline that synchronizes with `gitster/git` is triggered by all updates to `refs/heads/*`, for technical reasons polling every 180 seconds, i.e. every three minutes, the pipeline that synchronizes with `git/git` is triggered immediately.
44+
While the workflow that synchronizes with `gitster/git` is a scheduled workflow that is runs once per day, the workflow that synchronizes with `git/git` is triggered immediately (via `push` events, thanks to a read-only version of GitGitGadget's GitHub App being installed on the latter repository).
9645

9746
### What do the pipelines do, exactly?
9847

0 commit comments

Comments
 (0)