Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit 863d67f

Browse files
KamilcukKamilcuk
committedOct 22, 2024·
githubrunner: add documentation and fix template arguments
1 parent 18430bb commit 863d67f

File tree

4 files changed

+295
-131
lines changed

4 files changed

+295
-131
lines changed
 

‎doc/githubruner.md

+132
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# githubrunner
2+
3+
<!-- vim-markdown-toc GFM -->
4+
5+
* [What it does?](#what-it-does)
6+
* [How it works?](#how-it-works)
7+
* [How start the scheduler?](#how-start-the-scheduler)
8+
* [Nomad job template](#nomad-job-template)
9+
* [How to use it from Github workflow?](#how-to-use-it-from-github-workflow)
10+
* [Example workflow:](#example-workflow)
11+
12+
<!-- vim-markdown-toc -->
13+
14+
# What it does?
15+
16+
Runs a specific Nomad job matching the labels requested by pending GitHub actions jobs.
17+
Controls the number of Nomad jobs depending on the number of pending GithHub actions jobs.
18+
19+
# How it works?
20+
21+
- Every CONFIG.pool seconds
22+
- Get all actions workflow runs with the following algorithm:
23+
- For each entry configured CONFIG.repos
24+
- If the repository has no /
25+
- Get all repositories under this organization or user
26+
- else use the repository
27+
- For each such repository
28+
- For each repository actions workflow
29+
- For each repository actions workflow run
30+
- If the workflow run has status queued or in_progress
31+
- Add it to the list
32+
- Get all Nomad jobs with the following algorithm:
33+
- For all Nomad jobs in CONFIG.nomad.namespace
34+
- If the job name starts with CONFIG.nomad.jobprefix + "-"
35+
- If the `job["Meta"][CONFIG.nomad.meta]` is a valid JSON
36+
- Add it to the list
37+
- Group Github workflow runs and Nomad runners in groups indexed by repository url and runner labels
38+
- For each such group:
39+
- If there are more Github workflow runs than non-dead Nomad runners:
40+
- If there are less non-dead Nomad runners than CONFIG.max_runners
41+
- Generate Nomad job specification from Jinja2 template.
42+
- Start the job
43+
- If there are less Github workflow runs than non-dead Nomad runners:
44+
- Stop a Nomad job
45+
- For each dead Nomad job:
46+
- purge it when associated timeout if reached
47+
48+
# How start the scheduler?
49+
50+
Create a file `config.yml` with the following content:
51+
52+
---
53+
github:
54+
token: the_access_token_or_set_GH_TOKEN_env_variable
55+
repos:
56+
# for single repos
57+
- user/repo1
58+
- user/repo2
59+
# for all repositories of this user
60+
- user
61+
62+
Run the command line:
63+
64+
nomadtools githubrunner -c ./config.yml run
65+
66+
The default configuration can be listed with `nomadtools githubrunner -c $'{}\n' dumpconfig`.
67+
68+
The configuration description is in source code.
69+
70+
# Nomad job template
71+
72+
The following variables are available in the Jinja2 template:
73+
74+
- CONFIG
75+
- The whole parsed YAML configuration.
76+
- RUN
77+
- Dictionary with important generated values.
78+
- RUN.RUNNER_NAME
79+
- The Nomad job name and runner name, to be consistent.
80+
- RUN.REPO_URL
81+
- The repository the runner should connect to.
82+
- RUN.LABELS
83+
- Comma separated list of labels the runner should register to.
84+
- RUN.REPO
85+
- Like REPO_URL but without domain
86+
- RUN.INFO
87+
- Generated string with some information about the state of scheduler at the time the runner was requested to run.
88+
- RUNSON
89+
- The `runson:` labels in the workflow are parsed with `shlex.split` `k=v` values
90+
- For example, `nomadtools mem=1000` results in RUNSON={"nomadtools": "", "mem": "1000"}.
91+
- The values used by the default template:
92+
- RUNSON.tag
93+
- docker image tag
94+
- RUNSON.cpu
95+
- RUNSON.mem
96+
- RUNSON.maxmem
97+
- SETTINGS
98+
- Dictionary composed of, in order:
99+
- CONFIG.template_default_settings
100+
- CONFIG.template_settings
101+
- nomadlib.escape
102+
- Replace `${` by `$${` and `%{` by `%%{` . For embedding string in template.
103+
104+
# How to use it from Github workflow?
105+
106+
jobs:
107+
build:
108+
runs_on: nomadtools mem=16000
109+
110+
would template the job with `RUNSON={"nomadtools": "", "mem": "16000"}` .
111+
112+
The value of `RUNSON.mem` is then used as the memory settings in the Jinja2 template.
113+
114+
## Multiple elements in runs_on
115+
116+
I do not know what will happen with multiple elements in runs_on, like:
117+
118+
runs_no:
119+
- self-hosted
120+
- nomadtools mem=16000
121+
122+
## Comma in labels
123+
124+
The config.sh github runner configuration script takes comma separated list of elements.
125+
126+
So comma will split the labels into multiple.
127+
128+
Do not use comma in `runs_on:`.
129+
130+
# Example workflow:
131+
132+
https://github.com/Kamilcuk/runnertest/tree/main/.github/workflows

‎src/nomad_tools/entry_githubrunner/default.nomad.hcl

+82-63
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,34 @@
1-
job "{{ param.JOB_NAME }}" {
2-
{{ param.extra_job }}
1+
locals {
2+
INFO = <<EOFEOF
3+
This is a default runner shipped with nomadtools based on myoung34/github-runner image.
34
4-
type = "batch"
5-
meta {
6-
INFO = <<EOF
7-
This is a runner based on {{ image }} image.
8-
{% if param.docker == "dind" %}
9-
It also starts a docker daemon and is running as privileged
10-
{% elif param.docker == "host" %}
11-
It also mounts a docker daemon from the host it is running on
12-
{% endif %}
5+
User requested labels were parsed to the following:
6+
{{ nomadlib.escape(RUNSON | tojson) }}
137
14-
EOF
8+
The following parameters were generated for this job:
9+
{{ nomadlib.escape(RUN | tojson) }}
1510
16-
{% if param.debug %}
17-
PARAM = <<EOF
18-
{{param | tojson}}
19-
EOF
11+
Job is running with the following settings:
12+
{{ nomadlib.escape(SETTINGS | tojson) }}
13+
14+
{% if SETTINGS.docker == "dind" %}
15+
The container runs with --privileged and starts a docker-in-docker instance.
16+
{% elif SETTINGS.docker == "host" %}
17+
The container mounts the /var/run/docker.sock from the host.
2018
{% endif %}
21-
}
22-
group "{{ param.JOB_NAME }}" {
23-
{{ param.extra_group }}
19+
{% if RUN.nodocker is defined %}
20+
The user requested to run without docker.
21+
{% endif %}
22+
23+
EOFEOF
24+
}
25+
26+
job "{{ RUN.RUNNER_NAME }}" {
27+
{{ SETTINGS.extra_job }}
28+
type = "batch"
29+
30+
group "{{ RUN.RUNNER_NAME }}" {
31+
{{ SETTINGS.extra_group }}
2432

2533
reschedule {
2634
attempts = 0
@@ -31,80 +39,91 @@ EOF
3139
mode = "fail"
3240
}
3341

34-
task "{{ param.JOB_NAME }}" {
35-
{{ param.extra_task }}
42+
task "{{ RUN.RUNNER_NAME }}" {
43+
{{ SETTINGS.extra_task }}
3644

3745
driver = "docker"
3846
kill_timeout = "5m"
3947
config {
40-
{{ param.extra_config }}
48+
{{ SETTINGS.extra_config }}
4149

42-
image = "{{ param.image | default('myoung34/github-runner:latest') }}"
50+
image = "myoung34/github-runner:{{ RUNSON.tag | default('latest') }}"
4351
init = true
44-
entrypoint = ["bash", "/local/nomadtools_startscript.sh"]
52+
{% if SETTINGS.entrypoint %}
53+
entrypoint = ["${NOMAD_TASK_DIR}/nomadtools_entrypoint.sh"]
54+
{% endif %}
4555

46-
{% if param.cachedir %}
56+
{% if SETTINGS.cachedir %}
4757
mount {
4858
type = "bind"
49-
source = "{{ param.cachedir }}"
59+
source = "{{ SETTINGS.cachedir }}"
5060
target = "/_work"
5161
readonly = false
5262
}
53-
{% endif %}
63+
{% endif %}
5464

55-
{% if param.docker == "dind" %}
65+
{% if not RUNSON.nodocker %}
66+
{% if SETTINGS.docker == "dind" %}
5667
privileged = true
57-
{% elif param.docker == "host" %}
68+
{% elif SETTINGS.docker == "host" %}
5869
mount {
5970
type = "bind"
6071
source = "/var/run/docker.sock"
6172
target = "/var/run/docker.sock"
6273
}
63-
{% endif %}
74+
{% endif %}
75+
{% endif %}
76+
6477
}
78+
6579
env {
66-
ACCESS_TOKEN = "{{ CONFIG.github.token }}"
67-
REPO_URL = "{{ param.REPO_URL }}"
68-
RUNNER_NAME = "{{ param.JOB_NAME }}"
69-
LABELS = "{{ param.LABELS }}"
70-
RUNNER_SCOPE = "repo"
71-
DISABLE_AUTO_UPDATE = "true"
72-
{% if not param.RUN_AS_ROOT %}
73-
RUN_AS_ROOT = "false"
74-
{% endif %}
75-
{% if param.ephemeral %}
76-
EPHEMERAL = "true"
77-
{% endif %}
78-
{% if param.docker == "dind" %}
80+
ACCESS_TOKEN = "{{ SETTINGS.access_token or CONFIG.github.token }}"
81+
REPO_URL = "{{ RUN.REPO_URL }}"
82+
RUNNER_NAME = "{{ RUN.RUNNER_NAME }}"
83+
RUNSON = "{{ RUN.LABELS }}"
84+
RUNNER_SCOPE = "repo"
85+
DISABLE_AUTO_UPDATE = "true"
86+
{% if not SETTINGS.run_as_root %}
87+
RUN_AS_ROOT = "false"
88+
{% endif %}
89+
{% if SETTINGS.ephemeral %}
90+
EPHEMERAL = "true"
91+
{% endif %}
92+
{% if not RUNSON.nodocker %}
93+
{% if SETTINGS.docker == "dind" %}
7994
START_DOCKER_SERVICE = "true"
80-
{% endif %}
81-
{% if param.debug %}
82-
DEBUG_OUTPUT = "true"
83-
{% endif %}
95+
{% endif %}
96+
{% endif %}
97+
{% if SETTINGS.debug %}
98+
DEBUG_OUTPUT = "true"
99+
{% endif %}
84100
}
85-
{% if param.cpu or param.mem or param.maxmem %}
101+
86102
resources {
87-
{% if param.cpu %}
88-
cpu = {{ param.cpu }}
89-
{% endif %}
90-
{% if param.mem %}
91-
memory = {{ param.mem }}
92-
{% endif %}
93-
{% if param.maxmem %}
94-
memory_max = {{ param.maxmem }}
95-
{% endif %}
103+
{% if RUNSON.cpu %}
104+
cpu = {{ RUNSON.cpu }}
105+
{% endif %}
106+
{% if RUNSON.mem %}
107+
memory = {{ RUNSON.mem }}
108+
{% endif %}
109+
{% if RUNSON.maxmem %}
110+
memory_max = {{ RUNSON.maxmem }}
111+
{% endif %}
96112
}
97-
{% endif %}
113+
114+
{% if SETTINGS.entrypoint %}
98115
template {
99-
destination = "local/nomadtools_startscript.sh"
116+
destination = "local/nomadtools_entrypoint.sh"
100117
change_mode = "noop"
101118
left_delimiter = "QWEQWEQEWQEEQ"
102119
right_delimiter = "ASDASDADSADSA"
103-
data = <<EOF
104-
{% if not param.startscript %}{{ 1/0 }}{% endif %}
105-
{{ escape(param.startscript) }}
106-
EOF
120+
perms = "755"
121+
data = <<EOFEOF
122+
{{ nomadlib.escape(SETTINGS.entrypoint) }}
123+
EOFEOF
107124
}
125+
{% endif %}
126+
108127
}
109128
}
110129
}

‎src/nomad_tools/entry_githubrunner/entry_githubrunner.py

+81-68
Original file line numberDiff line numberDiff line change
@@ -178,11 +178,12 @@ def github_repo_from_url(repo: str):
178178
return user.capitalize() + "/" + repo.capitalize()
179179

180180

181-
def labelstr_to_kv(labelstr: str) -> Dict[str, str]:
181+
def labels_to_kv(labels: List[str]) -> Dict[str, str]:
182182
arg = {}
183-
for part in shlex.split(labelstr):
184-
split = part.split("=", 1)
185-
arg[split[0]] = split[1] if len(split) == 2 else ""
183+
for label in labels:
184+
for part in shlex.split(label):
185+
split = part.split("=", 1)
186+
arg[split[0]] = split[1] if len(split) == 2 else ""
186187
return arg
187188

188189

@@ -249,27 +250,30 @@ class Config(pydantic.BaseModel, extra=pydantic.Extra.forbid):
249250
A star '*' causes to get all the repositories available.
250251
"""
251252

252-
label_match: str = "nomadtools *(.*)"
253-
"""Only execute if github action workflow job runs-on labels matching this regex"""
253+
runson_match: str = "nomadtools *(.*)"
254+
""" Only execute if github action workflow job runs-on labels matching this regex. """
254255

255256
loop: int = 60
256-
"""How many seconds will the loop run"""
257+
""" How many seconds will the loop run. """
257258

258259
template: str = get_package_file("entry_githubrunner/default.nomad.hcl")
259260
"""Jinja2 template for the runner"""
260261

261-
default_template_context: Any = {
262+
template_default_settings: Dict[str, Any] = {
262263
"extra_config": "",
263264
"extra_group": "",
264265
"extra_task": "",
265266
"extra_job": "",
267+
"docker": "none",
266268
"ephemeral": True,
267-
"startscript": get_package_file("entry_githubrunner/startscript.sh"),
269+
"run_as_root": False,
270+
"cachedir": "",
271+
"entrypoint": get_package_file("entry_githubrunner/entrypoint.sh"),
268272
}
269-
"""Additional template variables"""
273+
"""Default values for SETTINGS template variable"""
270274

271-
template_context: Any = {}
272-
"""Additional template variables"""
275+
template_settings: Dict[str, Any] = {}
276+
""" Overwrite settings according to your whim """
273277

274278
runner_inactivity_timeout: str = "12h"
275279
"""How much time a runner will be inactive for it to be removed?"""
@@ -287,7 +291,7 @@ def __post_init__(self):
287291

288292
class ParsedConfig:
289293
def __init__(self, config: Config):
290-
self.label_match = re.compile(config.label_match)
294+
self.label_match = re.compile(config.runson_match)
291295
self.runner_inactivity_timeout = parse_time(config.runner_inactivity_timeout)
292296
self.purge_successfull_timeout = parse_time(config.purge_successfull_timeout)
293297
self.purge_failure_timeout = parse_time(config.purge_failure_timeout)
@@ -397,14 +401,14 @@ def print(self):
397401

398402
@dataclass(frozen=True)
399403
class Desc:
400-
labels: str
404+
labels: List[str]
401405
repo_url: str
402406

403407
def __post_init__(self):
404408
assert self.labels
405409
assert self.repo_url
406410
assert "://" in self.repo_url
407-
assert PARSEDCONFIG.label_match.match(self.labels)
411+
assert any(PARSEDCONFIG.label_match.match(x) for x in self.labels)
408412

409413
@property
410414
def repo(self):
@@ -413,6 +417,10 @@ def repo(self):
413417
def tostr(self):
414418
return f"labels={self.labels} url={self.repo_url}"
415419

420+
@property
421+
def labelsstr(self):
422+
return ",".join(sorted(self.labels))
423+
416424

417425
class DescProtocol(ABC):
418426
@abstractmethod
@@ -602,17 +610,8 @@ class GithubJob(DescProtocol):
602610
run: dict
603611
job: dict
604612

605-
def labelsstr(self):
606-
return "\n".join(sorted(list(set(self.job["labels"]))))
607-
608-
def job_url(self):
609-
return self.job["html_url"]
610-
611-
def repo_url(self):
612-
return self.run["repository"]["html_url"]
613-
614613
def get_desc(self):
615-
return Desc(self.labelsstr(), self.repo_url())
614+
return Desc(self.job["labels"], self.job["html_url"])
616615

617616

618617
def get_gh_repos_one_to_many(repo: str) -> List[GithubRepo]:
@@ -681,7 +680,7 @@ def get_gh_state(repos: Set[GithubRepo]) -> list[GithubJob]:
681680
# desc: str = ", ".join(s.labelsstr() + " for " + s.job_url() for s in reqstate)
682681
# log.info(f"Found {len(reqstate)} required runners to run: {desc}")
683682
for idx, s in enumerate(reqstate):
684-
logging.info(f"GHJOB={idx} {s.job_url()} {s.labelsstr()} {s.run['status']}")
683+
logging.info(f"GHJOB={idx} {s.get_desc().tostr()} {s.run['status']}")
685684
return reqstate
686685

687686

@@ -690,7 +689,13 @@ def get_gh_state(repos: Set[GithubRepo]) -> list[GithubJob]:
690689

691690

692691
def get_desc_from_nomadjob(job):
693-
return Desc(**json.loads((job["Meta"] or {})[CONFIG.nomad.meta]))
692+
txt = (job["Meta"] or {})[CONFIG.nomad.meta]
693+
try:
694+
data = json.loads(txt)
695+
except Exception:
696+
print(txt)
697+
raise
698+
return Desc(**data)
694699

695700

696701
@dataclass
@@ -914,7 +919,7 @@ def _generate_job_name(self):
914919
CONFIG.nomad.jobprefix,
915920
self.desc.repo,
916921
i,
917-
self.desc.labels.replace("nomadtools", ""),
922+
self.desc.labelsstr.replace("nomadtools", ""),
918923
self.info,
919924
]
920925
name = re.sub(r"[^a-zA-Z0-9_.-]", "", "-".join(str(x) for x in parts))
@@ -924,35 +929,29 @@ def _generate_job_name(self):
924929
return name
925930

926931
@staticmethod
927-
def make_example(labelsstr: str):
932+
def make_example(labels: List[str]):
928933
return RunnerGenerator(
929-
Desc(labels=labelsstr, repo_url="http://example.repo.url/user/repo"),
934+
Desc(labels=labels, repo_url="http://example.repo.url/user/repo"),
930935
"This is an information",
931936
TakenNames(),
932937
)
933938

934939
def _to_template_args(self) -> dict:
935-
arg = labelstr_to_kv(self.desc.labels)
936-
name = self._generate_job_name()
937940
return dict(
938-
param={
939-
**arg,
940-
**CONFIG.template_context,
941-
**CONFIG.default_template_context,
942-
**dict(
943-
REPO_URL=self.desc.repo_url,
944-
RUNNER_NAME=name,
945-
JOB_NAME=name,
946-
LABELS=self.desc.labels,
947-
),
941+
SETTINGS={
942+
**CONFIG.template_default_settings,
943+
**CONFIG.template_settings,
948944
},
949-
run=asdict(self),
950-
arg=arg,
945+
RUN=dict(
946+
REPO_URL=self.desc.repo_url,
947+
RUNNER_NAME=self._generate_job_name(),
948+
LABELS=self.desc.labelsstr.replace('"', r"\""),
949+
REPO=self.desc.repo,
950+
INFO=self.info,
951+
),
952+
RUNSON=labels_to_kv(self.desc.labels),
951953
CONFIG=CONFIG,
952-
TEMPLATE=TEMPLATE,
953-
ARGS=ARGS,
954-
escape=nomadlib.escape,
955-
META=json.dumps(asdict(self.desc)),
954+
nomadlib=nomadlib,
956955
)
957956

958957
def get_runnertorun(self) -> NomadJobToRun:
@@ -961,10 +960,9 @@ def get_runnertorun(self) -> NomadJobToRun:
961960
jobspec: dict = nomad_job_text_to_json(jobtext)
962961
# Apply default transformations.
963962
jobspec["Namespace"] = CONFIG.nomad.namespace
964-
for key in ["ID", "Name"]:
965-
if key in jobspec:
966-
jobspec[key] = tc["param"]["JOB_NAME"]
967-
jobspec["Meta"][CONFIG.nomad.meta] = tc["META"]
963+
jobspec.setdefault("Meta", {})[CONFIG.nomad.meta] = json.dumps(
964+
asdict(self.desc)
965+
)
968966
#
969967
nomadjobtorun = NomadJobToRun(nomadlib.Job(jobspec), jobtext)
970968
assert nomadjobtorun.get_desc() == self.desc
@@ -1089,7 +1087,7 @@ def loop():
10891087
class Args:
10901088
dryrun: bool = clickdc.option("-n")
10911089
parallel: int = clickdc.option("-P", default=4)
1092-
config: str = clickdc.option(
1090+
config: Optional[str] = clickdc.option(
10931091
"-c",
10941092
shell_complete=click.Path(
10951093
exists=True, dir_okay=False, path_type=Path
@@ -1101,7 +1099,15 @@ class Args:
11011099
)
11021100

11031101

1104-
@click.command("githubrunner", cls=AliasedGroup)
1102+
@click.command(
1103+
"githubrunner",
1104+
cls=AliasedGroup,
1105+
help="""
1106+
Execute Nomad job to run github self-hosted runner for user repositories.
1107+
1108+
See documentation in doc/githubrunner.md on github.
1109+
""",
1110+
)
11051111
@clickdc.adddc("args", Args)
11061112
@help_h_option()
11071113
@common_click.quiet_option()
@@ -1115,14 +1121,17 @@ def cli(args: Args):
11151121
datefmt="%Y-%m-%dT%H:%M:%S%z",
11161122
)
11171123
#
1118-
if "\n" in args.config:
1119-
configstr = args.config
1120-
else:
1121-
with open(args.config) as f:
1122-
configstr = f.read()
1123-
tmp = yaml.safe_load(configstr)
11241124
global CONFIG
1125-
CONFIG = Config(**tmp)
1125+
if args.config:
1126+
if "\n" in args.config:
1127+
configstr = args.config
1128+
else:
1129+
with open(args.config) as f:
1130+
configstr = f.read()
1131+
tmp = yaml.safe_load(configstr)
1132+
CONFIG = Config(**tmp)
1133+
else:
1134+
CONFIG = Config()
11261135
global PARSEDCONFIG
11271136
PARSEDCONFIG = ParsedConfig(CONFIG)
11281137
global GH
@@ -1200,12 +1209,16 @@ def once():
12001209
loop()
12011210

12021211

1203-
@cli.command()
1204-
@click.argument("label", required=True, nargs=-1)
1205-
def rendertemplate(label: Tuple[str, ...]):
1206-
labelsstr: str = ",".join(sorted(list(label)))
1207-
res = RunnerGenerator.make_example(labelsstr).get_runnertorun()
1208-
print(res)
1212+
@cli.command(
1213+
help="""
1214+
Render the template from configuration given the labels given on command line.
1215+
Usefull for testing the job.
1216+
"""
1217+
)
1218+
@click.argument("labels", required=True, nargs=-1)
1219+
def rendertemplate(labels: Tuple[str, ...]):
1220+
res = RunnerGenerator.make_example(list(labels)).get_runnertorun()
1221+
print(res.hcl)
12091222

12101223

12111224
@cli.command(help="Main entrypoint - run the loop periodically")
@@ -1237,7 +1250,7 @@ def listrunners():
12371250
x.ID,
12381251
x.state.state.name,
12391252
x.state.since.isoformat() if x.state.since else "None",
1240-
x.get_desc().labels,
1253+
x.get_desc().labelsstr,
12411254
x.get_desc().repo,
12421255
]
12431256
)

0 commit comments

Comments
 (0)
Please sign in to comment.