Skip to content

Commit d0e1236

Browse files
danceratopzdanceratopz
authored
refactor(tooling, ci): modularize tox envs for better feedback and parallelization (#1061)
* chore(tooling): update pre-commit config for modular tox envs * refactor(tooling): split tox environments by tool, not scope * style(ci): apply ruff formatting to github helper script * chore(tooling): don't write fill t8n logs in tox * refactor(tooling): renaming & add aliases for previous env names Also fix hard-coded solc_version. * docs(tooling): update docs for new tox environments * chore(tooling): run ruff with explicit python source dirs This avoided a problem that occurred in CI: execution-specs gets cloned within execution-spec-tests (not very clean, but ok for now). This change ensures that this directory is not checked by ruff in `tox -e lint`. * feat(ci): run a gh workflow job per tox environment * fix(tooling): set EELS_RESOLUTION_FILE var for tests-deployed tox envs * chore(ci): shorter/descriptive job and step names * chore(ci): even shorter job name * chore(ci): disable tests-deployed with python 3.10 due to timeout * feat(ci): use astral-sh/setup-uv action and new gh vars The official action allows convenient specification of the python version and supports caching. The setup-uv step now makes use of the new UV_VERSION and DEFAULT_PYTHON_VERSION (where appropriate). * docs(ci): update dev docs for running actions locally
1 parent 154ab82 commit d0e1236

10 files changed

+334
-175
lines changed

‎.github/scripts/get_release_props.py

+4-2
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,17 @@
66
# ]
77
# ///
88
"""Extract the properties of a configured EEST release from a YAML file."""
9+
910
import sys
1011

1112
import click
1213
import yaml
1314

14-
RELEASE_PROPS_FILE = './.github/configs/feature.yaml'
15+
RELEASE_PROPS_FILE = "./.github/configs/feature.yaml"
1516

1617

1718
@click.command()
18-
@click.argument('release', required=True)
19+
@click.argument("release", required=True)
1920
def get_release_props(release):
2021
"""Extract the properties from the YAML file for a given release."""
2122
with open(RELEASE_PROPS_FILE) as f:
@@ -25,5 +26,6 @@ def get_release_props(release):
2526
sys.exit(1)
2627
print("\n".join(f"{key}={value}" for key, value in data[release].items()))
2728

29+
2830
if __name__ == "__main__":
2931
get_release_props()

‎.github/workflows/tox_verify.yaml

+131-46
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,104 @@
1-
name: Run Tox Verifications
1+
name: Tox
22

33
on: [push, pull_request, workflow_dispatch]
44

55
jobs:
6-
build:
6+
7+
lint:
8+
name: Lint python sources with ruff
9+
runs-on: ubuntu-latest
10+
steps:
11+
- name: Checkout ethereum/execution-spec-tests
12+
uses: actions/checkout@v4
13+
- name: Install uv ${{ vars.UV_VERSION }} and python ${{ vars.DEFAULT_PYTHON_VERSION }}
14+
uses: astral-sh/setup-uv@v5
15+
with:
16+
enable-cache: true
17+
cache-dependency-glob: "uv.lock"
18+
version: ${{ vars.UV_VERSION }}
19+
python-version: ${{ vars.DEFAULT_PYTHON_VERSION }}
20+
- name: Run ruff linter via tox
21+
run: uvx --with=tox-uv tox -e lint
22+
23+
typecheck:
24+
name: Typecheck python sources with mypy
25+
runs-on: ubuntu-latest
26+
steps:
27+
- name: Checkout ethereum/execution-spec-tests
28+
uses: actions/checkout@v4
29+
- name: Install uv ${{ vars.UV_VERSION }} and python ${{ vars.DEFAULT_PYTHON_VERSION }}
30+
uses: astral-sh/setup-uv@v5
31+
with:
32+
enable-cache: true
33+
cache-dependency-glob: "uv.lock"
34+
version: ${{ vars.UV_VERSION }}
35+
python-version: ${{ vars.DEFAULT_PYTHON_VERSION }}
36+
- name: Run mypy typechecker via tox
37+
run: uvx --with=tox-uv tox -e typecheck
38+
39+
spellcheck:
40+
name: Spellcheck sources with pyspelling
41+
runs-on: ubuntu-latest
42+
steps:
43+
- name: Checkout ethereum/execution-spec-tests
44+
uses: actions/checkout@v4
45+
- name: Install uv ${{ vars.UV_VERSION }} and python ${{ vars.DEFAULT_PYTHON_VERSION }}
46+
uses: astral-sh/setup-uv@v5
47+
with:
48+
enable-cache: true
49+
cache-dependency-glob: "uv.lock"
50+
version: ${{ vars.UV_VERSION }}
51+
python-version: ${{ vars.DEFAULT_PYTHON_VERSION }}
52+
- name: Install dependencies (aspell, aspell-en)
53+
run: |
54+
sudo apt-get update && sudo apt-get install -y aspell aspell-en
55+
- name: Run spellcheck with pyspelling via tox
56+
run: uvx --with=tox-uv tox -e spellcheck
57+
58+
markdownlint:
59+
name: Lint markdown files with markdownlint
60+
runs-on: ubuntu-latest
61+
steps:
62+
- name: Checkout ethereum/execution-spec-tests
63+
uses: actions/checkout@v4
64+
- uses: DavidAnson/markdownlint-cli2-action@v16
65+
with:
66+
globs: |
67+
README.md
68+
docs/**/*.md
69+
70+
mkdocs:
71+
name: Build html documentation with mkdocs
72+
runs-on: ubuntu-latest
73+
steps:
74+
- name: Checkout ethereum/execution-spec-tests
75+
uses: actions/checkout@v4
76+
- name: Install uv ${{ vars.UV_VERSION }} and python ${{ vars.DEFAULT_PYTHON_VERSION }}
77+
uses: astral-sh/setup-uv@v5
78+
with:
79+
enable-cache: true
80+
cache-dependency-glob: "uv.lock"
81+
version: ${{ vars.UV_VERSION }}
82+
python-version: ${{ vars.DEFAULT_PYTHON_VERSION }}
83+
- name: Build html documentation with mkdocs via tox
84+
run: uvx --with=tox-uv tox -e mkdocs
85+
86+
pytest_framework:
87+
name: Run unit tests, ${{ matrix.os }}, ${{ matrix.python }}
788
runs-on: ${{ matrix.os }}
889
strategy:
990
matrix:
1091
include:
11-
# Temporarily disable
12-
# - os: ubuntu-latest
13-
# python: "3.10"
14-
# evm-type: "stable"
15-
# tox-cmd: "uvx --with=tox-uv tox" # run-parallel --parallel-no-spinner" # TODO: disable parallelisation for uv testing
1692
- os: ubuntu-latest
17-
python: "3.12"
18-
evm-type: "stable"
19-
tox-cmd: "uvx --with=tox-uv tox" # run-parallel --parallel-no-spinner"
20-
# Disabled due to unavailable evm implementation for devnet-1
21-
# - os: ubuntu-latest
22-
# python: '3.11'
23-
# evm-type: 'develop'
24-
# tox-cmd: 'tox -e tests-develop'
25-
# Disabled to not be gated by evmone implementation
26-
# - os: ubuntu-latest
27-
# python: '3.11'
28-
# evm-type: 'eip7692'
29-
# tox-cmd: 'tox -e tests-eip7692'
93+
python: "3.10"
3094
- os: macos-latest
3195
python: "3.11"
32-
evm-type: "stable"
33-
tox-cmd: "uvx --with=tox-uv tox" # run-parallel --parallel-no-spinner"
96+
- os: ubuntu-latest
97+
python: "3.12"
3498
steps:
3599
- name: Checkout ethereum/execution-spec-tests
36100
uses: actions/checkout@v4
37-
- name: Checkout ethereum/execution-specs
101+
- name: Checkout ethereum/execution-specs for local EELS implementation
38102
uses: actions/checkout@v4
39103
with:
40104
repository: ethereum/execution-specs
@@ -43,30 +107,51 @@ jobs:
43107
sparse-checkout: |
44108
src/ethereum
45109
fetch-depth: 1
46-
- uses: ./.github/actions/build-evm-base
47-
id: evm-builder
110+
- name: Install uv ${{ vars.UV_VERSION }} and python ${{ matrix.python }}
111+
uses: astral-sh/setup-uv@v5
48112
with:
49-
type: ${{ matrix.evm-type }}
50-
- name: Set up uv
51-
uses: ./.github/actions/setup-uv
52-
- name: Set up Python ${{ matrix.python }}
53-
run: uv python install ${{ matrix.python }}
54-
- name: Setup Tools/Dependencies Ubuntu
55-
if: runner.os == 'Linux'
56-
run: |
57-
sudo apt-get update && sudo apt-get install -y aspell aspell-en
58-
- name: Setup Tools/Dependencies macOS
59-
if: runner.os == 'macOS'
60-
run: |
61-
brew install aspell
62-
# Add additional packages on 3.11: https://github.com/ethereum/execution-spec-tests/issues/274
63-
if [ ${{ matrix.python }} == '3.11' ]; then brew install autoconf automake libtool; fi
64-
- name: Run Tox (CPython)
113+
enable-cache: true
114+
cache-dependency-glob: "uv.lock"
115+
version: ${{ vars.UV_VERSION }}
116+
python-version: ${{ matrix.python }}
117+
- name: Run tox - run framework unit tests with pytest
65118
env:
66119
EELS_RESOLUTIONS_FILE: ${{ github.workspace }}/.github/configs/eels_resolutions.json
67-
run: ${{ matrix.tox-cmd }}
68-
- uses: DavidAnson/markdownlint-cli2-action@v16
120+
run: uvx --with=tox-uv tox -e pytest
121+
122+
tests_deployed:
123+
name: Fill tests, deployed, ${{ matrix.os }}, ${{ matrix.python }}
124+
runs-on: ${{ matrix.os }}
125+
strategy:
126+
matrix:
127+
include:
128+
# Still getting requests.exceptions.ReadTimeout: UnixHTTPConnectionPool() in CI, even with local EELS
129+
# - os: ubuntu-latest
130+
# python: "3.10"
131+
- os: ubuntu-latest
132+
python: "3.11"
133+
- os: macos-latest
134+
python: "3.12"
135+
steps:
136+
- name: Checkout ethereum/execution-spec-tests
137+
uses: actions/checkout@v4
138+
- name: Checkout ethereum/execution-specs for local EELS implementation
139+
uses: actions/checkout@v4
69140
with:
70-
globs: |
71-
README.md
72-
docs/**/*.md
141+
repository: ethereum/execution-specs
142+
ref: 9b95554a88d2a8485f8180254d0f6a493a593fda
143+
path: execution-specs
144+
sparse-checkout: |
145+
src/ethereum
146+
fetch-depth: 1
147+
- name: Install uv ${{ vars.UV_VERSION }} and python ${{ matrix.python }}
148+
uses: astral-sh/setup-uv@v5
149+
with:
150+
enable-cache: true
151+
cache-dependency-glob: "uv.lock"
152+
version: ${{ vars.UV_VERSION }}
153+
python-version: ${{ matrix.python }}
154+
- name: Run tox - fill tests for deployed forks
155+
env:
156+
EELS_RESOLUTIONS_FILE: ${{ github.workspace }}/.github/configs/eels_resolutions.json
157+
run: uvx --with=tox-uv tox -e tests-deployed

‎.gitignore

+4-1
Original file line numberDiff line numberDiff line change
@@ -71,4 +71,7 @@ assets
7171
env.yaml
7272

7373
# Framework logs
74-
logs/
74+
logs/
75+
76+
# Running github actions locally with act
77+
.act_github_vars

‎.pre-commit-config.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@ repos:
33
hooks:
44
- id: tox
55
name: tox
6-
entry: tox run-parallel
6+
entry: uvx --with=tox-uv tox --parallel -e lint
77
language: system
88
types: [python]
99
pass_filenames: false
1010
- id: tox-docs
1111
name: tox-docs
12-
entry: tox -e docs
12+
entry: uvx --with=tox-uv tox --parallel -e spellcheck,markdownlint,mkdocs
1313
language: system
1414
files: \.md$
15-
pass_filenames: false
15+
pass_filenames: false

‎docs/dev/docs.md

+5-6
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,18 @@ The `execution-spec-tests` documentation is generated via [`mkdocs`](https://www
88
uv sync --all-extras
99
```
1010

11-
## Build the Documentation
11+
## Build and Verify the Documentation
1212

13-
One time build:
13+
One time build in strict mode:
1414

1515
```console
16-
uv run mkdocs build
16+
uv run mkdocs build --strict
1717
```
1818

19-
Do a pre-commit check: One time build and lint/type checking:
19+
Perform all docs related checks via `tox` in parallel:
2020

2121
```console
22-
pip install tox-uv
23-
tox -e docs
22+
uvx --with=tox-uv tox -e spellcheck,markdownlint,mkdocs --parallel
2423
```
2524

2625
### Local Deployment and Test

‎docs/dev/test_actions_locally.md

+37-14
Original file line numberDiff line numberDiff line change
@@ -43,26 +43,49 @@ gh act --list
4343
will output something similar to:
4444

4545
```bash
46-
INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'
47-
Stage Job ID Job name Workflow name Workflow file Events
48-
0 evmone-coverage-diff evmone-coverage-diff Evmone Coverage Report coverage.yaml pull_request
49-
0 deploy deploy Deploy Docs Main docs_main.yaml push
50-
0 deploy deploy Deploy Docs Tags docs_tags.yaml push
51-
0 features features Build and Package Fixtures fixtures.yaml push,workflow_dispatch
52-
0 feature-names feature-names Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch
53-
0 build build Run Tox Verifications tox_verify.yaml push,pull_request,workflow_dispatch
54-
1 build build Build and Package Fixtures fixtures.yaml push,workflow_dispatch
55-
1 build build Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch
56-
2 release release Build and Package Fixtures fixtures.yaml push,workflow_dispatch
57-
2 release release Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch
46+
INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'
47+
Stage Job ID Job name Workflow name Workflow file Events
48+
0 evmone-coverage-diff evmone-coverage-diff Evmone Coverage Report coverage.yaml pull_request
49+
0 deploy deploy Deploy Docs Main docs_main.yaml push
50+
0 deploy deploy Deploy Docs Tags docs_tags.yaml push
51+
0 features features Build and Package Fixtures fixtures.yaml push,workflow_dispatch
52+
0 feature-names feature-names Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch
53+
0 lint Lint python sources with ruff Tox tox_verify.yaml push,pull_request,workflow_dispatch
54+
0 typecheck Typecheck python sources with mypy Tox tox_verify.yaml push,pull_request,workflow_dispatch
55+
0 spellcheck Spellcheck sources with pyspelling Tox tox_verify.yaml push,pull_request,workflow_dispatch
56+
0 markdownlint Lint markdown files with markdownlint Tox tox_verify.yaml push,pull_request,workflow_dispatch
57+
0 mkdocs Build html documentation with mkdocs Tox tox_verify.yaml push,pull_request,workflow_dispatch
58+
0 pytest_framework Run unit tests, ${{ matrix.os }}, ${{ matrix.python }} Tox tox_verify.yaml push,pull_request,workflow_dispatch
59+
0 tests_deployed Fill tests, deployed, ${{ matrix.os }}, ${{ matrix.python }} Tox tox_verify.yaml push,pull_request,workflow_dispatch
60+
1 build build Build and Package Fixtures fixtures.yaml push,workflow_dispatch
61+
1 build build Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch
62+
2 release release Build and Package Fixtures fixtures.yaml push,workflow_dispatch
63+
2 release release Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch
5864
```
5965

6066
The `Job ID` is required to run a specific workflow and is provided to the `-j` option of `gh act`.
6167

62-
### Running Workflows Locally that use a Matrix Strategy
68+
### Running Workflows that require Github Vars
69+
70+
Create a text file containing the required variables and variables, e.g., `.act_github_vars` (this is in `.gitignore`):
71+
72+
```text
73+
UV_VERSION=0.5.15
74+
DEFAULT_PYTHON_VERSION=3.12
75+
```
76+
77+
and use the `--var-file` option to specify the file:
78+
79+
```bash
80+
gh act --workflows .github/workflows/tox_verify.yaml -s GITHUB_TOKEN=$(gh auth token) --var-file=gh_vars.txt -j lint
81+
```
82+
83+
### Running Workflows that use a Matrix Strategy
84+
85+
This is optional, recent versions will automatically detect the matrix strategy and run supported values. To run a specific matrix value, use the `--matrix` option:
6386

6487
```bash
65-
gh act -j build --workflows .github/workflows/tox_verify.yaml -s GITHUB_TOKEN=$(gh auth token) --matrix python:3.12
88+
gh act --workflows .github/workflows/tox_verify.yaml -s GITHUB_TOKEN=$(gh auth token) --matrix python:3.12 -j pytest_framework
6689
```
6790

6891
### Running Release Workflows

‎docs/writing_tests/test_markers.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -352,7 +352,7 @@ In this example, the test will be marked as expected to fail when it is being ex
352352

353353
### `@pytest.mark.slow`
354354

355-
This marker is used to mark tests that are slow to run. These tests are not run during tox testing, and are only run when a release is being prepared.
355+
This marker is used to mark tests that are slow to run. These tests are not run during [`tox` checks](./verifying_changes.md), and are only run when a release is being prepared.
356356

357357
### `@pytest.mark.pre_alloc_modify`
358358

0 commit comments

Comments
 (0)