Skip to content

Commit 84c494c

Browse files
SS-JIApytorchbot
SS-JIA
authored and
pytorchbot
committed
[ET-VK][tools] Write etvk_ops_bench JSON files compactly
Pull Request resolved: #24806 ## Ulterior Motive Keeping `etvk_ops_bench` JSON files reviewable. `convolution/conv2d/cases/correctness.json` is 8,445 lines, mostly whitespace, and the conv2d epilogue correctness cases would grow it further. ## Rationale **What**: - Adds `format_json.py` to `benchmark_v2`. It keeps a JSON value on one line when its one-line form, not counting its key and indentation, fits in 200 characters, and lists longer objects and arrays one entry per line. An object with one entry is measured by that entry's value, so a `{"tensor": {...}}` wrapper never splits on its own. In operator cases, each case key and each operator argument gets one line. - Reformats every JSON file under `benchmark_v2` with it: the 64 operator case files and the 13 examples. - Adds an `ETVKJSONFORMAT` lintrunner linter for these files. `lintrunner` reports unformatted or invalid files, and `lintrunner -a` formats them. - Replaces `fb/scripts/apply_prettier.sh`, whose Prettier layout would expand the files again. Both READMEs are updated. **Why**: The previous layout put every tensor field and every scalar wrapper on its own line, so one 17-argument set took about 95 lines. The JSON files shrink from 87,631 to 23,727 lines with the same content; conv2d `correctness.json` goes from 8,445 to 2,214. ## Details - Content is unchanged: every file parses to the same values, types and key order as before. Besides whitespace, 4 files re-spell float exponents (`e38` → `e+38`, `e-7` → `e-07`), which parse to the same values. - **Lint**: lintrunner is a required check on GitHub PRs (the `lintrunner` job in `lint.yml`, which lints the PR's changed files) and the lint developers run before submitting. The linter runs `format_json.py --lintrunner`, which prints one lintrunner message per file instead of writing it: - unformatted files get a warning with the formatted contents as the replacement; - files that cannot be formatted get an error. - **Rejected files**: invalid UTF-8, duplicate keys, `NaN` / `Infinity` values, numbers outside float range (such as `1e400`), and strings with no UTF-8 encoding (lone `\ud800` escapes). Without `--lintrunner`, the script exits with the file and the error, before writing that file. - The script writes UTF-8 with `\n` line endings. - Example operator argument set: ```json "args_sweep": [ [ {"tensor": {"name": "input", "sizes": [1, 8, 8, 8], "dtype": "float32", "storage": "texture3d", "layout": "channels", "data": {"rand_uniform_float": [-1, 1]}}}, {"tensor": {"name": "weight", "sizes": [8, 8, 3, 3], "dtype": "float32", "storage": "constant", "layout": "channels", "data": {"rand_uniform_float": [-1, 1]}}}, null, {"int_list": [1, 1]}, ... ] ] ``` - Longer values, such as long `v_SIZES` sweep lists, list one entry per line. - In-flight diffs that edit these files will conflict. Reformatting their side with `format_json.py` before resolving leaves only the real edits in the conflict. Line blame for the JSON files points at this commit. Usage, from the ExecuTorch root: ```sh lintrunner # reports unformatted JSON files among the changed files lintrunner -a # formats them python3 backends/vulkan/test/ops/benchmark_v2/format_json.py # every JSON file under benchmark_v2 python3 backends/vulkan/test/ops/benchmark_v2/format_json.py PATH [...] # given files, or JSON files under given directories ``` ghstack-source-id: 469036179 @exported-using-ghexport Differential Revision: [D130231969](https://our.internmc.facebook.com/intern/diff/D130231969/)
1 parent a168321 commit 84c494c

79 files changed

Lines changed: 11427 additions & 75148 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.lintrunner.toml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -692,6 +692,19 @@ command = [
692692
'@{{PATHSFILE}}',
693693
]
694694

695+
[[linter]]
696+
code = 'ETVKJSONFORMAT'
697+
include_patterns = [
698+
'backends/vulkan/test/ops/benchmark_v2/**/*.json',
699+
]
700+
command = [
701+
'python',
702+
'backends/vulkan/test/ops/benchmark_v2/format_json.py',
703+
'--lintrunner',
704+
'@{{PATHSFILE}}',
705+
]
706+
is_formatter = true
707+
695708
[[linter]]
696709
code = 'RUFF_DOCS'
697710
include_patterns = [

‎backends/vulkan/test/ops/benchmark_v2/README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,30 @@ framework/ Parsing, graph setup, validation, timing, and reportin
2525
operators/ CPU references, routes, throughput counters, and checks
2626
<family>/.../cases/ JSON case collections by operator and quantization scheme
2727
examples/ Small shader and operator configuration examples
28+
format_json.py Formatter for the JSON files
2829
```
2930

3031
Each expanded case runs separately. Shader variants can share one binding
3132
interface through `shader_sweep`; operators can sweep argument sets, dtype and
3233
storage variables, and execution routes.
3334

35+
## Formatting JSON files
36+
37+
`format_json.py` keeps a JSON value on one line when its one-line form is at
38+
most 200 characters, not counting its key and indentation, and lists longer
39+
objects and arrays one entry per line. An object with one entry is measured by
40+
that entry's value. Parsed content is unchanged, and files with duplicate keys
41+
are rejected.
42+
43+
The `ETVKJSONFORMAT` lintrunner linter checks these files, and `lintrunner -a`
44+
formats them. To format without lintrunner, run the script; without paths, it
45+
formats every JSON file under `benchmark_v2`. Other formatters, such as
46+
Prettier on save, expand these files; format them again afterwards.
47+
48+
```sh
49+
python3 backends/vulkan/test/ops/benchmark_v2/format_json.py [PATH ...]
50+
```
51+
3452
## Building with CMake
3553

3654
From the ExecuTorch root, `build.sh` builds ExecuTorch with the Vulkan backend

‎backends/vulkan/test/ops/benchmark_v2/examples/buffer_shader.json‎

Lines changed: 3 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,9 @@
55
"target_ms": 10,
66
"benchmark_runs": 5,
77
"bindings": [
8-
{
9-
"buffer": {
10-
"name": "output",
11-
"sizes": [8],
12-
"access": "write",
13-
"data": {"fill": 0}
14-
}
15-
},
16-
{
17-
"buffer": {
18-
"name": "input",
19-
"sizes": [8],
20-
"data": {"rand_uniform_float": [-1, 1]}
21-
}
22-
},
23-
{
24-
"meta_ubo": "output",
25-
"field": "numel"
26-
}
8+
{"buffer": {"name": "output", "sizes": [8], "access": "write", "data": {"fill": 0}}},
9+
{"buffer": {"name": "input", "sizes": [8], "data": {"rand_uniform_float": [-1, 1]}}},
10+
{"meta_ubo": "output", "field": "numel"}
2711
],
2812
"outputs": ["output"]
2913
}

‎backends/vulkan/test/ops/benchmark_v2/examples/conv2d.json‎

Lines changed: 5 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -6,57 +6,19 @@
66
"show_memory_throughput": true,
77
"show_compute_throughput": true,
88
"max_dispatches": 512,
9-
"sweep_variables": {
10-
"v_DTYPE": ["float32", "float16"]
11-
},
9+
"sweep_variables": {"v_DTYPE": ["float32", "float16"]},
1210
"args_sweep": [
1311
[
14-
{
15-
"tensor": {
16-
"name": "input",
17-
"sizes": [1, 4, 4, 4],
18-
"dtype": "v_DTYPE",
19-
"storage": "texture3d",
20-
"layout": "channels",
21-
"data": {"rand_uniform_float": [-1, 1]}
22-
}
23-
},
24-
{
25-
"tensor": {
26-
"name": "weight",
27-
"sizes": [8, 4, 3, 3],
28-
"dtype": "v_DTYPE",
29-
"storage": "constant",
30-
"layout": "width",
31-
"data": {"rand_uniform_float": [-1, 1]}
32-
}
33-
},
34-
{
35-
"tensor": {
36-
"name": "bias",
37-
"sizes": [8],
38-
"dtype": "v_DTYPE",
39-
"storage": "constant",
40-
"layout": "width",
41-
"data": {"rand_uniform_float": [-1, 1]}
42-
}
43-
},
12+
{"tensor": {"name": "input", "sizes": [1, 4, 4, 4], "dtype": "v_DTYPE", "storage": "texture3d", "layout": "channels", "data": {"rand_uniform_float": [-1, 1]}}},
13+
{"tensor": {"name": "weight", "sizes": [8, 4, 3, 3], "dtype": "v_DTYPE", "storage": "constant", "layout": "width", "data": {"rand_uniform_float": [-1, 1]}}},
14+
{"tensor": {"name": "bias", "sizes": [8], "dtype": "v_DTYPE", "storage": "constant", "layout": "width", "data": {"rand_uniform_float": [-1, 1]}}},
4415
{"int_list": [1, 1]},
4516
{"int_list": [1, 1]},
4617
{"int_list": [1, 1]},
4718
{"bool": false},
4819
{"int_list": [0, 0]},
4920
{"int": 1},
50-
{
51-
"tensor": {
52-
"name": "output",
53-
"sizes": [1, 8, 4, 4],
54-
"dtype": "v_DTYPE",
55-
"storage": "texture3d",
56-
"layout": "channels",
57-
"data": {"fill": 0}
58-
}
59-
}
21+
{"tensor": {"name": "output", "sizes": [1, 8, 4, 4], "dtype": "v_DTYPE", "storage": "texture3d", "layout": "channels", "data": {"fill": 0}}}
6022
]
6123
]
6224
}

‎backends/vulkan/test/ops/benchmark_v2/examples/conv2d_gemm_sustained.json‎

Lines changed: 7 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -5,67 +5,12 @@
55
"lwg_size": [8, 8, 1],
66
"sustain_s": 12,
77
"bindings": [
8-
{
9-
"texture3d": {
10-
"name": "out",
11-
"sizes": [1, 128, 128, 128],
12-
"dtype": "float16",
13-
"layout": "channels",
14-
"access": "write",
15-
"data": {
16-
"fill": 0
17-
}
18-
}
19-
},
20-
{
21-
"texture2d": {
22-
"name": "im2col",
23-
"sizes": [14464, 576],
24-
"dtype": "float16",
25-
"data": {
26-
"rand_uniform_float": [-1, 1]
27-
}
28-
}
29-
},
30-
{
31-
"texture2d": {
32-
"name": "weight_packed",
33-
"sizes": [144, 512],
34-
"dtype": "float16",
35-
"data": {
36-
"rand_uniform_float": [-0.05, 0.05]
37-
}
38-
}
39-
},
40-
{
41-
"texture2d": {
42-
"name": "bias",
43-
"sizes": [128],
44-
"dtype": "float16",
45-
"data": {
46-
"rand_uniform_float": [-1, 1]
47-
}
48-
}
49-
},
50-
{
51-
"meta_ubo": "out",
52-
"field": "sizes"
53-
}
8+
{"texture3d": {"name": "out", "sizes": [1, 128, 128, 128], "dtype": "float16", "layout": "channels", "access": "write", "data": {"fill": 0}}},
9+
{"texture2d": {"name": "im2col", "sizes": [14464, 576], "dtype": "float16", "data": {"rand_uniform_float": [-1, 1]}}},
10+
{"texture2d": {"name": "weight_packed", "sizes": [144, 512], "dtype": "float16", "data": {"rand_uniform_float": [-0.05, 0.05]}}},
11+
{"texture2d": {"name": "bias", "sizes": [128], "dtype": "float16", "data": {"rand_uniform_float": [-1, 1]}}},
12+
{"meta_ubo": "out", "field": "sizes"}
5413
],
55-
"specialization_constants": [
56-
{
57-
"dtype": "int32",
58-
"value": 0
59-
}
60-
],
61-
"push_constants": [
62-
{
63-
"dtype": "int32",
64-
"data": [144, 0, 113, 0]
65-
},
66-
{
67-
"dtype": "float32",
68-
"data": [0, 0, 0, 0]
69-
}
70-
]
14+
"specialization_constants": [{"dtype": "int32", "value": 0}],
15+
"push_constants": [{"dtype": "int32", "data": [144, 0, 113, 0]}, {"dtype": "float32", "data": [0, 0, 0, 0]}]
7116
}

‎backends/vulkan/test/ops/benchmark_v2/examples/conv2d_routes.json‎

Lines changed: 15 additions & 92 deletions
Original file line numberDiff line numberDiff line change
@@ -10,101 +10,24 @@
1010
"auto",
1111
"direct",
1212
"im2col",
13-
{
14-
"name": "im2col",
15-
"options": {
16-
"intermediate_storage": "buffer"
17-
}
18-
},
19-
{
20-
"name": "im2col",
21-
"options": {
22-
"intermediate_storage": "texture2d"
23-
}
24-
},
25-
{
26-
"name": "im2col",
27-
"options": {
28-
"intermediate_storage": "texture3d"
29-
}
30-
}
13+
{"name": "im2col", "options": {"intermediate_storage": "buffer"}},
14+
{"name": "im2col", "options": {"intermediate_storage": "texture2d"}},
15+
{"name": "im2col", "options": {"intermediate_storage": "texture3d"}}
3116
],
32-
"validation_method": {
33-
"function": "is_close",
34-
"atol": 0.05,
35-
"rtol": 0.01
36-
},
37-
"sweep_variables": {
38-
"v_DTYPE": ["float32", "float16"]
39-
},
17+
"validation_method": {"function": "is_close", "atol": 0.05, "rtol": 0.01},
18+
"sweep_variables": {"v_DTYPE": ["float32", "float16"]},
4019
"args_sweep": [
4120
[
42-
{
43-
"tensor": {
44-
"name": "input",
45-
"sizes": [1, 32, 32, 32],
46-
"dtype": "v_DTYPE",
47-
"storage": "texture3d",
48-
"layout": "channels",
49-
"data": {
50-
"rand_uniform_float": [-1, 1]
51-
}
52-
}
53-
},
54-
{
55-
"tensor": {
56-
"name": "weight",
57-
"sizes": [64, 32, 3, 3],
58-
"dtype": "v_DTYPE",
59-
"storage": "constant",
60-
"layout": "width",
61-
"data": {
62-
"rand_uniform_float": [-0.05, 0.05]
63-
}
64-
}
65-
},
66-
{
67-
"tensor": {
68-
"name": "bias",
69-
"sizes": [64],
70-
"dtype": "v_DTYPE",
71-
"storage": "constant",
72-
"layout": "width",
73-
"data": {
74-
"rand_uniform_float": [-1, 1]
75-
}
76-
}
77-
},
78-
{
79-
"int_list": [1, 1]
80-
},
81-
{
82-
"int_list": [1, 1]
83-
},
84-
{
85-
"int_list": [1, 1]
86-
},
87-
{
88-
"bool": false
89-
},
90-
{
91-
"int_list": [0, 0]
92-
},
93-
{
94-
"int": 1
95-
},
96-
{
97-
"tensor": {
98-
"name": "output",
99-
"sizes": [1, 64, 32, 32],
100-
"dtype": "v_DTYPE",
101-
"storage": "texture3d",
102-
"layout": "channels",
103-
"data": {
104-
"fill": 0
105-
}
106-
}
107-
}
21+
{"tensor": {"name": "input", "sizes": [1, 32, 32, 32], "dtype": "v_DTYPE", "storage": "texture3d", "layout": "channels", "data": {"rand_uniform_float": [-1, 1]}}},
22+
{"tensor": {"name": "weight", "sizes": [64, 32, 3, 3], "dtype": "v_DTYPE", "storage": "constant", "layout": "width", "data": {"rand_uniform_float": [-0.05, 0.05]}}},
23+
{"tensor": {"name": "bias", "sizes": [64], "dtype": "v_DTYPE", "storage": "constant", "layout": "width", "data": {"rand_uniform_float": [-1, 1]}}},
24+
{"int_list": [1, 1]},
25+
{"int_list": [1, 1]},
26+
{"int_list": [1, 1]},
27+
{"bool": false},
28+
{"int_list": [0, 0]},
29+
{"int": 1},
30+
{"tensor": {"name": "output", "sizes": [1, 64, 32, 32], "dtype": "v_DTYPE", "storage": "texture3d", "layout": "channels", "data": {"fill": 0}}}
10831
]
10932
]
11033
}

0 commit comments

Comments
 (0)