Skip to content

Commit af04ee1

Browse files
functionstackxfunctionstackx
authored
docs: add public manual run ingestion guide (#1302)
中文：添加适用于公开仓库的手动运行结果入库指南，使用通用示例，说明目标数据库确认、产物选择、缓存刷新和验证步骤。
1 parent d5d3d2f commit af04ee1

1 file changed

Lines changed: 206 additions & 0 deletions

File tree

‎docs/ingesting-runs.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Manual Run Ingest
2+
3+
How to load a GitHub Actions benchmark run into a dashboard database by hand. CI already ingests `SemiAnalysisAI/InferenceX` sweeps through the ingest workflows. Use this guide for approved runs that need manual ingestion.
4+
5+
Every path below goes through the same ETL entry point, `packages/db/src/ingest-ci-run.ts`. It has two modes:
6+
7+
| Mode | Invocation | Reads artifacts from |
8+
| -------- | ---------------------------------------------- | ---------------------------------------- |
9+
| Download | `ingest-ci-run.ts --download <run-url> [repo]` | GitHub, into a temp directory it creates |
10+
| CI | `ingest-ci-run.ts` with `INGEST_*` env vars | A directory you already populated |
11+
12+
`bun run admin:db:ingest:run <run-url>` is download mode with the repo-root `.env` loaded. You need Bun, the GitHub CLI, `unzip`, and database credentials supplied through your approved credential channel. Never commit `.env` or paste tokens and database URLs into issues, PRs, or logs.
13+
14+
## Set up a clean checkout
15+
16+
Run ingest from current `origin/master`. Model and hardware mappings live in `packages/db/src/etl/normalizers.ts`, and a stale checkout can skip rows it cannot map.
17+
18+
```bash
19+
cd InferenceX-app
20+
git fetch origin
21+
git worktree add --detach /tmp/ifx-app-ingest origin/master
22+
cp .env /tmp/ifx-app-ingest/.env
23+
cd /tmp/ifx-app-ingest
24+
CYPRESS_INSTALL_BINARY=0 bun install --frozen-lockfile
25+
```
26+
27+
## Pick the target database
28+
29+
The ingest writes to `DATABASE_WRITE_URL`. Inspect the host without printing credentials, and confirm that it belongs to the intended database:
30+
31+
```bash
32+
bun --env-file=.env -e '
33+
const value = process.env.DATABASE_WRITE_URL;
34+
if (!value) throw new Error("DATABASE_WRITE_URL is required");
35+
console.log(new URL(value).host);
36+
'
37+
```
38+
39+
Writing to the public database requires explicit publication approval for the run and its artifacts, including server logs and trace payloads. Do not ingest confidential or NDA-covered data into it. A successful ingest is not publication clearance.
40+
41+
Check that the schema is current. Compare `packages/db/migrations` with the applied entries in `schema_migrations`; if migrations are pending, review them and run `bun run admin:db:migrate` from the repo root after confirming the target.
42+
43+
## Inspect the run
44+
45+
Replace the example run ID with the approved GitHub Actions run ID. Keep `REPO` and `RUN` set in the same shell for the commands below.
46+
47+
```bash
48+
REPO=SemiAnalysisAI/InferenceX
49+
RUN=12345678901 # Placeholder: replace before running.
50+
gh api "repos/$REPO/actions/runs/$RUN" \
51+
--jq '{display_title, conclusion, run_attempt, created_at, head_branch}'
52+
gh api "repos/$REPO/actions/runs/$RUN/artifacts?per_page=100" --paginate \
53+
--jq '.artifacts[] | [.name, .size_in_bytes] | @tsv'
54+
```
55+
56+
Things to check:
57+
58+
- **Run attempt:** CI mode needs it as `INGEST_RUN_ATTEMPT`. Download mode reads it from the API.
59+
- **Framework and model:** Read them from the artifact names rather than assuming the requested framework produced the run.
60+
- **Server-log size:** Multinode `multinode_server_logs_*` artifacts can be large. Choose the download path below accordingly.
61+
- **Mappings:** Check `normalizers.ts` for the model prefix and hardware in the artifact names. Unmapped rows are skipped; `UNMAPPED_ENTITIES_OUTPUT=<path>` writes a report for review.
62+
- **Artifact availability:** Confirm that the selected artifacts have not expired.
63+
64+
## Ingest
65+
66+
### GitHub authentication
67+
68+
The ingest requires `GITHUB_TOKEN`, and its artifact helpers use `gh`. Use credentials authorized to read the source repository. An exported token can override your stored GitHub CLI login, so verify access with the metadata command above using the credentials you intend to use.
69+
70+
If using your stored `gh` login instead of an existing environment token, the following sets both variables without printing the token. Do not run it with shell tracing enabled.
71+
72+
```bash
73+
T="$(env -u GITHUB_TOKEN -u GH_TOKEN gh auth token)" &&
74+
export GITHUB_TOKEN="$T" GH_TOKEN="$T"
75+
unset T
76+
```
77+
78+
Re-run the metadata command after changing credentials. Stop if authentication fails.
79+
80+
### Option A: download mode
81+
82+
Use this when the server logs are small enough to download. It also attaches available server logs to benchmark points.
83+
84+
```bash
85+
cd /tmp/ifx-app-ingest/packages/db
86+
bun --env-file=../../.env src/ingest-ci-run.ts --download \
87+
"https://github.com/$REPO/actions/runs/$RUN"
88+
```
89+
90+
Download mode cleans up its temporary directory when it finishes. To compare against raw artifacts afterward, download the benchmark artifacts separately into a directory you retain.
91+
92+
### Option B: skip server logs
93+
94+
Download mode fetches the server logs paired with each selected benchmark artifact. For large logs, download the other artifacts yourself, then ingest in CI mode. This uses the same latest-per-logical-name selection for non-log artifacts; it does not inspect retry success before selecting the latest artifact.
95+
96+
```bash
97+
cd /tmp/ifx-app-ingest/packages/db
98+
DEST="$(mktemp -d "/tmp/ifx-ingest-$RUN.XXXXXX")"
99+
bun -e '
100+
import { dedupeArtifactsByLogicalName, downloadArtifact, listRunArtifacts } from "./src/lib/github-artifacts";
101+
const [repo, runId, dest] = process.argv.slice(1);
102+
for (const a of dedupeArtifactsByLogicalName(listRunArtifacts(repo, runId)).values()) {
103+
if (a.name.startsWith("server_logs_") || a.name.startsWith("multinode_server_logs_")) continue;
104+
console.log(a.name);
105+
downloadArtifact(a, dest);
106+
}
107+
' "$REPO" "$RUN" "$DEST"
108+
```
109+
110+
Stop if any download fails. Only after all downloads finish, fetch the attempt number and run ingest:
111+
112+
```bash
113+
ATTEMPT="$(gh api "repos/$REPO/actions/runs/$RUN" --jq '.run_attempt')" &&
114+
INGEST_RUN_ID="$RUN" INGEST_RUN_ATTEMPT="$ATTEMPT" INGEST_ARTIFACTS_PATH="$DEST" INGEST_REPO="$REPO" \
115+
bun --env-file=../../.env src/ingest-ci-run.ts
116+
```
117+
118+
Skipping server logs preserves trace replay and server metrics only when the `agentic_*` artifacts contain the required AIPerf profile and metrics exports, including `server_metrics_export.json`. Legacy or incomplete direct artifacts may need the server-log tarball as a fallback. Inspect those artifacts before choosing this path, and verify the stored metrics afterward.
119+
120+
### Read the summary
121+
122+
An illustrative summary:
123+
124+
```text
125+
Benchmarks: +7 new, 0 dup
126+
Trace replay: 7 rows linked, 0 agentic point(s) missing sibling artifact
127+
Eval results (agg): +6 new
128+
Skipped: 1 rows
129+
failed run (0 successful): 1
130+
```
131+
132+
- **`failed run (0 successful)`:** A selected artifact had no successful requests and was skipped. Confirm that the expected successful points were stored; this message alone does not prove a good retry was selected.
133+
- **`auto-captured unexpected metric '<key>'`:** The value is stored in `metrics` JSONB. Review the key and add it to `METRIC_KEYS` when it should become a first-class metric.
134+
- **`missing sibling artifact`:** A nonzero count means some agentic points have no matching trace-replay sibling. Check that the required `agentic_*` artifacts downloaded. A zero count alone does not establish that every server-metrics export is present.
135+
136+
## Apply overrides and refresh the cache
137+
138+
From `packages/db`, preview the applicable overrides before applying them:
139+
140+
```bash
141+
bun --env-file=../../.env src/apply-overrides.ts --run-id "$RUN" --allow-unregistered-run
142+
```
143+
144+
After reviewing the preview:
145+
146+
```bash
147+
bun --env-file=../../.env src/apply-overrides.ts --run-id "$RUN" --allow-unregistered-run --yes
148+
```
149+
150+
The ingest refreshes `latest_benchmarks` itself. The dashboard can keep serving cached responses until the cache is invalidated. From the repo root, choose the option for the approved target:
151+
152+
- **Public dashboard:** Dispatch the Cache Refresh workflow, which targets the public deployment.
153+
154+
```bash
155+
gh workflow run cache-refresh.yml --repo SemiAnalysisAI/InferenceX-app --ref master
156+
```
157+
158+
- **Another deployment with `INVALIDATE_SECRET` available:** Use its deployment URL.
159+
160+
```bash
161+
bun run admin:cache:invalidate "https://your-deployment.example"
162+
```
163+
164+
Do not dispatch the public cache workflow for an unrelated database or deployment.
165+
166+
## Verify
167+
168+
Ensure `DATABASE_READONLY_URL` points to the same database as the write connection. Compare stored rows with the source artifacts; this query lists each point with headline metrics and whether server-metrics JSON was attached.
169+
170+
```bash
171+
cd /tmp/ifx-app-ingest/packages/db
172+
bun --env-file=../../.env -e '
173+
import { neon } from "@neondatabase/serverless";
174+
const sql = neon(process.env.DATABASE_READONLY_URL);
175+
console.table(await sql`
176+
select br.conc, c.model, c.hardware, c.framework, c.precision, c.disagg, wr.date,
177+
round((br.metrics->>$$tput_per_gpu$$)::numeric) tput_per_gpu,
178+
round((br.metrics->>$$mean_ttft$$)::numeric, 3) mean_ttft,
179+
atr.server_metrics_json_gz is not null server_metrics
180+
from benchmark_results br
181+
join workflow_runs wr on wr.id = br.workflow_run_id
182+
join configs c on c.id = br.config_id
183+
left join agentic_trace_replay atr on atr.id = br.trace_replay_id
184+
where wr.github_run_id = ${Number(process.argv[1])}
185+
order by br.conc`);' "$RUN"
186+
```
187+
188+
For Option B, `DEST` already points to the retained artifacts. For Option A, set `DEST` to your separately downloaded artifact directory. For AgentX benchmark JSON in the following layout:
189+
190+
```bash
191+
for f in "$DEST"/bmk_agentic_*/*.json; do
192+
jq -r '[.conc, .num_requests_successful, (.request_metrics.throughput.per_gpu.total_tput_tps | round),
193+
.request_metrics.latency.ttft.mean] | @tsv' "$f"
194+
done | sort -n
195+
```
196+
197+
Compare matching configurations and units, accounting for rounding in the displayed query. Also check output throughput, E2EL, and TPOT where present. Agentic ITL and interactivity can differ from legacy artifact fields because ingest uses full-response metrics and can reconstruct them from the AIPerf profile; do not assume a fixed tolerance for that difference.
198+
199+
Finally, open the target dashboard's model page in the AgentX view and verify the run's points. Model slugs live in `packages/app/src/lib/compare-slug.ts`; use a model present in the approved run.
200+
201+
## Behavior to expect
202+
203+
- **Dates:** A run is dated by its `created_at`, not by when a later attempt ran.
204+
- **Re-ingest:** The pipeline uses conflict handling for existing rows. Re-ingest is not a database reset and does not guarantee replacement of already-linked trace payloads.
205+
- **Newer runs of the same config:** A new run does not delete an older run's points. Latest-view selection follows the snapshot and append-only rules in [Data Pipeline](./data-pipeline.md). Removing old data is a separate, explicit step through the run-override registry.
206+
- **Cleanup:** Retained artifacts can consume substantial disk space. After verification, remove only the download directory and temporary worktree you created, including the copied `.env`.

0 commit comments

Comments
 (0)