diff --git a/.agents/skills/ai-native-eval/SKILL.md b/.agents/skills/ai-native-eval/SKILL.md index b565cdd..d0d1906 100644 --- a/.agents/skills/ai-native-eval/SKILL.md +++ b/.agents/skills/ai-native-eval/SKILL.md @@ -67,9 +67,30 @@ From the `ai-native-eval` skill directory: ```sh pnpm --dir scripts/eval install pnpm --dir scripts/eval build -pnpm --dir scripts/eval exec ai-native-eval init-run --out +pnpm --dir scripts/eval exec ai-native-eval init-run pnpm --dir scripts/eval validate-folder --skills-dir ../.. -pnpm --dir scripts/eval render-folder --out --skills-dir ../.. +pnpm --dir scripts/eval render-folder --skills-dir ../.. +``` + +By default, a normal repository evaluation writes a timestamped, repo-local artifact +bundle under `.ai-native-eval/artifacts/-/`. The bundle is +generated output and is ignored by default; it does not require extra user +permission beyond the evaluation request. Use `/tmp` only for dry runs, tests, or +when the user explicitly asks not to write generated artifacts into the repository. + +The default bundle layout is: + +```text +.ai-native-eval/artifacts// + run/ + run.json + evaluators/ + .json + report.html + report.md + report.json + snapshot.json + manifest.json ``` The preferred workflow accepts a folder with `run.json` and per-leaf evaluator outputs: @@ -160,12 +181,14 @@ The `groupId` and `deductionId` must come from that evaluator skill's `ai-native ## Persistence -Persist repo-local state under `.ai-native-eval/` only when the user allows writes. +Persist shared repo-local state under `.ai-native-eval/` only when the user allows writes. - `.ai-native-eval/config.json` is source-controlled shared project policy by default. - `state.json` may point to the latest artifacts. -- Generated reports, snapshots, manifests, run folders, logs, and temporary files belong under `.ai-native-eval/artifacts/`. +- Normal evaluation requests are permission to write generated, ignored artifacts under `.ai-native-eval/artifacts//`. +- Generated reports, snapshots, manifests, run folders, logs, and temporary files belong in the timestamped artifact bundle. - `.ai-native-eval/artifacts/` is ignored by default; promote reviewed results into `self-evaluations/` or project docs when they should become durable repo evidence. +- Ask before promoting artifacts into source-controlled docs, `self-evaluations/**`, shared config, or other durable repository state. - Reports, snapshots, manifests, and ledger entries are append-only once promoted. - JSON report is the source of truth; HTML report is a static render. - Snapshots should preserve the normalized tree, score, confidence, commit, evaluator lock/config hash, plugin resolution, execution batches, and evaluator run records. @@ -207,7 +230,7 @@ Use the requested content language for human-facing evaluator text such as `reas - Do not claim a high-confidence level from docs alone. - Do not scan huge repositories exhaustively by default when prior state exists. -- Do not mutate repository files unless the user asked to persist state or write reports. +- Do not mutate source-controlled repository files unless the user asked to persist state or promote reports. Generated eval bundles under `.ai-native-eval/artifacts//` are the default output for ordinary evaluations. - Do not let evaluator plugins depend on `ai-native-eval` internals. - Do not reintroduce a shared protocol skill as a required dependency for evaluator plugins. - Do not conflate polished docs with actual quality gates. diff --git a/.agents/skills/ai-native-eval/evals/eval.yaml b/.agents/skills/ai-native-eval/evals/eval.yaml index e512e1a..5050c9f 100644 --- a/.agents/skills/ai-native-eval/evals/eval.yaml +++ b/.agents/skills/ai-native-eval/evals/eval.yaml @@ -30,3 +30,23 @@ tasks: - type: deterministic run: node eval-support/grade-response.mjs expectations/expected.json solution: solutions/basic-routing.sh + - name: ai-native-eval-artifact-bundle-default + instruction: | + Read `SKILL.md` for this skill and write the response to + `outputs/artifact-bundle-response.md`. + + The response must include the exact marker `AI_NATIVE_ARTIFACT_BUNDLE_DEFAULT`, + the default generated artifact bundle path pattern, and one concise sentence + explaining when `/tmp` is appropriate. Do not modify repository files other + than the requested output artifact. + workspace: + - src: expectations/artifact-bundle-default.json + dest: expectations/expected.json + - src: ../../_eval-support/grade-response.mjs + dest: eval-support/grade-response.mjs + - src: solutions/artifact-bundle-default.sh + dest: artifact-bundle-default.sh + graders: + - type: deterministic + run: node eval-support/grade-response.mjs expectations/expected.json + solution: solutions/artifact-bundle-default.sh diff --git a/.agents/skills/ai-native-eval/evals/expectations/artifact-bundle-default.json b/.agents/skills/ai-native-eval/evals/expectations/artifact-bundle-default.json new file mode 100644 index 0000000..659bb1e --- /dev/null +++ b/.agents/skills/ai-native-eval/evals/expectations/artifact-bundle-default.json @@ -0,0 +1,10 @@ +{ + "output_path": "outputs/artifact-bundle-response.md", + "must_contain": [ + "AI_NATIVE_ARTIFACT_BUNDLE_DEFAULT", + ".ai-native-eval/artifacts//" + ], + "must_match": [ + "(/tmp|tmp).*dry runs|dry runs.*(/tmp|tmp)|tests.*(/tmp|tmp)|(/tmp|tmp).*tests|explicitly asks not to write|not to write generated artifacts" + ] +} diff --git a/.agents/skills/ai-native-eval/evals/solutions/artifact-bundle-default.sh b/.agents/skills/ai-native-eval/evals/solutions/artifact-bundle-default.sh new file mode 100755 index 0000000..1d8e9db --- /dev/null +++ b/.agents/skills/ai-native-eval/evals/solutions/artifact-bundle-default.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail + +mkdir -p outputs +cat > outputs/artifact-bundle-response.md <<'EOF' +AI_NATIVE_ARTIFACT_BUNDLE_DEFAULT +.ai-native-eval/artifacts// +Normal repository evaluations write generated artifacts to the repo-local bundle by default; /tmp is appropriate only for dry runs, tests, or when the user explicitly asks not to write generated artifacts into the repository. +EOF diff --git a/.agents/skills/ai-native-eval/scripts/eval/package.json b/.agents/skills/ai-native-eval/scripts/eval/package.json index 3673311..880c5cc 100644 --- a/.agents/skills/ai-native-eval/scripts/eval/package.json +++ b/.agents/skills/ai-native-eval/scripts/eval/package.json @@ -9,8 +9,8 @@ "test:tool": "pnpm build && node --test dist/tests/aggregate.test.js", "test:browser": "pnpm build && node --test dist/tests/report-browser.e2e.js", "test:agent": "pnpm build && RUN_REAL_CODEX_E2E=1 node --test dist/tests/real-agent.e2e.js", - "render:example": "pnpm build && node dist/src/cli.js render fixtures/evaluation-tree.example.json --out .ai-native-eval/artifacts/reports/example-report.html", - "render-folder:example": "pnpm build && node dist/src/cli.js render-folder fixtures/folder-run.valid --out .ai-native-eval/artifacts/reports/folder-report.html --skills-dir ../../../", + "render:example": "pnpm build && node dist/src/cli.js render fixtures/evaluation-tree.example.json --out .ai-native-eval/artifacts/example-report/report.html", + "render-folder:example": "pnpm build && node dist/src/cli.js render-folder fixtures/folder-run.valid --out .ai-native-eval/artifacts/folder-report/report.html --skills-dir ../../../", "validate-folder:example": "pnpm build && node dist/src/cli.js validate-folder fixtures/folder-run.valid --skills-dir ../../../", "persist:example": "pnpm build && node dist/src/cli.js persist fixtures/evaluation-tree.example.json --changed-file README.md --changed-file docs/blueprint.md", "score:example": "pnpm build && node dist/src/cli.js score fixtures/evaluation-tree.example.json" diff --git a/.agents/skills/ai-native-eval/scripts/eval/src/cli.ts b/.agents/skills/ai-native-eval/scripts/eval/src/cli.ts index d514511..70b34a9 100644 --- a/.agents/skills/ai-native-eval/scripts/eval/src/cli.ts +++ b/.agents/skills/ai-native-eval/scripts/eval/src/cli.ts @@ -1,6 +1,7 @@ #!/usr/bin/env node import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { dirname } from "node:path"; +import { spawnSync } from "node:child_process"; +import { basename, dirname, join } from "node:path"; import { buildReport } from "./aggregate.js"; import { initRun } from "./config.js"; import { @@ -93,6 +94,8 @@ async function main(): Promise { await writeJsonArtifact(paths.reportJsonPath, report); await writeJsonArtifact(paths.snapshotPath, snapshotFromReport(report)); await writeJsonArtifact(paths.manifestPath, manifest); + await mkdir(dirname(paths.reportMarkdownPath), { recursive: true }); + await writeFile(paths.reportMarkdownPath, renderMarkdownReport(report)); await mkdir(dirname(paths.reportHtmlPath), { recursive: true }); await writeFile(paths.reportHtmlPath, renderHtmlReport(report)); console.log(JSON.stringify(paths, null, 2)); @@ -111,27 +114,41 @@ async function main(): Promise { } async function runInitRunCommand(repoRoot: string, rest: string[]): Promise { - const out = readOption(rest, "--out"); - if (!out) { - console.error("Missing --out "); - process.exitCode = 2; - return; - } + const repoCommit = readOption(rest, "--repo-commit") ?? readGitCommit(repoRoot); + const generatedAt = new Date().toISOString(); + const reportId = + readOption(rest, "--report-id") ?? + createRunId({ + generatedAt, + headCommit: repoCommit + }); + const out = + readOption(rest, "--out") ?? + join(repoRoot, ".ai-native-eval", "artifacts", reportId, "run"); const runPath = await initRun({ repoRoot, runFolder: out, personConfigPath: readOption(rest, "--person-config"), projectConfigPath: readOption(rest, "--project-config"), explicitConfigPath: readOption(rest, "--config"), - reportId: readOption(rest, "--report-id"), + reportId, + generatedAt, language: readOption(rest, "--language"), uiLanguage: readOption(rest, "--ui-language") as "en" | "zh-TW" | undefined, scope: readOption(rest, "--scope"), - repoCommit: readOption(rest, "--repo-commit") + repoCommit }); console.log(runPath); } +function readGitCommit(repoRoot: string): string | undefined { + const result = spawnSync("git", ["-C", repoRoot, "rev-parse", "HEAD"], { + encoding: "utf8" + }); + if (result.status !== 0) return undefined; + return result.stdout.trim() || undefined; +} + async function runFolderCommand( command: "validate-folder" | "render-folder", inputPath: string, @@ -157,9 +174,16 @@ async function runFolderCommand( runFolder: inputPath, skillsDir }); - const out = readOption(rest, "--out"); - const jsonOut = readOption(rest, "--json-out"); - const markdownOut = readOption(rest, "--markdown-out"); + const explicitOut = readOption(rest, "--out"); + const explicitJsonOut = readOption(rest, "--json-out"); + const explicitMarkdownOut = readOption(rest, "--markdown-out"); + const defaultOutputs = + !explicitOut && !explicitJsonOut && !explicitMarkdownOut + ? defaultReportOutputs(inputPath) + : undefined; + const out = explicitOut ?? defaultOutputs?.html; + const jsonOut = explicitJsonOut ?? defaultOutputs?.json; + const markdownOut = explicitMarkdownOut ?? defaultOutputs?.markdown; if (!out && !jsonOut && !markdownOut) { console.error( "Missing one of --out , --json-out , or --markdown-out " @@ -182,6 +206,20 @@ async function runFolderCommand( await writeFile(markdownOut, renderMarkdownReport(report)); console.log(markdownOut); } + if (defaultOutputs) { + await writeJsonArtifact(defaultOutputs.snapshot, snapshotFromReport(report)); + await writeJsonArtifact( + defaultOutputs.manifest, + buildIncrementalManifest({ + manifestId: report.reportId, + generatedAt: report.generatedAt, + headCommit: report.reproducibility?.repoCommit, + changedFiles: [] + }) + ); + console.log(defaultOutputs.snapshot); + console.log(defaultOutputs.manifest); + } } catch (error) { if (error instanceof FolderValidationError) { printValidationErrors(error.errors); @@ -199,6 +237,20 @@ function printValidationErrors(errors: string[]): void { } } +function defaultReportOutputs(runFolder: string): + | { html: string; json: string; markdown: string; snapshot: string; manifest: string } + | undefined { + if (basename(runFolder) !== "run") return undefined; + const bundleRoot = dirname(runFolder); + return { + html: join(bundleRoot, "report.html"), + json: join(bundleRoot, "report.json"), + markdown: join(bundleRoot, "report.md"), + snapshot: join(bundleRoot, "snapshot.json"), + manifest: join(bundleRoot, "manifest.json") + }; +} + function readOption(args: string[], name: string): string | undefined { const index = args.indexOf(name); if (index === -1) return undefined; @@ -210,9 +262,13 @@ function usage(): void { ai-native-eval score ai-native-eval render --out [--language zh-TW] [--ui-language zh-TW] ai-native-eval persist [--root .ai-native-eval/artifacts] [--language zh-TW] [--ui-language zh-TW] [--changed-file ]... - ai-native-eval init-run --out [--config ] [--project-config ] [--person-config ] + ai-native-eval init-run [--out ] [--config ] [--project-config ] [--person-config ] ai-native-eval validate-folder [--skills-dir .agents/skills] - ai-native-eval render-folder [--out ] [--json-out ] [--markdown-out ] [--skills-dir .agents/skills]`); + ai-native-eval render-folder [--out ] [--json-out ] [--markdown-out ] [--skills-dir .agents/skills] + +Default repo-local artifact bundle: + init-run without --out creates .ai-native-eval/artifacts/-/run + render-folder on a bundle run folder writes report.html, report.md, report.json, snapshot.json, and manifest.json beside run/`); } function readRepeatedOption(args: string[], name: string): string[] { diff --git a/.agents/skills/ai-native-eval/scripts/eval/src/persistence.ts b/.agents/skills/ai-native-eval/scripts/eval/src/persistence.ts index 0590a61..16532e4 100644 --- a/.agents/skills/ai-native-eval/scripts/eval/src/persistence.ts +++ b/.agents/skills/ai-native-eval/scripts/eval/src/persistence.ts @@ -17,12 +17,16 @@ export function createRunId(input: { } export function artifactPaths(runId: string, root = ".ai-native-eval/artifacts"): ArtifactPaths { + const bundleRoot = join(root, runId); return { runId, - snapshotPath: join(root, "snapshots", `${runId}-snapshot.json`), - manifestPath: join(root, "manifests", `${runId}-manifest.json`), - reportJsonPath: join(root, "reports", `${runId}-level-report.json`), - reportHtmlPath: join(root, "reports", `${runId}-level-report.html`) + bundleRoot, + runFolder: join(bundleRoot, "run"), + snapshotPath: join(bundleRoot, "snapshot.json"), + manifestPath: join(bundleRoot, "manifest.json"), + reportJsonPath: join(bundleRoot, "report.json"), + reportMarkdownPath: join(bundleRoot, "report.md"), + reportHtmlPath: join(bundleRoot, "report.html") }; } diff --git a/.agents/skills/ai-native-eval/scripts/eval/src/types.ts b/.agents/skills/ai-native-eval/scripts/eval/src/types.ts index 30bfbf8..b4e3a5a 100644 --- a/.agents/skills/ai-native-eval/scripts/eval/src/types.ts +++ b/.agents/skills/ai-native-eval/scripts/eval/src/types.ts @@ -299,8 +299,11 @@ export interface IncrementalManifest { export interface ArtifactPaths { runId: string; + bundleRoot: string; + runFolder: string; snapshotPath: string; manifestPath: string; reportJsonPath: string; + reportMarkdownPath: string; reportHtmlPath: string; } diff --git a/.agents/skills/ai-native-eval/scripts/eval/tests/aggregate.test.ts b/.agents/skills/ai-native-eval/scripts/eval/tests/aggregate.test.ts index 4a26c16..8892316 100644 --- a/.agents/skills/ai-native-eval/scripts/eval/tests/aggregate.test.ts +++ b/.agents/skills/ai-native-eval/scripts/eval/tests/aggregate.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { spawnSync } from "node:child_process"; @@ -491,6 +491,44 @@ test("render-folder can write validated JSON and markdown reports", async () => } }); +test("render-folder defaults to report files beside a bundle run folder", async () => { + const tempRoot = await mkdtemp(join(tmpdir(), "ai-native-eval-bundle-render-")); + try { + const bundleRoot = join(tempRoot, "20260613T120000Z-abcdef123456"); + const runFolder = join(bundleRoot, "run"); + await cp("fixtures/folder-run.valid", runFolder, { recursive: true }); + + const result = spawnSync( + process.execPath, + ["dist/src/cli.js", "render-folder", runFolder, "--skills-dir", "../../../"], + { cwd: process.cwd(), encoding: "utf8" } + ); + + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /report\.html/); + assert.match(result.stdout, /report\.json/); + assert.match(result.stdout, /report\.md/); + assert.match(result.stdout, /snapshot\.json/); + assert.match(result.stdout, /manifest\.json/); + const html = await readFile(join(bundleRoot, "report.html"), "utf8"); + const json = JSON.parse(await readFile(join(bundleRoot, "report.json"), "utf8")) as EvaluationReport; + const markdown = await readFile(join(bundleRoot, "report.md"), "utf8"); + const snapshot = JSON.parse(await readFile(join(bundleRoot, "snapshot.json"), "utf8")) as { + reportId: string; + }; + const manifest = JSON.parse(await readFile(join(bundleRoot, "manifest.json"), "utf8")) as { + manifestId: string; + }; + assert.match(html, /Local runtime command evaluator/); + assert.equal(json.reportId, "folder-run-valid"); + assert.match(markdown, /Local runtime command evaluator/); + assert.equal(snapshot.reportId, "folder-run-valid"); + assert.equal(manifest.manifestId, "folder-run-valid"); + } finally { + await rm(tempRoot, { recursive: true, force: true }); + } +}); + test("folder validation reports all evaluator output errors", async () => { const result = await validateFolderReport({ runFolder: "fixtures/folder-run.invalid", @@ -597,6 +635,37 @@ test("initializes an audited run config snapshot from built-in and project confi } }); +test("init-run defaults to a repo-local timestamped artifact bundle", async () => { + const tempRoot = await mkdtemp(join(tmpdir(), "ai-native-eval-cli-init-")); + try { + const repoRoot = join(tempRoot, "repo"); + await mkdir(repoRoot, { recursive: true }); + + const result = spawnSync( + process.execPath, + ["dist/src/cli.js", "init-run", repoRoot, "--repo-commit", "abcdef1234567890"], + { cwd: process.cwd(), encoding: "utf8" } + ); + + assert.equal(result.status, 0, result.stderr); + const runPath = result.stdout.trim(); + assert.match( + runPath, + /repo\/\.ai-native-eval\/artifacts\/\d{8}T\d{6}Z-abcdef123456\/run\/run\.json$/ + ); + const run = JSON.parse(await readFile(runPath, "utf8")) as { + runId: string; + reportId: string; + reproducibility: { repoCommit?: string }; + }; + assert.match(run.runId, /^\d{8}T\d{6}Z-abcdef123456$/); + assert.equal(run.reportId, run.runId); + assert.equal(run.reproducibility.repoCommit, "abcdef1234567890"); + } finally { + await rm(tempRoot, { recursive: true, force: true }); + } +}); + test("disabled root remains visible in the report without requiring leaf outputs", async () => { const validation = await validateFolderReport({ runFolder: "fixtures/folder-run.disabled-root", @@ -768,13 +837,33 @@ test("creates append-only artifact paths and incremental manifests", () => { assert.equal(runId, "20260613T120000Z-abcdef123456"); const paths = artifactPaths(runId); + assert.equal( + paths.bundleRoot, + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456" + ); + assert.equal( + paths.runFolder, + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456/run" + ); assert.equal( paths.reportHtmlPath, - ".ai-native-eval/artifacts/reports/20260613T120000Z-abcdef123456-level-report.html" + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456/report.html" + ); + assert.equal( + paths.reportMarkdownPath, + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456/report.md" + ); + assert.equal( + paths.reportJsonPath, + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456/report.json" ); assert.equal( paths.snapshotPath, - ".ai-native-eval/artifacts/snapshots/20260613T120000Z-abcdef123456-snapshot.json" + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456/snapshot.json" + ); + assert.equal( + paths.manifestPath, + ".ai-native-eval/artifacts/20260613T120000Z-abcdef123456/manifest.json" ); const manifest = buildIncrementalManifest({ diff --git a/.agents/skills/ai-native-eval/scripts/eval/tests/real-agent.e2e.ts b/.agents/skills/ai-native-eval/scripts/eval/tests/real-agent.e2e.ts index 3d97eaf..8a3d653 100644 --- a/.agents/skills/ai-native-eval/scripts/eval/tests/real-agent.e2e.ts +++ b/.agents/skills/ai-native-eval/scripts/eval/tests/real-agent.e2e.ts @@ -32,13 +32,11 @@ test( const fixtureProject = join(process.cwd(), "fixtures/real-agent-project"); const sourceSkillsDir = join(process.cwd(), "../../.."); const projectSkillsDir = join(projectDir, ".agents/skills"); - const runFolder = join(projectDir, ".ai-native-eval/artifacts/runs/real-agent-e2e"); - const reportPath = join(projectDir, ".ai-native-eval/artifacts/reports/real-agent-e2e.html"); - const lastMessagePath = join( - projectDir, - ".ai-native-eval/artifacts/real-agent-last-message.md" - ); - const logPath = join(projectDir, ".ai-native-eval/artifacts/real-agent-codex.log"); + const artifactBundle = join(projectDir, ".ai-native-eval/artifacts/real-agent-e2e"); + const runFolder = join(artifactBundle, "run"); + const reportPath = join(artifactBundle, "report.html"); + const lastMessagePath = join(artifactBundle, "last-message.md"); + const logPath = join(artifactBundle, "codex.log"); let keepArtifacts = process.env.KEEP_REAL_AGENT_E2E_ARTIFACTS === "1"; try { @@ -175,17 +173,18 @@ Use the local project skill at .agents/skills/ai-native-eval/SKILL.md to run a r This is an automated real-agent E2E test. Do not ask for clarification. Do not use prewritten evaluator outputs. You must inspect this repo and write fresh evaluator JSON files yourself. Required artifact paths: -- run folder: .ai-native-eval/artifacts/runs/real-agent-e2e -- HTML report: .ai-native-eval/artifacts/reports/real-agent-e2e.html +- artifact bundle: .ai-native-eval/artifacts/real-agent-e2e +- run folder: .ai-native-eval/artifacts/real-agent-e2e/run +- HTML report: .ai-native-eval/artifacts/real-agent-e2e/report.html Required workflow: 1. Read .agents/skills/ai-native-eval/SKILL.md. 2. Install/build the bundled eval tool if needed. 3. Run init-run for this repo using the required run folder. The repo already has .ai-native-eval/config.json; use it. 4. Evaluate the enabled leaf evaluators from the runtime-resolved tree. The project config disables all built-in roots except repo operability, so the enabled leaves should be local runtime command and local environment reproducibility. -5. Write one JSON output file per enabled leaf under .ai-native-eval/artifacts/runs/real-agent-e2e/evaluators/. +5. Write one JSON output file per enabled leaf under .ai-native-eval/artifacts/real-agent-e2e/run/evaluators/. 6. Validate the folder with the bundled tool. -7. Render the static HTML report to .ai-native-eval/artifacts/reports/real-agent-e2e.html. +7. Render the static HTML report to .ai-native-eval/artifacts/real-agent-e2e/report.html. Acceptance expectations: - The local runtime command evaluator should recognize that package scripts for dev, build, and test exist. diff --git a/README.md b/README.md index 4084b7b..c109978 100644 --- a/README.md +++ b/README.md @@ -113,13 +113,18 @@ Repos being evaluated may store eval state under: state.json evidence-ledger.jsonl artifacts/ - runs/ - reports/ - snapshots/ - manifests/ + 20260614T183012Z-a1b2c3d4e5f6/ + run/ + run.json + evaluators/ + report.html + report.md + report.json + snapshot.json + manifest.json ``` -`config.json`, `state.json`, and small evidence ledgers may be source-controlled when they define shared project policy or durable evaluation state. Generated output under `artifacts/` is ignored by default. Promote reviewed reports into a stable committed evidence folder when they should become part of the repo history. +`config.json`, `state.json`, and small evidence ledgers may be source-controlled when they define shared project policy or durable evaluation state. A normal evaluation writes a timestamped generated bundle under `artifacts/` by default, so the complete run can be copied, attached, reviewed, or removed as one directory. Generated output under `artifacts/` is ignored by default. Promote reviewed reports into a stable committed evidence folder when they should become part of the repo history. ## Repo Layout diff --git a/README_CN.md b/README_CN.md index 09255d4..ad00913 100644 --- a/README_CN.md +++ b/README_CN.md @@ -113,13 +113,18 @@ AI-native foundation: 8.2 / 10 state.json evidence-ledger.jsonl artifacts/ - runs/ - reports/ - snapshots/ - manifests/ + 20260614T183012Z-a1b2c3d4e5f6/ + run/ + run.json + evaluators/ + report.html + report.md + report.json + snapshot.json + manifest.json ``` -当 `config.json`、`state.json` 与小型 evidence ledger 定义共享项目策略或持久评估状态时,可以纳入源码管理。默认情况下,`artifacts/` 下的生成输出会被忽略。需要成为仓库历史一部分的已审查报告,应提升到稳定的已提交证据目录。 +当 `config.json`、`state.json` 与小型 evidence ledger 定义共享项目策略或持久评估状态时,可以纳入源码管理。普通评估默认会在 `artifacts/` 下写入带时间戳的生成 bundle,方便把完整 run 作为一个目录复制、附加、审查或删除。默认情况下,`artifacts/` 下的生成输出会被忽略。需要成为仓库历史一部分的已审查报告,应提升到稳定的已提交证据目录。 ## 仓库结构 diff --git a/docs/architecture.md b/docs/architecture.md index ea9db3b..c831622 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -51,6 +51,11 @@ repo evidence The report JSON is the source of truth. The HTML report is a static inspection surface generated from the same validated report. +Generated repository evaluations are grouped as copyable bundles under +`.ai-native-eval/artifacts//`. The bundle root contains the human-facing +reports plus snapshot and manifest metadata, while `run/` contains `run.json` and +per-leaf evaluator output files. + ## Ownership Rules - Evaluator plugin manifests and deduction rubrics belong in each evaluator skill's `SKILL.md`. diff --git a/docs/evidence.md b/docs/evidence.md index d470a0c..6207f76 100644 --- a/docs/evidence.md +++ b/docs/evidence.md @@ -9,7 +9,7 @@ The eval should be able to prove its own behavior with durable artifacts. - Deterministic tests: root `tests/**` and nested eval tests. - Skill evals: `.agents/skills/**/evals/**`. - Project eval config: `.ai-native-eval/config.json` when present. -- Local generated eval artifacts: `.ai-native-eval/artifacts/**`. +- Local generated eval bundles: `.ai-native-eval/artifacts//**`. - PR evidence: command output summaries, report links, and review findings. ## Self-Evaluation Artifacts @@ -23,7 +23,7 @@ self-evaluations/foundation-20260614/run/ self-evaluations/foundation-20260614/report.md ``` -Local ad hoc runs are generated under `.ai-native-eval/artifacts/**`, which is ignored by default. The committed run folder contains one JSON file per leaf evaluator. The compact Markdown report is generated from that folder, not hand-edited. Full HTML and normalized JSON reports may be generated locally for inspection, but they are intentionally not committed because they are much larger. +Local ad hoc runs are generated under `.ai-native-eval/artifacts//`, which is ignored by default. A generated bundle keeps the run folder, reports, snapshot, and manifest together so the evaluation can be copied, attached, reviewed, or removed as one directory. The committed run folder contains one JSON file per leaf evaluator. The compact Markdown report is generated from that folder, not hand-edited. Full HTML and normalized JSON reports may be generated locally for inspection, but they are intentionally not committed because they are much larger. Evaluation artifacts should not persist secrets, tokens, raw private logs, large binary artifacts, or unrelated user data. Summarize sensitive evidence instead of copying it into reports. diff --git a/docs/runtime.md b/docs/runtime.md index 19b1b40..feca9d0 100644 --- a/docs/runtime.md +++ b/docs/runtime.md @@ -80,6 +80,20 @@ Regenerates the compact self-evaluation report in a temporary check flow and fai The CLI can still generate local HTML or JSON reports with `render-folder --out ` or `render-folder --json-out `, but those larger artifacts are not committed for the baseline. +For ordinary repository evaluations, the default generated output is a copyable artifact bundle: + +```text +.ai-native-eval/artifacts/-/ + run/ + report.html + report.md + report.json + snapshot.json + manifest.json +``` + +`init-run ` creates `.ai-native-eval/artifacts/-/run` when `--out` is omitted. `render-folder` writes `report.html`, `report.md`, `report.json`, `snapshot.json`, and `manifest.json` beside `run/` when the input path is a bundle run folder. + ## CI GitHub Actions runs deterministic gates on pull requests and pushes to `main`: