Skip to content

Add adaptive red-teaming, judge calibration, and evolutionary auto-learn #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
criptogus merged 2 commits into from
May 17, 2026
Merged

Add adaptive red-teaming, judge calibration, and evolutionary auto-learn #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a8f385
feat(skills): golden set + judge calibration, evolutionary auto-learn…
claude May 17, 2026
4337f31
fix(skills): address Codex review on PR #14
claude May 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/lib/skills/autolearn.functions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ export const autoLearnPackage = createServerFn({ method: "POST" })
.order("created_at", { ascending: false })
.limit(80);

const { data: feedback } = await supabase
.from("package_feedback")
.select("source, rating, sentiment, comments, agent_model")
.eq("package_id", pkg.id)
.order("created_at", { ascending: false })
.limit(120);

const result = await autoLearnPipeline({
pkg: { name: pkg.name, type: pkg.type },
version: {
Expand All @@ -62,6 +69,13 @@ export const autoLearnPackage = createServerFn({ method: "POST" })
weight: number;
created_at: string;
}>,
feedback: (feedback || []) as Array<{
source: string;
rating: number | null;
sentiment: string | null;
comments: string | null;
agent_model: string | null;
}>,
});

let createdVersion: { id: string; version: string; status: string } | null = null;
Expand Down
11 changes: 11 additions & 0 deletions src/lib/skills/evaluator.functions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,15 @@ export const evaluatePackage = createServerFn({ method: "POST" })
: await verQuery.order("created_at", { ascending: false }).limit(1).maybeSingle();
if (vErr || !ver) throw new Response("Version not found", { status: 404 });

const { data: goldenRows } = await supabase
.from("package_golden_cases")
.select("title, input, expected_output, label_pass, label_source")
.eq("package_id", pkg.id)
.eq("is_active", true)
.order("created_at", { ascending: true })
.order("id", { ascending: true })
.limit(20);
Comment on lines +36 to +43

@chatgpt-codex-connector chatgpt-codex-connector Bot May 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Order golden cases before LIMIT to keep calibration stable

The golden-set query applies .limit(20) without any deterministic ordering, so once a package has more than 20 active golden rows the subset fed into calibration can change between runs. Because calibration can cap verdicts, this introduces run-to-run instability unrelated to model behavior. Add an explicit order(...) (e.g., by created_at/id) before limit so the same frozen cases are evaluated consistently.

Useful? React with 👍 / 👎.


const result = await evaluatorPipeline({
pkg: { name: pkg.name, type: pkg.type, description: pkg.description },
version: {
Expand All @@ -41,6 +50,7 @@ export const evaluatePackage = createServerFn({ method: "POST" })
examples: (ver.examples as Array<{ title: string; input: string; expected_output: string }>) || [],
},
extraCases: data.extra_cases,
goldenCases: (goldenRows as Array<{ title: string; input: string; expected_output: string; label_pass: boolean; label_source: string }>) || [],
});

if (data.persist) {
Expand All @@ -61,6 +71,7 @@ export const evaluatePackage = createServerFn({ method: "POST" })
example_results: result.evaluation.example_results,
adversarial_results: { probes: result.adversarial, trigger_rate: result.triggerRate },
pipeline_stages: result.stages,
judge_calibration: result.judgeCalibration,
});
}

Expand Down
30 changes: 30 additions & 0 deletions src/lib/skills/forge-loop.functions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,17 @@ export const runForgeLoop = createServerFn({ method: "POST" })
.maybeSingle();
if (!ver) throw new Response("Version not found", { status: 404 });

const { data: goldenRows } = await supabase
.from("package_golden_cases")
.select("title, input, expected_output, label_pass, label_source")
.eq("package_id", pkg.id)
.eq("is_active", true)
.order("created_at", { ascending: true })
.order("id", { ascending: true })
.limit(20);
const goldenCases =
(goldenRows as Array<{ title: string; input: string; expected_output: string; label_pass: boolean; label_source: string }>) || [];

// 1) Evaluate current
const before = await evaluatorPipeline({
pkg: { name: pkg.name, type: pkg.type },
Expand All @@ -41,6 +52,7 @@ export const runForgeLoop = createServerFn({ method: "POST" })
rules: ver.rules,
examples: (ver.examples as Array<{ title: string; input: string; expected_output: string }>) || [],
},
goldenCases,
});
await supabase.from("package_evaluations").insert({
package_id: pkg.id,
Expand All @@ -59,6 +71,7 @@ export const runForgeLoop = createServerFn({ method: "POST" })
example_results: before.evaluation.example_results,
adversarial_results: before.adversarial,
pipeline_stages: before.stages,
judge_calibration: before.judgeCalibration,
});

// 2) Auto-learn proposal
Expand All @@ -75,6 +88,13 @@ export const runForgeLoop = createServerFn({ method: "POST" })
.order("created_at", { ascending: false })
.limit(80);

const { data: feedback } = await supabase
.from("package_feedback")
.select("source, rating, sentiment, comments, agent_model")
.eq("package_id", pkg.id)
.order("created_at", { ascending: false })
.limit(120);

const learn = await autoLearnPipeline({
pkg: { name: pkg.name, type: pkg.type },
version: { version: ver.version, system_prompt: ver.system_prompt, rules: ver.rules, examples: ver.examples },
Expand All @@ -86,6 +106,13 @@ export const runForgeLoop = createServerFn({ method: "POST" })
weight: number;
created_at: string;
}>,
feedback: (feedback || []) as Array<{
source: string;
rating: number | null;
sentiment: string | null;
comments: string | null;
agent_model: string | null;
}>,
});

// 3) Hot-swap (only if requested AND no regression AND verdict suggests change is needed)
Expand Down Expand Up @@ -128,6 +155,7 @@ export const runForgeLoop = createServerFn({ method: "POST" })
rules: mergedRules,
examples: mergedExamples,
},
goldenCases,
});
await supabase.from("package_evaluations").insert({
package_id: pkg.id,
Expand All @@ -146,6 +174,8 @@ export const runForgeLoop = createServerFn({ method: "POST" })
example_results: after.evaluation.example_results,
adversarial_results: after.adversarial,
pipeline_stages: after.stages,
judge_calibration: after.judgeCalibration,
evolution_trace: { evolution: learn.evolution, feedback_summary: learn.feedback_summary },
});
}

Expand Down
Loading
Loading