feat(test): evals by harijoe · Pull Request #1058 · alpic-ai/skybridge

harijoe · 2026-08-20T13:58:06Z

Summary

Adds @skybridge/test: eval runner answering whether a prompt reaches the right tool with the right arguments.
Enabled via skybridge({ evals }); scenarios use vitest's own it/expect plus start<AppType>().
The model is a provider/model-id string, and only the provider named gets imported.
Anthropic, OpenAI, Mistral and Google are optional peers; each reads its own API key variable.
For anything those prefixes cannot reach, defineEvalModel() registers a model from a setup file.
Extracts core's Vite plugin to @skybridge/vite-plugin.
Breaking: skybridge/vite is removed; import skybridge from @skybridge/vite-plugin instead.
server.run() now reports its bound port over IPC, letting the runner use an ephemeral port with no bind race.
Repeated runs with an acceptance threshold are deferred to SKY-551.

Implements SKY-546. API validated on the POC (#1052) and the design review thread.

mintlify · 2026-08-20T13:58:10Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
skybridge-staging	🟢 Ready	View Preview	Aug 20, 2026, 1:58 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

greptile-apps · 2026-08-20T14:01:51Z

Greptile Summary

The PR adds the Skybridge eval runner and extracts the Vite integration into @skybridge/vite-plugin.

Adds model-driven MCP evaluation sessions and Vitest chat matchers.
Starts eval servers on ephemeral ports using IPC readiness reporting.
Migrates projects, examples, templates, and publishing workflows to the extracted Vite plugin.

Confidence Score: 5/5

The PR appears safe to merge because the previously reported matcher failure is fixed and no blocking failure remains.

No blocking failure remains.

_{Reviews (19): Last reviewed commit: "feat(test): assert on what the assistant..." | Re-trigger Greptile}

harijoe · 2026-08-21T09:38:39Z

@greptile-apps

fredericbarthelet · 2026-08-22T10:14:56Z

+  const evals = options?.evals;
+  if (evals !== undefined) {
+    plugins.push(loadEvalsPlugin(evals));
+  }


I don't think it's a good idea to have polymorphic plugins and using skybridge/vite-plugin should "always" package both views folder for build and prepare server for unit test.
Happy to discuss split of concerns between skybridge/test and this package if you want

fredericbarthelet · 2026-08-22T10:15:52Z

+ * Everything the runner needs is configured here, so a scenario file imports
+ * nothing but `start` and its own `AppType`.
+ */
+export function evals(options: EvalsOptions): Plugin {


why not put this directly in @skybridge/vite-plugin ?

fredericbarthelet · 2026-08-22T10:17:08Z

+    throw new Error("The evals plugin needs either `server` or `project`");
+  }
+
+  const server = await startServer(options.project);


this should be the responsability of the start method instead to avoid booting the server on test suits that don't need it

The thing is, moving the server booting the start method means spawning a new server per worker. I agree the server shouldn't start for test suites that don't need it though — probably via a config somewhere?

Discussed together, we keep server startup in the setup script of the plugin for now, and see how it affects vitest run performances that are not using chat assertions

fredericbarthelet · 2026-08-24T07:41:09Z

+   * Wires the eval runner into vitest: contributes `setupFiles`,
+   * `globalSetup`, `provide` and `testTimeout` from `@skybridge/test`, loaded
+   * lazily so production builds never touch it. Set it only in the config
+   * vitest uses for evals; when omitted, no test config is contributed and
+   * ordinary unit tests are unaffected. Requires `@skybridge/test` as a dev
+   * dependency.


this feels more like a claude comment that something to help end user understand what this config option will allow him to configure

fredericbarthelet · 2026-08-24T07:42:37Z

+
+The Vite plugin behind [Skybridge](https://skybridge.tech): it discovers view
+modules, serves them as virtual entries, applies the `data-llm` transform, and
+tells `skybridge build` which packages to leave unbundled.


it also extend vitest assertions with chat-oriented APIs to assert various scenario occurring chances

fredericbarthelet

a few comments, I think you can greatly reduce complexity by avoiding plugin definition polymorphism based on config options and by starting the server in the start chat method itself

fredericbarthelet · 2026-08-25T08:20:29Z

            fi

            echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+            echo "beta_version=$VERSION-beta" >> "$GITHUB_OUTPUT"


don't forget to remove those lines before merging :)

fredericbarthelet · 2026-08-25T08:49:12Z

+  }
+
+  /** The tool definitions the model was looking at, for failure reports. */
+  get toolDefinitions(): ToolDefinition[] {


why is this part of the public API, I feel like assertions on the tool definition is like asserting that an exact line of code is written 😅

fredericbarthelet · 2026-08-25T08:51:00Z

+  }
+
+  /** Takes a turn and returns the model's answer. */
+  async send(prompt: string): Promise<string> {


returning only the last turn text response might be a bit restrictive : it's probably better to have assertion helpers that can dig in all properties of the assistant messages to test for the presence of a response that providing it in the public API and letting people run their own assertion on the string content, WDYT?

you agreed and resolved, but didn't change the API 😅

fredericbarthelet · 2026-08-25T08:51:54Z

+    "./vitest-config": {
+      "types": "./dist/vitest-config.d.ts",
+      "default": "./dist/vitest-config.js"


this is not used anymore

fredericbarthelet · 2026-08-25T08:52:35Z

+    "@ai-sdk/anthropic": "^2.0.0",
+    "@ai-sdk/google": "^2.0.0",
+    "@ai-sdk/mistral": "^2.0.0",
+    "@ai-sdk/openai": "^2.0.0",


you need all 4?

fredericbarthelet · 2026-08-25T08:53:06Z

+    "@ai-sdk/anthropic": {
+      "optional": true
+    },
+    "@ai-sdk/google": {
+      "optional": true
+    },
+    "@ai-sdk/mistral": {
+      "optional": true
+    },
+    "@ai-sdk/openai": {
+      "optional": true
+    }


if you switch to having a model being a LanguageModel , you only need a peer dep to the core ai sdk package

The ecosystem convention is peer + dev on ai, and no @ai-sdk/{openai,anthropic,…} at all.
Evalite uses LanguageModel interface from ai as input for their instrumentation

Eve does that too

fredericbarthelet · 2026-08-25T09:15:26Z

+```ts
+skybridge({
+  evals: {
+    model: "anthropic/claude-sonnet-4-5",


this notation is usually reserved for people using the vercel ai gateway, it can be also supported in the project if you don't provide a LanguageModel

fredericbarthelet · 2026-08-25T09:16:00Z

+`openai/gpt-5` also resolves through the Vercel AI Gateway if you have
+`AI_GATEWAY_API_KEY` set and no provider package installed.
+
+## Models the prefixes cannot reach


this use case is supported by ai sdk custom provider

fredericbarthelet

A few comments, but it starts to take shape :)
Also, could you throw a first version of documentation for this new package in this PR?

fredericbarthelet

LGTM :) Awesome work @harijoe !

mintlify Bot deployed to staging - docs August 20, 2026 13:58 View deployment

greptile-apps Bot reviewed Aug 20, 2026

View reviewed changes

Comment thread packages/test/src/matchers.ts Outdated

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 7466734 to 7aada9d Compare August 20, 2026 15:02

mintlify Bot deployed to staging - docs August 20, 2026 15:02 View deployment

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch 4 times, most recently from 34007b8 to 50c10e1 Compare August 21, 2026 13:30

mintlify Bot deployed to staging - docs August 21, 2026 14:01 View deployment

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 50c10e1 to df0cfdd Compare August 21, 2026 14:19

mintlify Bot deployed to staging - docs August 21, 2026 15:45 View deployment

harijoe changed the title ~~feat(test): eval runner as @skybridge/test, extract @skybridge/vite-plugin~~ feat(test): evals Aug 21, 2026

mintlify Bot deployed to staging - docs August 21, 2026 16:18 View deployment

mintlify Bot deployed to staging - docs August 21, 2026 16:24 View deployment

mintlify Bot deployed to staging - docs August 21, 2026 17:17 View deployment

fredericbarthelet reviewed Aug 22, 2026

View reviewed changes

Comment thread packages/test/src/chat.ts

fredericbarthelet reviewed Aug 24, 2026

View reviewed changes

Comment thread packages/test/src/chat.ts

fredericbarthelet reviewed Aug 24, 2026

View reviewed changes

Comment thread packages/vite-plugin/src/plugin.ts

fredericbarthelet reviewed Aug 24, 2026

View reviewed changes

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 874c2eb to dde647a Compare August 24, 2026 09:12

mintlify Bot deployed to staging - docs August 24, 2026 09:14 View deployment

fredericbarthelet reviewed Aug 25, 2026

View reviewed changes

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 0ad9d4b to 8a4b6f5 Compare August 25, 2026 12:54

mintlify Bot deployed to staging - docs August 25, 2026 12:54 View deployment

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 8a4b6f5 to b6f5d60 Compare August 25, 2026 13:02

mintlify Bot deployed to staging - docs August 25, 2026 13:02 View deployment

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from b6f5d60 to f7943e6 Compare August 25, 2026 13:06

mintlify Bot deployed to staging - docs August 25, 2026 13:07 View deployment

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from f7943e6 to 2cdace2 Compare August 25, 2026 13:18

mintlify Bot deployed to staging - docs August 25, 2026 13:20 View deployment

fredericbarthelet approved these changes Aug 26, 2026

View reviewed changes

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 1cfaa1d to 2cdace2 Compare August 26, 2026 08:05

mintlify Bot deployed to staging - docs August 26, 2026 08:05 View deployment

harijoe changed the base branch from main to v2 August 26, 2026 08:10

harijoe added 3 commits August 26, 2026 10:53

refactor!: extract @skybridge/vite-plugin and invert the core dependency

48c0ef8

feat(core): report the bound port over IPC

c6ddc9f

feat(test): add the @skybridge/test eval runner (SKY-546)

8ceb3ae

harijoe force-pushed the julien/sky-546-set-up-the-skybridgeevals-package-beta branch from 2cdace2 to 8ceb3ae Compare August 26, 2026 09:56

harijoe changed the base branch from v2 to julien/sky-546-ipc-port August 26, 2026 09:56

mintlify Bot deployed to staging - docs August 26, 2026 09:56 View deployment

harijoe changed the base branch from julien/sky-546-ipc-port to v2 August 26, 2026 09:57

harijoe merged commit 0afcaf6 into v2 Aug 28, 2026
25 checks passed

harijoe deleted the julien/sky-546-set-up-the-skybridgeevals-package-beta branch August 28, 2026 12:44

harijoe added a commit that referenced this pull request Sep 2, 2026

feat(test): evals (#1058)

de1a3aa

Uh oh!

Conversation

harijoe commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

mintlify Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Uh oh!

Uh oh!

harijoe commented Aug 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fredericbarthelet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fredericbarthelet Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fredericbarthelet left a comment

Choose a reason for hiding this comment

Uh oh!

fredericbarthelet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

harijoe commented Aug 20, 2026 •

edited

Loading

mintlify Bot commented Aug 20, 2026 •

edited

Loading

greptile-apps Bot commented Aug 20, 2026 •

edited

Loading

fredericbarthelet Aug 25, 2026 •

edited

Loading