chatgpt-host-conformance

Runs the MCP Apps conformance suite against ChatGPT.

This is what adopting mcp-apps-conformance looks like for one host: a single adapter describing ChatGPT's UI (src/chatgpt-host.ts, ~50 lines) plus a runner entry point. The tests, the suite, the ui:// server and the whole capability protocol come from the package.

Run it

npm install
npx playwright install chromium
npm run conformance

A real Chrome window opens — it is not headless by design (headless Chromium drops the cross-origin MessagePort transfers the MCP Apps handshake needs). The first run needs a one-off manual login:

1. Chrome opens on chatgpt.com; log in.
2. Add the conformance server under Settings → Apps & Connectors: https://mcp-apps-conformance.alpic.live/mcp
3. Re-run. The profile persists in .profile/chatgpt (gitignored — it holds a real session), so later runs are unattended.

The adapter prompts the agent to render the app, then the suite drives itself. You get a per-test breakdown on stdout, a results file in out/, and a non-zero exit on any FAIL or TIMEOUT so this can gate CI.

Serve the suite yourself

Step 2 above uses a hosted server so you can start immediately. To run your own instead — to test an unreleased suite, or to keep everything local — the server ships in the same package:

npm run server          # http://localhost:3000/mcp
npm run server:stdio    # or stdio, for a desktop host's config
PORT=4000 npm run server

Nothing to build: the ui:// TestSuite is bundled inside the package.

⚠️ ChatGPT can't reach localhost. It's a hosted product, so the server URL you register has to be publicly resolvable — expose your local one first:

npm run server &
npx cloudflared tunnel --url http://localhost:3000    # or: ngrok http 3000

then register the tunnel's https://…/mcp URL in Settings → Apps & Connectors. (Desktop hosts like Cursor and Goose reach localhost directly, so they need no tunnel.)

What's in here

File	What
src/chatgpt-host.ts	The only ChatGPT-specific code: how to send a prompt, dismiss the onboarding tooltip, and confirm a turn reached the conversation.
src/run.ts	Wires that adapter into the package's Runner and writes the results.

Three hooks are all the base BrowserHost can't infer — it already launches Chrome, locates the app iframe, performs real cross-origin clicks, inspects frames, reads the console and answers the rest of the protocol. Anything a host genuinely can't do resolves to unsupported, and those tests skip rather than fail.

To point this at a different host, change the three hooks and the two selectors.

Results

Cross-host matrix, including ChatGPT: alpic-ai.github.io/mcp-app-conformance

MIT.