AI Ping

Test whether your "OpenAI-compatible" LLM endpoint is actually compatible.

Many endpoints can return a chat response. Fewer behave like real client integrations expect when streaming, tool calls, error responses, model listing, or the newer Responses API are involved.

Ping AI APIs before your clients do.

Run one command. Get a shareable protocol compatibility report.

npx @starroy/ai-ping check \
  --profile openai-chat \
  --base-url https://your-endpoint.example/v1 \
  --model your-model \
  --html reports/aiping.html

Example result:

Result: OK
Required checks passed, but recommended compatibility gaps were found.
Risk: clients that rely on tool calls or OpenAI-style error handling may break.

Try the Web Playground: demo.aiping.dev

Read this in other languages:

AI Ping is a protocol checker, not a network ping tool, benchmark, load test, uptime monitor, API proxy, chat client, or official certification. It helps you verify whether an AI / LLM API endpoint behaves like the protocol your clients expect.

Example Reports

These sanitized examples show why "it returns text" is not the same as protocol compatibility:

OpenAI-compatible happy path: basic chat, streaming, tool calls, error format, and model listing are usable.
Streaming broken example: basic chat works, but streaming is not compatible with OpenAI-style clients.
Tool calls and error format gap: required checks pass, but clients relying on tool calls or OpenAI-style error metadata may break.

Each example also includes the matching JSON report in docs/examples.

Install

npm install -g @starroy/ai-ping

The CLI package is @starroy/ai-ping; the command is aiping. The reusable core library is @starroy/ai-ping-core.

Desktop Preview

AI Ping v1.4 provides unsigned Tauri Desktop Preview artifacts for macOS, Windows, and Linux through GitHub Releases. The app provides a local UI for choosing a profile, entering a base URL, model, API key, and timeout, running checks, inspecting results, and exporting JSON or HTML reports. The Desktop UI supports English, Simplified Chinese, and Traditional Chinese, detects the initial language from the system locale, and can be switched instantly from the app without saving a language preference.

pnpm install
pnpm desktop:dev
pnpm desktop:build:dmg

The Desktop Preview does not replace or translate CLI output, JSON reports, or HTML reports. It does not save API keys or language preferences, and does not add new checks or profiles. Desktop Preview artifacts are attached to GitHub Releases only and are not published to npm.

The preview builds are unsigned / ad-hoc. Windows may show Microsoft SmartScreen warnings, and Linux behavior depends on the distribution, desktop environment, package format, and WebKitGTK availability. Automatic updates, saved settings, and history are not included. See apps/desktop/README.md for packaging commands, smoke checks, and preview limitations.

Web Playground Preview

AI Ping v1.3 adds a Web Playground Preview for quick browser-based checks of openai-chat and openai-responses endpoints. It runs as a static Vite app: the browser sends requests directly to the endpoint you enter, and AI Ping does not provide a Worker or server proxy.

Open it at demo.aiping.dev.

pnpm install
pnpm web:dev
pnpm web:build

For Cloudflare Pages, use pnpm --filter @starroy/ai-ping-web build as the build command and apps/web/dist as the output directory. No API environment variables are required or expected.

API keys are kept in the current browser tab memory only and clear when the page reloads. AI Ping does not proxy, store, or log browser requests or reports. Because checks run in the browser, endpoint CORS policy applies; if a request is blocked by CORS or local-network browser rules, use the CLI or Desktop version.

See apps/web/README.md for deployment notes and the no-proxy privacy boundary.

Quick Start

aiping profiles

aiping check \
  --profile openai-chat \
  --base-url https://api.openai.com/v1 \
  --model gpt-5.1-mini

Pass API keys with --api-key, AI_PING_API_KEY, or the profile-specific environment variable listed below.

OPENAI_API_KEY=your-key aiping check \
  --profile openai-chat \
  --base-url https://api.openai.com/v1 \
  --model gpt-5.1-mini

Agent Readiness Preview

Chat works. Agents may still break. Agent Readiness Preview is an opt-in openai-chat extension that checks protocol behavior needed by agent workflows.

aiping check \
  --profile openai-chat \
  --base-url https://your-endpoint/v1 \
  --model your-model \
  --agent-readiness \
  --html report.html

The preview currently adds these recommended checks:

openai-chat.agent.tool_choice.named
openai-chat.agent.tool_result.continuation
openai-chat.agent.stream_tool_call.arguments

It is not a benchmark, model intelligence evaluation, provider ranking, or official certification. It does not change default baseline checks unless --agent-readiness is used. Recommended agent check failures do not change the existing required-check exit-code semantics.

Supported Profiles

Profile	Description	Alias	API key environment
`openai-chat`	OpenAI-compatible Chat Completions API	`openai`	`OPENAI_API_KEY`
`openai-responses`	OpenAI-compatible Responses API	-	`OPENAI_API_KEY`
`ollama`	Ollama native API	-	Usually none
`gemini`	Gemini Developer API REST	-	`GEMINI_API_KEY`
`anthropic`	Anthropic Claude Messages API	-	`ANTHROPIC_API_KEY`

The openai alias maps to openai-chat and remains supported in AI Ping 1.x.

Which Profile Should I Use?

Use openai-chat for Chat Completions-compatible endpoints, including local proxies, gateways, and Ollama's OpenAI-compatible /v1/chat/completions.

Use openai-responses for endpoints that implement OpenAI-compatible POST /responses with Responses-style input, output, output_text, and semantic streaming events. AI Ping 1.x does not yet check Responses tools, built-in tools, multimodal input, or conversation state.

Use ollama for Ollama native /api/tags, /api/generate, and /api/chat. Ollama native streaming uses JSON lines, not SSE.

Use gemini for Gemini Developer API REST under a base URL such as https://generativelanguage.googleapis.com/v1beta. Vertex AI Gemini and Gemini OpenAI compatibility are separate APIs.

Use anthropic for Anthropic Claude Messages API under a base URL such as https://api.anthropic.com/v1. Bedrock Anthropic, Vertex AI Anthropic, tool use, extended thinking, and computer use are outside the current 1.x profile.

Reports

AI Ping produces three report forms:

Console report: default human-readable terminal output
JSON report: structured output for issue reports, automation, and CI
HTML report: static report file for sharing, screenshots, and CI artifacts

aiping check \
  --profile openai-chat \
  --base-url http://localhost:3000/v1 \
  --model gpt-5.5 \
  --json

aiping check \
  --profile openai-chat \
  --base-url http://localhost:3000/v1 \
  --model gpt-5.5 \
  --html reports/aiping.html

--html can be combined with --json; stdout remains pure JSON while the HTML file is written separately.

Reports do not include request headers or full API keys by default. Still, review JSON and HTML reports before sharing because endpoint names, model names, base URL hostnames, error bodies, or provider messages may reveal private details.

See Reports for schema notes and safe sharing guidance.

What Does OK Mean?

summary.ok is true when the check run completed and no required checks failed. Recommended checks can fail while the overall result remains OK.

Required checks cover protocol behavior that most clients need to call the endpoint successfully. Recommended checks diagnose useful compatibility details, such as modern OpenAI-compatible tools / tool_calls, but do not fail the overall run while required checks pass.

CLI Commands

aiping --help
aiping check --help
aiping profiles
aiping checks --profile openai-chat
aiping checks --profile openai-chat --agent-readiness

Common aiping check options are stable in 1.x:

--profile
--base-url
--model
--api-key
--timeout
--json
--html
--only
--skip
--agent-readiness
--verbose

Use aiping check --agent-readiness to run the opt-in Agent Readiness Preview. Use aiping checks --agent-readiness to list the preview check IDs alongside the baseline openai-chat checks.

Limit or skip checks with comma-separated check IDs:

aiping check \
  --profile openai-chat \
  --base-url http://localhost:3000/v1 \
  --model gpt-5.5 \
  --only openai-chat.chat.basic,openai-chat.chat.stream

Exit Codes

Exit code	Meaning
`0`	Check completed and required checks passed
`1`	Check completed but at least one required check failed
`2`	CLI usage or configuration error
`3`	Unexpected runtime error

Compatibility Promise

AI Ping 1.x aims to keep CLI commands, stable check options, core report fields, profile names, the openai alias, npm package names, and exit code semantics backward-compatible.

The stable report fields are version, profile, endpoint, model, startedAt, durationMs, summary, and results. Minor versions may add profiles, checks, output options, or optional report fields without removing or renaming stable fields.

See Compatibility Policy for the full 1.x promise.

CI Usage

name: AI Ping
on:
  pull_request:
jobs:
  ai-ping:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g @starroy/ai-ping
      - run: |
          mkdir -p reports
          aiping check \
            --profile openai-chat \
            --base-url http://localhost:3000/v1 \
            --model test-model \
            --html reports/aiping.html
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: ai-ping-report
          path: reports/aiping.html

Troubleshooting

Most failures come from one of four places: the wrong profile, a base URL with the wrong prefix, an API key that is missing or sent with the wrong header, or a provider that implements only part of the target protocol.

See Troubleshooting before filing an issue.

Core Usage

import { runChecks } from "@starroy/ai-ping-core";

const report = await runChecks({
  profile: "openai-chat",
  baseUrl: "http://localhost:3000/v1",
  apiKey: "sk-test",
  model: "gpt-5.5",
});

console.log(report.summary.ok);
console.log(report.results);

runChecks returns a structured report. It does not print terminal output, write files, set exit codes, or manage UI state.

Development

pnpm install
pnpm test
pnpm typecheck
pnpm build
pnpm desktop:dev

See CONTRIBUTING.md for development and pull request guidelines.

Security

Do not post API keys, Authorization headers, private endpoint secrets, private base URLs, or sensitive JSON / HTML reports in public issues. See SECURITY.md for details.

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
.github		.github
apps		apps
article-assets		article-assets
docs		docs
examples/openai-compatible-mock		examples/openai-compatible-mock
packages		packages
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.base.json		tsconfig.base.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

AI Ping

Example Reports

Install

Desktop Preview

Web Playground Preview

Quick Start

Agent Readiness Preview

Supported Profiles

Which Profile Should I Use?

Reports

What Does OK Mean?

CLI Commands

Exit Codes

Compatibility Promise

CI Usage

Troubleshooting

Core Usage

Development

Security

About

Uh oh!

Releases 6

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

AI Ping

Example Reports

Install

Desktop Preview

Web Playground Preview

Quick Start

Agent Readiness Preview

Supported Profiles

Which Profile Should I Use?

Reports

What Does OK Mean?

CLI Commands

Exit Codes

Compatibility Promise

CI Usage

Troubleshooting

Core Usage

Development

Security

About

Topics

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 6

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages