feat(aio): add system one boolean judges with shared egress - #103752
bernatixer wants to merge 36 commits into
Conversation
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
🤖 CI report
|
| Function | Location | Complexity | Limit |
|---|---|---|---|
AIObservabilityEvaluation |
products/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsx:78 |
115 | 10 |
EditKeyModal |
products/ai_observability/frontend/settings/LLMProviderKeysSettings.tsx:424 |
26 | 10 |
AddKeyModal |
products/ai_observability/frontend/settings/LLMProviderKeysSettings.tsx:169 |
19 | 10 |
normalizeLLMProvider |
products/ai_observability/frontend/settings/llmProviderKeysLogic.ts:68 |
16 | 10 |
normalizeEvaluationResultProperties |
products/ai_observability/frontend/utils.ts:1188 |
16 | 10 |
queryEvaluationRunsStats |
products/ai_observability/frontend/utils.ts:1357 |
16 | 10 |
getGeminiInlineData |
products/ai_observability/frontend/utils.ts:838 |
15 | 10 |
looksLikeXml |
products/ai_observability/frontend/utils.ts:1074 |
15 | 10 |
isToolStepItem |
products/ai_observability/frontend/utils.ts:628 |
13 | 10 |
isAnthropicImageMessage |
products/ai_observability/frontend/utils.ts:798 |
13 | 10 |
isAnthropicDocumentMessage |
products/ai_observability/frontend/utils.ts:816 |
13 | 10 |
parseToolArgumentsForDisplay |
products/ai_observability/frontend/utils.ts:967 |
13 | 10 |
EvaluationRunsTable |
products/ai_observability/frontend/evaluations/components/EvaluationRunsTable.tsx:43 |
12 | 10 |
handleSubmit |
products/ai_observability/frontend/settings/LLMProviderKeysSettings.tsx:450 |
12 | 10 |
formatLLMEventTitle |
products/ai_observability/frontend/utils.ts:1026 |
12 | 10 |
queryEvaluationRuns |
products/ai_observability/frontend/utils.ts:1276 |
12 | 10 |
getKeyPlaceholder |
products/ai_observability/frontend/settings/LLMProviderKeysSettings.tsx:90 |
11 | 10 |
DeleteKeyModal |
products/ai_observability/frontend/settings/LLMProviderKeysSettings.tsx:606 |
11 | 10 |
LLMProviderKeysSettings |
products/ai_observability/frontend/settings/LLMProviderKeysSettings.tsx:721 |
11 | 10 |
isOpenAIFileMessage |
products/ai_observability/frontend/utils.ts:771 |
11 | 10 |
✅ Duplication (Python) — clean
New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.
✅ Duplication (TypeScript) — clean
New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.
⚠️ Bundle size — 🔺 +4.8 KiB (+0.0%)
Uncompressed size of every built .js bundle, compared against the base branch.
Total: 68.82 MiB · 🔺 +4.8 KiB (+0.0%)
| File | Size | Δ vs base |
|---|---|---|
render-query/src/render-query/render-query.js |
20.24 MiB | 🔺 +1.6 KiB (+0.0%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.57 MiB · 22 files | no change | █████████░ 85.1% of 1.84 MiB |
logged-out boot: index + App + bootApp (preloaded by every page, including /login)src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts |
3.57 MiB · 628 files | 🔺 +76 B (+0.0%) | █████████░ 88.6% of 4.03 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
7.27 MiB · 2,301 files | 🔺 +408 B (+0.0%) | █████████░ 87.2% of 8.34 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 src/layout/navigation-3000/navigationLogic.tsx stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 src/scenes/dashboard/dashboardLogic.tsx stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 src/lib/lemon-ui/LemonMarkdown/ stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 src/lib/components/RichContentEditor/ stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 src/lib/components/CodeSnippet/ stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 src/taxonomy/core-filter-definitions-by-group.json stays out of src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/scenes/session-recordings/player/sessionRecordingPlayerLogic.ts stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/index.tsx |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 854 B | src/scenes/ChunkLoadErrorBoundary.tsx |
Largest files eagerly shipped from src/index.tsx + src/scenes/App.tsx + src/scenes/bootApp.ts
| Size | File |
|---|---|
| 301.8 KiB | ../node_modules/.pnpm/posthog-js@1.434.13_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.mjs |
| 267.6 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 100.4 KiB | src/lib/api.ts |
| 84.9 KiB | src/products.tsx |
| 69.1 KiB | src/lib/lemon-ui/icons/icons.tsx |
| 63.9 KiB | src/lib/utils/eventUsageLogic.ts |
| 38.7 KiB | ../node_modules/.pnpm/@dnd-kit+core@6.0.8_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@dnd-kit/core/dist/core.esm.js |
| 33.9 KiB | ../node_modules/.pnpm/kea@4.0.0-pre.6_patch_hash=139b8d1f1304f9d9da452a9a1244c94ea679dbcb85687d8999563146879fb6f5_react@18.3.1/node_modules/kea/lib/index.cjs.js |
| 28.3 KiB | src/scenes/scenes.ts |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 301.8 KiB | ../node_modules/.pnpm/posthog-js@1.434.13_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.mjs |
| 272.0 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 267.6 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 153.7 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 100.4 KiB | src/lib/api.ts |
| 98.5 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
| 84.9 KiB | src/products.tsx |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.37 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.37 MiB · 19 files | 🔺 +76 B (+0.0%) | ████░░░░░░ 41.4% of 5.72 MiB |
| Deferred (lazy) | 2.10 MiB · 44 files | 🔺 +332 B (+0.0%) | n/a — loads on demand |
Loader dist/toolbar.js |
1.2 KiB | no change | █░░░░░░░░░ 6.0% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 790.2 KiB | dist/toolbar/toolbar-app-ZNXS4H6W.css |
| 650.5 KiB | dist/toolbar/chunk-chunk-SCJBJUCD.js |
| 483.6 KiB | dist/toolbar/chunk-chunk-DPLA2GAY.js |
| 138.3 KiB | dist/toolbar/chunk-chunk-VUAOBILF.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-FDH2IBXT.js |
| 75.2 KiB | dist/toolbar/toolbar-app-X75LXL7Z.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-TSAL54PB.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-ECAITW5Z.js |
| 21.0 KiB | dist/toolbar/chunk-chunk-P7KVYVSM.js |
| 6.8 KiB | dist/toolbar/chunk-chunk-DV7IWQNF.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +79.9 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 943.60 MiB · 🔺 +79.9 KiB (+0.0%)
ℹ️ MCP UI apps size — 33 app(s), 17628.2 KB JS
Built size of each MCP UI app (main.js + styles.css).
| App | JS | CSS |
|---|---|---|
| debug | 597.8 KB | 196.2 KB |
| action | 454.1 KB | 196.2 KB |
| action-list | 564.1 KB | 196.2 KB |
| cohort | 453.1 KB | 196.2 KB |
| cohort-list | 563.1 KB | 196.2 KB |
| email-template | 452.9 KB | 196.2 KB |
| error-details | 469.1 KB | 196.2 KB |
| error-issue | 453.8 KB | 196.2 KB |
| error-issue-list | 564.0 KB | 196.2 KB |
| experiment | 561.2 KB | 196.2 KB |
| experiment-list | 564.9 KB | 196.2 KB |
| experiment-results | 566.2 KB | 196.2 KB |
| feature-flag | 566.7 KB | 196.2 KB |
| feature-flag-list | 570.5 KB | 196.2 KB |
| feature-flag-testing | 457.3 KB | 196.2 KB |
| inline-scan | 453.6 KB | 196.2 KB |
| insight-actors | 562.3 KB | 196.2 KB |
| invite-email-preview | 452.3 KB | 196.2 KB |
| llm-costs | 559.3 KB | 196.2 KB |
| session-recording | 454.9 KB | 196.2 KB |
| survey | 454.7 KB | 196.2 KB |
| survey-global-stats | 561.8 KB | 196.2 KB |
| survey-list | 564.8 KB | 196.2 KB |
| survey-stats | 561.8 KB | 196.2 KB |
| trace-span | 453.5 KB | 196.2 KB |
| trace-span-list | 564.0 KB | 196.2 KB |
| vision-observation-list | 563.2 KB | 196.2 KB |
| workflow | 453.4 KB | 196.2 KB |
| workflow-list | 563.5 KB | 196.2 KB |
| loops-review | 457.8 KB | 196.2 KB |
| query-results | 774.0 KB | 196.2 KB |
| render-ui | 856.5 KB | 196.2 KB |
| visual-review-snapshots | 457.9 KB | 196.2 KB |
⚠️ MCP snapshots — 3 updated (3 modified, 0 added, 0 deleted)
Snapshots: MCP unit test snapshots updated
Changes: 3 snapshots (3 modified, 0 added, 0 deleted)
What this means:
- Snapshots have been automatically updated to match current output
Next steps:
- Review the changes to ensure they're intentional
- If unexpected, investigate what caused the output to change
ℹ️ Docs preview — preview build triggered
Docs from this PR will be published at posthog.com.
| Project | Preview | Updated (UTC) |
|---|---|---|
| posthog.com | Open preview | Sep 21, 2026, 1:19 PM |
The preview should be ready in about 10 minutes. Open the preview at /handbook/engineering/.
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe changes add System One as a provider for boolean evaluations. They add connection configuration, endpoint and response validation, probability-based verdicts, retry and rejection handling, and probability event data. The frontend adds connection settings, separates evaluation models from generative BYOK models, and displays probabilities in evaluation details. Provider schemas, tests, pricing, and internal documentation are also updated. Priority: ⬇️ Low Merge Risk: 🟠 High · up to Do not merge until TypeSafe evaluations enforce the required data-use controls. Rejected requests and timeouts can also interrupt evaluations, and the remaining configuration and display concerns need resolution. Security Architecture ReviewSecurity architecture risk: 🟠 High · up to The new evaluation path can send customer content to an external provider, but the required controls for enabling that data sharing are not established in the reviewed path. Endpoint protections and connection validation reduce other risks, but do not address that approval boundary. Retained concerns
Security review detailsSecurity Blast Radius
Security Findings and Attack Paths
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 1✅ Passed checks (1 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
bernatixer
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: aec83079-1dab-4875-86b3-07e43d775502
⛔ Files ignored due to path filters (3)
products/ai_observability/frontend/generated/api.schemas.tsis excluded by!**/generated/**products/ai_observability/frontend/generated/api.zod.tsis excluded by!**/generated/**services/mcp/src/generated/ai_observability/api.tsis excluded by!**/generated/**
📒 Files selected for processing (36)
docs/internal/ai-observability-judge-inputs.mdposthog/settings/web.pyposthog/temporal/ai_observability/eval_reports/report_agent/prompts.pyposthog/temporal/ai_observability/evaluation_errors.pyposthog/temporal/ai_observability/evaluation_llm_judge.pyposthog/temporal/ai_observability/evaluation_types.pyposthog/temporal/ai_observability/evaluation_workflow_activities.pyposthog/temporal/ai_observability/test_run_evaluation.pyproducts/ai_observability/backend/api/evaluation_config.pyproducts/ai_observability/backend/api/evaluations.pyproducts/ai_observability/backend/api/provider_keys.pyproducts/ai_observability/backend/api/proxy.pyproducts/ai_observability/backend/api/taggers.pyproducts/ai_observability/backend/api/test/test_evaluations.pyproducts/ai_observability/backend/api/test/test_provider_keys.pyproducts/ai_observability/backend/llm/client.pyproducts/ai_observability/backend/llm/system_one.pyproducts/ai_observability/backend/llm/test/test_system_one.pyproducts/ai_observability/backend/models/model_configuration.pyproducts/ai_observability/backend/models/provider_keys.pyproducts/ai_observability/frontend/ConversationDisplay/EvaluationDisplay.tsxproducts/ai_observability/frontend/components/GenerationEvalRunsTable.tsxproducts/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsxproducts/ai_observability/frontend/evaluations/components/EvaluationRunsTable.tsxproducts/ai_observability/frontend/evaluations/types.tsproducts/ai_observability/frontend/modelPickerLogic.test.tsproducts/ai_observability/frontend/settings/LLMProviderKeysSettings.tsxproducts/ai_observability/frontend/settings/SystemOneConnectionFields.stories.tsxproducts/ai_observability/frontend/settings/SystemOneConnectionFields.tsxproducts/ai_observability/frontend/settings/llmProviderKeysLogic.test.tsproducts/ai_observability/frontend/settings/llmProviderKeysLogic.tsproducts/ai_observability/frontend/utils.test.tsproducts/ai_observability/frontend/utils.tsservices/mcp/src/api/generated.tsservices/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.jsonservices/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json
Included review availability: Your plan provides up to 12 included reviews per hour; 7 remain after this review.
bernatixer
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments (1)
products/ai_observability/backend/llm/system_one.py-160-171 (1)
160-171: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winClassify 408 as a temporary failure, not a permanent rejection.
At Line 168, the catch-all branch sends HTTP 408 (Request Timeout) to
SystemOneRequestRejectedError. The PR says a rejected request stops the evaluation and requires connection revalidation. A 408 means the request timed out, and proxies and load balancers can return it for temporary reasons. With the current mapping, one timed-out request stops a healthy evaluation until a user revalidates the connection. Add 408 to the retryable branch.🐛 Proposed fix
- if response.status_code in (429, 503, 529): + if response.status_code in (408, 429, 503, 529): raise SystemOneRateLimitError(response.headers.get("Retry-After"))Based on learnings: "Treat HTTP 429, 529, 5xx, and network/transient failures as retryable; treat ... permanent client errors (400, 401, 403, 404, 413) as non-retryable."
Source: Learnings
🧹 Nitpick comments (1)
products/ai_observability/frontend/settings/SystemOneConnectionFields.tsx (1)
15-15: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a
data-attrto the advanced-configuration toggle.The
<summary>is the interactive control for the connection fields. Give it a stable kebab-casedata-attrfor autocapture and Playwright selectors. As per coding guidelines, “New buttons and key interactive elements get a kebab-casedata-attr.”Source: Coding guidelines
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: 3ddc84af-4c67-46dc-be18-723a6e2f4942
⛔ Files ignored due to path filters (3)
products/ai_observability/frontend/generated/api.schemas.tsis excluded by!**/generated/**products/ai_observability/frontend/generated/api.zod.tsis excluded by!**/generated/**services/mcp/src/generated/ai_observability/api.tsis excluded by!**/generated/**
📒 Files selected for processing (29)
docs/internal/ai-observability-judge-inputs.mdposthog/settings/web.pyposthog/temporal/ai_observability/eval_reports/report_agent/prompts.pyposthog/temporal/ai_observability/evaluation_llm_judge.pyposthog/temporal/ai_observability/test_run_evaluation.pyproducts/ai_observability/backend/api/evaluations.pyproducts/ai_observability/backend/api/proxy.pyproducts/ai_observability/backend/api/test/test_evaluations.pyproducts/ai_observability/backend/llm/system_one.pyproducts/ai_observability/backend/llm/test/test_system_one.pyproducts/ai_observability/backend/models/evaluation_configs.pyproducts/ai_observability/backend/models/provider_keys.pyproducts/ai_observability/backend/models/test/test_evaluation_configs.pyproducts/ai_observability/frontend/components/EvaluationExplanation.tsxproducts/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsxproducts/ai_observability/frontend/evaluations/components/NumericEvaluationConfig.stories.tsxproducts/ai_observability/frontend/evaluations/components/NumericEvaluationConfig.test.tsxproducts/ai_observability/frontend/evaluations/components/NumericEvaluationConfig.tsxproducts/ai_observability/frontend/evaluations/constants.tsproducts/ai_observability/frontend/evaluations/llmEvaluationLogic.test.tsproducts/ai_observability/frontend/evaluations/llmEvaluationLogic.tsproducts/ai_observability/frontend/modelPickerLogic.test.tsproducts/ai_observability/frontend/settings/SystemOneConnectionFields.tsxproducts/ai_observability/frontend/settings/llmProviderKeysLogic.test.tsproducts/ai_observability/frontend/settings/llmProviderKeysLogic.tsservices/mcp/src/api/generated.tsservices/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.jsonservices/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-test-hog.jsonservices/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json
Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.
bernatixer
changed the title
bernatixer
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments (1)
services/mcp/src/api/generated.ts-38325-38325 (1)
38325-38325: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCorrect the
/v1guidance for custom endpoints.Both descriptions imply that every System One base URL must include
/v1. The client accepts other HTTPS paths and appends/systemone;/v1is part of the TypeSafe default, not a universal requirement. A client following this guidance can submit the wrong URL and fail connection validation. Update the serializer help text and regenerate both descriptions. (raw.githubusercontent.com)Also applies to: 73670-73670
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: e7c9dd2c-5b92-496e-820e-641522966aa9
⛔ Files ignored due to path filters (3)
products/ai_observability/frontend/generated/api.schemas.tsis excluded by!**/generated/**products/ai_observability/frontend/generated/api.zod.tsis excluded by!**/generated/**services/mcp/src/generated/ai_observability/api.tsis excluded by!**/generated/**
📒 Files selected for processing (12)
docs/internal/ai-observability-judge-inputs.mdposthog/egress/test/test_typesafe.pyposthog/egress/typesafe/README.mdposthog/egress/typesafe/client.pyposthog/egress/typesafe/limiter.pyposthog/egress/typesafe/transport.pyposthog/temporal/ai_observability/evaluation_llm_judge.pyposthog/temporal/ai_observability/test_run_evaluation.pyproducts/ai_observability/backend/api/test/test_provider_keys.pyproducts/ai_observability/backend/llm/system_one.pyproducts/ai_observability/backend/llm/test/test_system_one.pyservices/mcp/src/api/generated.ts
Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.
|
👋 Visual changes detected for this PR. Review and approve in PostHog Visual Review If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix. |
This branch has not been deployed
1 participant
Problem
AI observability users need to run boolean evaluations against a System One-compatible model they configure themselves.
This PR builds on #106735, which provides the shared protocol types, parser, and HTTP client.
Changes
Before:
flowchart LR A[LLM judge evaluation] --> B[Completion provider] --> C[Boolean or numeric result] classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000; classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff; class A,C phYellow; class B phRed;After:
flowchart LR A[LLM judge evaluation] --> B[Completion provider] --> C[Boolean or numeric result] A --> D[System One connection] --> E[Shared egress client] --> F[Compatible endpoint] --> G[Boolean result and probability] classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000; classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff; classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff; class A,C,G phYellow; class B,F phRed; class D,E phBlue;The connection form at 520px replaces the earlier screenshots that showed hosted defaults:
How did you test this code?
👉 Stay up-to-date with PostHog coding conventions for a smoother review.
Release status
The
llm-analytics-system-one-evaluationsflag defaults off. Deploy ingestion and evaluation workers before enabling it.Automatic notifications
Docs update
Updated AI observability judge inputs with connection settings, result semantics, and compatibility limits.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Agent: Codex, GPT-6
The existing CodeRabbit CLI pass identified rejected requests classified as unreadable output; the evaluation error handling and regression tests address it.
CodeRabbit's GitHub review also identified shared-key invalidation and endpoint-controlled metric labels. Both are addressed in the current code.
Public examples and screenshots use invented values. The session has no shareable link.
Tools: shell, Git, GitHub CLI, hogli, Playwright, and web documentation.
Repository skills: stacking-prs, writing-pr-descriptions, running-ci-preflight, reviewing-with-coderabbit, routing-outbound-api-calls, writing-tests, writing-ui-components, writing-user-facing-copy, and adopting-generated-api-types.
Public skills: Boffin, Ponytail, and Superpowers brainstorming, test-driven-development, and verification-before-completion.