Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
42540c2
feat(ai): add gated context selection for web and Slack
adboio Oct 1, 2026
069f393
feat(ai): support context selection in Codex and Pi
adboio Oct 1, 2026
1c879a9
fix(ai): address context selection review findings
adboio Oct 1, 2026
61ef110
chore: merge master into context selection draft
adboio Oct 1, 2026
01623fa
refactor(ai): reuse existing system one configuration
adboio Oct 1, 2026
bd1b628
fix(ai): retain gateway trace in context receipts
adboio Oct 1, 2026
d04acca
feat(context-layer): search context in postgres
adboio Oct 1, 2026
a6aeb75
fix(ai): avoid ambiguous pi context receipts
adboio Oct 2, 2026
72c54f0
fix(ai): preserve normal flow when context selection fails
adboio Oct 2, 2026
3c576b9
chore: merge master into context selection branch
adboio Oct 2, 2026
0463da0
feat(ai): simplify prompt context selection
adboio Oct 2, 2026
a9e0b7b
Merge remote-tracking branch 'origin/master' into codex/context-selec…
adboio Oct 2, 2026
dfa5b97
fix(ai): keep selected context invisible in agent replies
adboio Oct 2, 2026
5cc264b
Merge remote-tracking branch 'origin/master' into codex/context-selec…
adboio Oct 2, 2026
feaa5db
fix(ai): scope context selection to claude and codex
adboio Oct 2, 2026
115ef85
Merge remote-tracking branch 'origin/master' into codex/context-selec…
adboio Oct 2, 2026
ff7e2b3
fix(ai): require fetching selected skills before use
adboio Oct 2, 2026
7fd4bc6
Merge remote-tracking branch 'origin/master' into codex/context-selec…
adboio Oct 2, 2026
191d1fc
fix(ai): address context selection review findings
adboio Oct 2, 2026
68d9f7d
chore: merge master into context selection branch
adboio Oct 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/published/handbook/engineering/ai/sandboxed-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,45 @@ the run's saved `pending_user_message` when logs do not yet contain it. This is
display fallback: it strips context wrappers, gives way to the selected run's log
or stream echo, and never submits the message again.

## Context selection experiment

Staff in `CONTEXT_SELECTION_ALLOWED_TEAM_IDS` can receive hidden organizational context on human prompts in web and Slack cloud runs using Claude or Codex.
The `phai-context-selection` flag selects `control`, `shadow`, or `treatment` using the task ID.
Other flag values disable selection. Local runs, Pi, and other runtime adapters skip it.

Before a human prompt reaches Claude or Codex, the sandbox calls the task-bound selection endpoint.
Autonomous continuations, steering, and slash commands do not trigger selection.

System One first checks whether organizational context could help, using the request and bounded conversation history.
When the gate passes, the selector searches current team skills and semantic catalog rows directly, alongside business knowledge hybrid search.
There is no separate projection or refresh job. Candidate counts are bounded per source.
OAuth scopes, current actor permissions, and shared-context access checks constrain retrieval.
System One reranks candidates concurrently. Sources are checked again before rendering in case definitions or access changed during scoring.
At most five references and 8,000 characters survive into a hidden context block, identified by `selection_id`.
Retrieved content is data to verify through existing tools, rather than instructions or approval.
Skill references contain descriptions, so the prompt requires `skill-get` with the referenced name and version before the agent relies on or follows a skill.
A general knowledge search does not replace that fetch. If the fetch fails or access is denied, the agent must not treat the description as a verified definition or instruction.
This is a prompt requirement, rather than a runtime guarantee that the fetch occurs.
The agent is instructed to use these references silently, including during progress updates, tool-call explanations, and task summaries.
It can cite the underlying sources and explain verification failures, but must not mention selection, suggestions, injection, or the hidden block's metadata.

Control skips retrieval. Shadow records the selected bundle without injecting it. Treatment injects the bundle.
Selection has a three-second budget by default. Saturation, timeout, and selection failures leave the ordinary prompt flow available.
Business Knowledge retrieval uses at most half the budget remaining after local retrieval, leaving time to score ready skills and catalog sources.
Knowledge candidates are ranked anchor passages, without neighbor expansion, and child environments use the canonical project's knowledge and permissions.
Skill access filtering happens before the retrieval limit. Catalog selection skips projects with customized warehouse access and checks system-table denials without building the full catalog.
The agent validates the context budget in Unicode code points, matching the backend renderer.
Native and summary resumes preserve bounded prior conversation history for later turns; a successful `/clear` resets selection history.
Actor refreshes update the selection credential, and cancellation during preparation prevents the prepared prompt from reaching the model.
Selection spans retain scorer error types and candidate identifiers for failed gate or relevance calls.

A best-effort `Context selection` LLM span records the outcome, scores, retrieval time, and exact bounded bundle.
Its `selection_id`, `task_id`, `task_run_id`, and `message_id` connect it to System One calls and the hidden marker in downstream model input.
This span describes prepared context; it does not confirm model acceptance or use.
Offline evals can check downstream inputs, outputs, and tool calls through the existing LLM traces.
Existing trace retention and truncation apply, so absent output or context does not prove the agent ignored it.
No dedicated evidence tables, prompt archive, or dispatch receipts are required, and telemetry failure does not block injection.

## Local development

To set up sandboxed agents for local development:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,7 @@ const taskRunStateFields = {
slack_notified_pr_url: optionalField(z.string()),
slack_thread_url: optionalField(z.string()),
snapshot_kind: optionalField(z.string()),
context_selection_eligible: optionalField(z.boolean()),
store_skills: optionalField(z.array(storeSkillStubSchema)),
token_usage: optionalField(z.record(z.string(), z.unknown())),
} satisfies z.ZodRawShape;
Expand Down
23 changes: 23 additions & 0 deletions packages/agent/packages/agent/src/context-selection/schemas.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import { z } from "zod/v4";

export const contextSelectionResponseSchema = z
.object({
selection_id: z.string().max(128),
context: z.string().refine((value) => Array.from(value).length <= 8_000, {
message: "Context exceeds 8,000 Unicode code points",
}),
mode: z.enum(["disabled", "control", "shadow", "treatment"]),
reason: z.string().max(128),
})
.refine(
(value) =>
!value.context ||
(value.mode === "treatment" && Boolean(value.selection_id)),
{
message: "Only a treatment selection may supply context",
},
);

export type ContextSelectionResponse = z.infer<
typeof contextSelectionResponseSchema
>;
24 changes: 24 additions & 0 deletions packages/agent/packages/agent/src/posthog-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ import {
taskRunStateSchema,
} from "@posthog/agent-contracts";
import packageJson from "../package.json" with { type: "json" };
import {
type ContextSelectionResponse,
contextSelectionResponseSchema,
} from "./context-selection/schemas";
import type { PostHogAPIConfig, StoredEntry, Task, TaskRun } from "./types";
import { getGatewayUsageUrl, getLlmGatewayUrl } from "./utils/gateway";

Expand Down Expand Up @@ -100,6 +104,26 @@ export class PostHogAPIClient {
return this.http.performRequestWithRetry(endpoint, options);
}

async prepareContextSelection(input: {
run_id: string;
message_id: string;
prompt: string;
prompt_char_count: number;
history: string;
history_source: string;
runtime_version: string;
}): Promise<ContextSelectionResponse> {
const response = await this.apiRequest<unknown>(
`/api/projects/${this.getTeamId()}/context_layer/selection/prepare/`,
{
method: "POST",
body: JSON.stringify(input),
signal: AbortSignal.timeout(5_000),
},
);
return contextSelectionResponseSchema.parse(response);
}

async getApiKey(forceRefresh = false): Promise<string> {
return this.http.resolveApiKey(forceRefresh);
}
Expand Down
Loading
Loading