[FEAT]: Deprecate omitted `observability_level` across public APIs

[mahdi-al-hakim]

Summary

Deprecate omitting observability_level on every public API that defaults it, so a caller that misses the argument gets a signal instead of silently keeping the old behavior.

Follow-up to a review comment on #141, filed at Nina Chikanov (@nina-msft)'s request and rescoped to the feedback below.

Motivation

#141 added observability_level to evaluate_turn_async with a compatibility default:

observability_level: ObservabilityLevel = ObservabilityLevel.TOOL_AND_SIDE_EFFECTS,

That default keeps every existing caller working, which is the right call for that PR. The cost is that it fails open. A caller that omits the argument treats every adapter as fully observable, which is the bug #140 describes: an evaluator that needs an evidence channel the adapter does not report returns NOT_DETECTED, and resolve_as_attack turns that into SAFE.

The same default is on EvalContext and on EvalContext.from_response. Result defaults the same field the other way, to RESPONSE_ONLY, where it is reporting metadata that every built-in path overwrites with the adapter's actual profile. evaluate_turn_async is exported from rampart.core and docs/contributing/extending-rampart.md presents it as the seam for a custom execution strategy, so out-of-repo callers are the ones most likely to miss it.

No shipped code relies on any of these defaults: every Result and EvalContext built in rampart/ passes observability_level explicitly. The repo's own tests are another matter. Counting the construction sites by AST, 83 calls under tests/ omit the argument: 64 Result, 13 EvalContext, 3 EvalContext.from_response and 3 evaluate_turn_async, including test_observability_level_defaults_to_no_declared_limit, which asserts the very default the warning would be about. Those would need the argument or an explicit filter before the warning lands.

Proposed solution

Making observability_level required is out of scope. The deprecation is:

1. After [BREAKING] [FIX]: Return UNDETERMINED when an evaluator cannot observe the evidence it needs #141 merges, inventory every public API that defaults observability_level: evaluate_turn_async, direct EvalContext construction, EvalContext.from_response, and direct Result construction.
2. When the argument is omitted, emit an actionable DeprecationWarning that names the affected API, explains the compatibility fallback being applied, directs the caller to pass the adapter's declared level, normally adapter.observability_profile, and names the removal release.
3. Warn on omission only. Passing any ObservabilityLevel explicitly, including the value that is the current default, must not warn.
4. Emit exactly one warning per omitted public call, including wrapper paths such as EvalContext.from_response.
5. Preserve the current fallback behavior for the whole deprecation window.
6. Name the removal release under the existing two-minor-version policy in docs/contributing/release-process.md: a warning shipped in 0.x.0 targets 0.(x+2).0.
7. Update the docs and the generated reference: docs/contributing/extending-rampart.md, the direct Result(...) example in docs/usage/pytest-integration.md, the Result(...) helper in docs/contributing/testing.md, the EvalContext, Result and evaluate_turn_async docstrings, and add evaluate_turn_async to the rampart.core.execution members on docs/api/core-protocols.md, where it is public but absent.
8. Add tests proving an omitted argument warns once and an explicit value does not warn at all.

Alternatives considered

• Leave the defaults. Cheapest, but the failure is silent and points the wrong way for a safety framework.
• Make the argument required now. A breaking API change with no migration window, and out of scope per the review discussion.
• Default EvalContext to RESPONSE_ONLY to match Result. Fails safe, but it changes behavior for every caller that omits the argument, with no migration window, and makes a fully observable adapter look unobservable, which is its own false signal.
• Infer the level inside evaluate_turn_async. It receives the evaluator and the turn data, not the adapter, so there is nothing to infer from.

Additional context

• [BREAKING] [FIX]: Return UNDETERMINED when an evaluator cannot observe the evidence it needs #141 and [BUG]: A safety test passes against an agent that exfiltrated, because the adapter cannot observe side effects #140 for the original problem, and the review comment that asked for this: [BREAKING] [FIX]: Return UNDETERMINED when an evaluator cannot observe the evidence it needs #141 (comment)
• rampart/common/deprecation.py already provides emit_deprecation_warning(old_item=, new_item=, removed_in=), used for the rampart_sinks fixture with removed_in="0.3.0". Its message maps an old item to a replacement, so an omitted argument needs different wording, and the helper may need a second entry point.
• Telling omission apart from an explicit value that happens to equal the default means the parameter and the dataclass fields need a sentinel default rather than an ObservabilityLevel member, with the real fallback applied after the check. That is the part worth agreeing on before implementation.
• Happy to take this one if you would like it picked up.

[nina-msft]

Thanks for opening this Mahdi Alhakim (@mahdi-al-hakim) - it is a bit more scoped than I intended. Would you mind revising this to cover every public API where observability_level has a default after #141 merges?

That includes not only evaluate_turn_async, but also direct EvalContext construction, EvalContext.from_response, and direct Result construction. Each should emit a DeprecationWarning only when the argument is omitted; explicitly passing any ObservabilityLevel, including the current default value, should not warn.

Documentation should cover:

• docs/contributing/extending-rampart.md
• direct Result(...) examples in docs/usage/pytest-integration.md
• the Result(...) helper in docs/contributing/testing.md
• generated API documentation/docstrings for EvalContext, Result, and evaluate_turn_async
• adding evaluate_turn_async to the core API reference, since it is public but currently absent from that page

The change to make observability_level required is intentionally out of scope. The warnings should identify the release in which the defaults will become required, following the existing two-minor-version policy: a warning introduced in 0.x.0 targets 0.(x+2).0.

Suggested title: "[FEAT]: Deprecate omitted observability_level across public APIs"

Revised proposed solution

1. After [BREAKING] [FIX]: Return UNDETERMINED when an evaluator cannot observe the evidence it needs #141 merges, inventory all public APIs with an observability_level default.

2. When the argument is omitted, emit an actionable DeprecationWarning that:

   • names the affected API;
   • explains the compatibility fallback being applied;
   • directs callers to pass the adapter’s declared observability level, normally adapter.observability_profile;
   • names the removal release, two minor versions after the warning first ships.

3. Preserve current fallback behavior during the deprecation window.

4. Emit exactly one warning per omitted public call, including wrapper paths such as EvalContext.from_response.

5. Update extension guidance, direct-construction examples, docstrings, and API reference pages.

6. Add tests proving omitted arguments warn and explicit values do not.

[mahdi-al-hakim]

Rewritten to that scope: the APIs you listed, warn only on omission, one warning per omitted public call, current fallback preserved, and 0.x.0 targeting 0.(x+2).0 per release-process.md. Title is now yours. Motivation now counts the in-repo call sites that omit the argument, since the warning would land on the test suite, and there are two implementation notes at the end.

[mahdi-al-hakim]

Superseded by #141, per Nina Chikanov (@nina-msft)'s review there: observability_level is now required on evaluate_turn_async, EvalContext, EvalContext.from_response and Result rather than deprecated, so there is no deprecation window left to document. The doc items from this issue moved with it, including adding evaluate_turn_async to the API reference.

#141 closes this on merge. Leaving it open until then.