NVIDIA-Medtech · wyli · Sep 23, 2026 · Sep 23, 2026 · Sep 23, 2026
diff --git a/skills/nv-reason-ct/BENCHMARK.md b/skills/nv-reason-ct/BENCHMARK.md
@@ -0,0 +1,156 @@
+# Skill Benchmark: nv-reason-ct
+
+> ✅ **Overall verdict: PASS — Recommended for publication**
+
+## Publication Recommendation
+
+Recommended for publication based on the completed evaluation evidence in this report.
+
+## Evaluation Metadata
+
+- Skill: `nv-reason-ct`
+- Evaluation date: 2026-09-23
+- Evaluator version: `1.5.6`
+- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
+- Tasks: 9 evaluation tasks (9 positive)
+- Dataset digest: `sha256:df577b70bec14f79cd9d434a2db86fd8aaab1d9fda79314f3d42d6e9494e6ab9` (skill-evaluator-dataset-snapshot/1)
+- Attempts per task: 3
+- Environment: `k8s-sandbox`
+- Tier 2 evidence: required for publication
+- Tier 3 evidence: required for publication
+
+Each task attempt ran in its own isolated sandbox pod.
+
+## What This Report Answers
+
+The three-tier evaluation checks whether the skill:
+
+- is safe to use;
+- produces correct answers;
+- is discovered and activated when needed;
+- helps the agent complete the user's goal and expected workflow; and
+- avoids wasted skill and tool usage.
+
+## Results at a Glance
+
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
+|---|---:|---:|
+| Overall | 90.9% — baseline ran, but no comparable score was available; uplift unavailable | 79.3% — baseline ran, but no comparable score was available; uplift unavailable |
+| Security | 83.3% → 100.0% (+16.7 points) | 63.3% → 72.7% (+9.4 points) |
+| Correctness | 61.7% → 97.8% (+36.1 points) | 41.3% → 83.6% (+42.3 points) |
+| Discoverability | 86.1% — baseline ran, but no comparable score was available; uplift unavailable | 84.6% — baseline ran, but no comparable score was available; uplift unavailable |
+| Effectiveness | 62.2% → 87.4% (+25.2 points) | 47.8% → 75.0% (+27.2 points) |
+| Efficiency | 83.1% — baseline ran, but no comparable score was available; uplift unavailable | 80.6% — baseline ran, but no comparable score was available; uplift unavailable |
+
+**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
+
+Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
+
+A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it.
+
+## Token Usage
+
+Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.
+
+| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
+|---|---|---:|---:|---:|---:|---|
+| claude-code | All cases | 2,732,689 | 3,984,895 | N/A | N/A | skill 9/9; base 12/12 |
+| claude-code | diagnosis-treatment-refusal | 29,965 | 188,796 | -158,831 | -84.13% | skill 1/1; base 1/1 |
+| claude-code | git-lfs-pointer-is-not-volume | 280,162 | 765,739 | -485,577 | -63.41% | skill 1/1; base 1/1 |
+| claude-code | live-abdominal-nifti-routing | 730,528 | 310,202 | +420,326 | +135.50% | skill 1/1; base 1/1 |
+| claude-code | local-trained-export-routing | 340,050 | 192,051 | +147,999 | +77.06% | skill 1/1; base 1/1 |
+| claude-code | model-code-consent-withheld | 364,312 | 534,456 | -170,144 | -31.83% | skill 1/1; base 1/1 |
+| claude-code | no-silent-history-loss | 66,543 | 122,197 | -55,654 | -45.54% | skill 1/1; base 1/1 |
+| claude-code | read-only-setup-check | 337,869 | 289,763 | +48,106 | +16.60% | skill 1/1; base 1/1 |
+| claude-code | training-records-are-not-demo-volumes | 108,138 | 574,305 | N/A | N/A | skill 1/1; base 2/2 |
+| claude-code | upstream-example-end-to-end | 475,122 | 1,007,386 | N/A | N/A | skill 1/1; base 3/3 |
+| codex | All cases | 5,769,477 | 5,913,667 | N/A | N/A | skill 11/11; base 15/15 |
+| codex | diagnosis-treatment-refusal | 29,793 | 17,851 | +11,942 | +66.90% | skill 1/1; base 1/1 |
+| codex | git-lfs-pointer-is-not-volume | 203,536 | 526,324 | -322,788 | -61.33% | skill 1/1; base 1/1 |
+| codex | live-abdominal-nifti-routing | 129,524 | 482,417 | N/A | N/A | skill 1/1; base 3/3 |
+| codex | local-trained-export-routing | 110,176 | 118,719 | -8,543 | -7.20% | skill 1/1; base 1/1 |
+| codex | model-code-consent-withheld | 161,931 | 275,905 | -113,974 | -41.31% | skill 1/1; base 1/1 |
+| codex | no-silent-history-loss | 104,700 | 41,568 | +63,132 | +151.88% | skill 1/1; base 1/1 |
+| codex | read-only-setup-check | 89,420 | 185,848 | -96,428 | -51.89% | skill 1/1; base 1/1 |
+| codex | training-records-are-not-demo-volumes | 4,694,337 | 3,773,895 | +920,442 | +24.39% | skill 3/3; base 3/3 |
+| codex | upstream-example-end-to-end | 246,060 | 491,140 | N/A | N/A | skill 1/1; base 3/3 |
+| ALL AGENTS | Dataset aggregate | 8,502,166 | 9,898,562 | N/A | N/A | skill 20/20; base 27/27 |
+
+Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated.
+
+## Tier Status
+
+| Tier | Purpose | Status | Evidence |
+|---|---|---|---|
+| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 14 finding(s) |
+| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) |
+| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 9 task(s) |
+
+## Findings and Observations
+
+<details>
+<summary>Show detailed findings and successful checks</summary>
+
+- **MEDIUM** SECURITY/subprocess module call (AST4): Dangerous Code Execution:     return subprocess.run(
+        [sys.executable, str(SCRIPT), *map(str, args)],
+        capture_output=True,
+        text=True,
+        timeout=30,
+    ) (`tests/test_nv_reason_ct.py:100`)
+- **MEDIUM** SECURITY/subprocess module call (AST4): Dangerous Code Execution:     result = subprocess.run(
+        command,
+        cwd=output_dir,
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=600,
+    ) (`tests/test_upstream_examples.py:74`)
+- **MEDIUM** SECURITY/Tainted flow: 'env' from os.environ.get (line 70, credential/environment) → subprocess.run (code execution) (TT2): Data Flow:     result = subprocess.run(
+        command,
+        cwd=output_dir,
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=600,
+    ) (`tests/test_upstream_examples.py:74`)
+- **LOW** QUALITY/quality_reliability: Inputs are used but no dedicated Inputs section is documented (`skills/nv-reason-ct/SKILL.md`)
+- **LOW** QUALITY/quality_reliability: Structured output is used but no dedicated Output Format section is documented (`skills/nv-reason-ct/SKILL.md`)
+- 9 additional finding(s) are available in the full evaluation artifacts.
+
+</details>
+
+## Scoring Methodology
+
+<details>
+<summary>Show dimension definitions, source signals, and thresholds</summary>
+
+| Dimension | Question | Scored signals |
+|---|---|---|
+| Security | Is it safe to use? | `security` (100%) |
+| Correctness | Is the answer correct? | `accuracy` (100%) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
+| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |
+
+- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
+- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
+- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
+- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
+- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
+- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.
+
+Signals present in this run:
+
+- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
+- `accuracy` (Accuracy): final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
+- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
+- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).
+
+</details>
+
+## Freshness
+
+Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.
diff --git a/skills/nv-reason-ct/EXAMPLES.md b/skills/nv-reason-ct/EXAMPLES.md
@@ -79,6 +79,8 @@ python -m pytest skills/nv-reason-ct/tests \
 
 Without these options only the three real-volume and two GPU integration cases skip; every offline contract test runs. When requested, missing examples, invalid revisions, or unavailable model assets fail rather than silently skipping or substituting test data. Offline test doubles do not establish live inference success. The local-export loading contract has offline regression coverage; the recorded live baseline in `SKILL.md` used the released-model snapshot, not a newly trained checkpoint.
 
+Passing local tests or reproducing the historical GPU baseline does not establish managed evaluation or signing status. A signed-skill claim requires verification of `skill.oms.sig` against the exact skill contents and the catalog's trusted certificate. That signature covers the skill, not separately downloaded model code or weights, and does not replace the caller's explicit `--trust-model-code` authorization.
+
 ## Cleanup
 
 After inference and when cleanup is authorized, remove only the task-created checkout, environment, and caches. Preserve result JSON, source/content hashes, and setup versions separately; do not purge shared caches or delete a caller-owned input scan.
diff --git a/skills/nv-reason-ct/skill-card.md b/skills/nv-reason-ct/skill-card.md
@@ -0,0 +1,82 @@
+## Description: <br>
+Run NV-Reason-CT inference on user-provided 3D NIfTI chest or abdominal CT volumes for engineering and research workflows. Not for diagnosis, treatment, or clinical reporting. <br>
+
+This skill is for research and development only. <br>
+
+## Owner
+NVIDIA <br>
+
+### License/Terms of Use: <br>
+OpenMDW-1.1 <br>
+## Use Case: <br>
+Developers and engineers running NV-Reason-CT model inference on 3D NIfTI chest or abdominal CT volumes for engineering validation and research workflows. <br>
+
+### Deployment Geography for Use: <br>
+Global <br>
+
+## Requirements / Dependencies: <br>
+**Requires API Key or External Credential:** [Optional] <br>
+**Credential Type(s):** [API key] <br>
+
+Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>
+
+## Known Risks and Mitigations: <br>
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
+Mitigation: Review and scan skill before deployment. <br>
+
+## Reference(s): <br>
+- [NV-Reason-CT Installation Instructions](https://github.com/NVIDIA-Medtech/NV-Reason-CT#installation) <br>
+
+
+## Skill Output: <br>
+**Output Type(s):** [Analysis, JSON] <br>
+**Output Format:** [JSON on stdout] <br>
+**Output Parameters:** [1D] <br>
+**Other Properties Related to Output:** [Structured result including input geometry and hash, anatomy-region selection, response text, runtime identity, dependency versions, and limitations] <br>
+
+## Evaluation Agents Used: <br>
+- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) <br>
+- Codex (`openai/openai/gpt-5.5`) <br>
+
+
+
+## Evaluation Tasks: <br>
+9 evaluation tasks with 3 attempts per task, each in an isolated sandbox pod. Dataset digest: sha256:df577b70bec14f79cd9d434a2db86fd8aaab1d9fda79314f3d42d6e9494e6ab9. <br>
+
+## Evaluation Metrics Used: <br>
+Reported benchmark dimensions: <br>
+- Security: Whether the skill is safe to use, checking for unsafe operations, secret leakage, and unauthorized access. <br>
+- Correctness: Whether the answer produced by the skill-assisted agent is correct against the reference. <br>
+- Discoverability: Whether the right skill was selected and activated when needed, and decoys were avoided. <br>
+- Effectiveness: Whether the skill helped complete the user's goal and followed the expected workflow. <br>
+- Efficiency: Whether the skill avoided wasted tool calls and token usage. <br>
+
+Underlying evaluation signals used in this run: <br>
+- `security`: Unsafe operations, secret leakage, and unauthorized access. <br>
+- `accuracy`: Final-answer correctness against the reference answer. <br>
+- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed. <br>
+- `goal_accuracy`: Whether the user's goal was achieved. <br>
+- `behavior_check`: Whether the expected workflow behavior was followed. <br>
+- `skill_efficiency`: Tool-call productivity. <br>
+- `token_efficiency`: Actual uncached prompt plus completion usage. <br>
+
+
+
+## Evaluation Results: <br>
+| Measure | Claude Code | Codex |
+|---|---:|---:|
+| Overall | 90.9% | 79.3% |
+| Security | 100.0% | 72.7% |
+| Correctness | 97.8% | 83.6% |
+| Discoverability | 86.1% | 84.6% |
+| Effectiveness | 87.4% | 75.0% |
+| Efficiency | 83.1% | 80.6% |
+
+## Skill Version(s): <br>
+ffeef8f (source: git SHA, committed 2026-09-23) <br>
+
+## Ethical Considerations: <br>
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
+
+(For Release on NVIDIA Platforms Only) <br>
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). <br>