Skip to content

feat: add optional live LLM baseline - #6

Open
xianjianlf2 wants to merge 1 commit into
jbisaccia-9:mainfrom
xianjianlf2:feat/live-llm-baseline-4
Open

feat: add optional live LLM baseline#6
xianjianlf2 wants to merge 1 commit into
jbisaccia-9:mainfrom
xianjianlf2:feat/live-llm-baseline-4

Conversation

@xianjianlf2

Copy link
Copy Markdown

Closes #4

Summary

  • add an optional live extra backed by the OpenAI Responses API
  • run the existing five synthetic attacks against a full record protected only by the existing guard prompt
  • reuse the deterministic suite's attack loader and secret scanner
  • keep live results informational because model behavior is nondeterministic
  • document model selection and disable response storage

The default CI path remains offline and requires no API key.

Verification

  • .venv/bin/python -m pytest -q (6 passed)
  • .venv/bin/python -m permgate suite
  • .venv/bin/python -m permgate gate permission
  • .venv/bin/python -m permgate gate prompt
  • installed .[live] and imported the OpenAI SDK
  • verified missing OPENAI_API_KEY exits with a clear status 2 message

A live request was not sent, so no contributor API key or spend was used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an optional real-LLM naive assistant behind a [live] extra

1 participant