[FEAT] Resolve Key Vault-Backend Environment References and Update .env_example

[Victor Valbuena (ValbuenaVC)]

Description

This PR makes Azure Key Vault the canonical source for shared and deployed environment configuration while preserving process variables, explicit dotenv files, and a two-version migration path for legacy ~/.pyrit/.env. It also updates .env_example and adds an integration test to monitor drift between environment variables referenced in the repo and those mentioned in .env_example.

Configuration

env_akv_ref:
  - https://my-vault.vault.azure.net/secrets/pyrit-env
env_akv_strict: true
env_akv_write_env: false

env_akv_strict raises on an unresolved or malformed secret when true, and env_akv_write_env writes the fully resolved .env with secrets to disk for debugging purposes.

Source Precedence

Sources load in this order:

1. Existing process environment (unchanged)
2. Key Vault bootstrap documents in env_akv_ref order
3. Legacy .env or ordinary explicit env_files
4. Files named .env.local

Process values are retained. Key Vault, legacy .env, and ordinary explicit files fill only missing values. Only a file named .env.local overrides existing values.

When Key Vault is configured:

• Auto-discovered ~/.pyrit/.env is ignored and emits a deprecation warning. In a later version of PyRIT, this will raise an error.
• ~/.pyrit/.env.local still loads as the supported local override.
• Explicit env_files remain supported regardless of filename or location.

Auto-discovered .env remains available as a legacy source until PyRIT 1.3.0. Users should migrate shared configuration to Key Vault and use .env.local for temporary plaintext overrides or cases where Azure is unavailable.

Key Vault References

Bootstrap documents and local dotenv files support complete-value Key Vault references:

ENDPOINT="https://example.openai.azure.com/openai/v1"
MODEL="${PYRIT_MODEL}"
API_KEY="kv:https://my-vault.vault.azure.net/secrets/api-key"
PINNED_KEY="kv:https://my-vault.vault.azure.net/secrets/api-key/version-id"

kv: is canonical. akv:, azure_key_vault:, and env_akv_ref: are valid compatibility aliases.

Remote bootstrap references must target the bootstrap document's vault. Local files may reference any validated supported Key Vault URL. References resolve one hop, so fetched child values are terminal (you can't chain secret references across key vaults).

Supported vault DNS suffixes:

• .vault.azure.net
• .vault.azure.cn
• .vault.usgovcloudapi.net

Strict mode (env_akv_strict) rejects malformed bootstrap entries and malformed Key Vault references. Non-strict mode warns and skips malformed references while preserving the previous value. Authentication, authorization, transport, missing-secret, and empty-value failures always raise KeyVaultInitializationException.

Key Vault clients use asynchronous retries with exponential backoff.

Debug .env Output

Setting env_akv_write_env: true writes a fully resolved ~/.pyrit/.env for debugging.

The generated file:

• Is always named .env
• Contains only bootstrap assignments, comments, and resolved child-secret values
• Excludes unrelated process and .env.local values
• Preserves dotenv ordering and interpolation behavior
• Safely round-trips multiline values, quotes, and terminal ${NAME} text
• Uses a restricted temporary file and atomic replacement
• Rejects symbolic-link destinations
• Uses owner-only permissions where supported

If ~/.pyrit/.env already exists, initialization fails before fetching Key Vault secrets and instructs the user to rename or remove it.

The generated file contains plaintext secrets and should be removed after debugging.

.env_example

.env_example now:

• Places alias sources before their ${NAME} aliases
• Removes pre-existing documentation-style < > URL wrappers because python-dotenv preserves them literally
• Keeps consecutive comment lines together without blank lines (this is unchanged, but flagging because some editors automatically space them out)
• Contains 199 unique assignments with no duplicates
• Retains endpoint paths unchanged

Public consistency tests verify repository references, URL formatting, comment formatting, and clean-environment alias resolution.

Integration-Test Scope

This PR removes the live env-new Key Vault schema test and its Azure DevOps environment wiring from previous iterations of the PR.

Key Vault behavior is covered through mocked unit tests. Public integration coverage is limited to .env_example and PyRIT consistency. Cross-store drift and operational alerting belong in a separate internal pipeline.

The remaining legacy env-global pipeline write is now performed atomically with restrictive directory and file permissions.

Validation

• Full unit suite: 15,643 passed, 8 skipped
• Affected setup suite: 178 passed, 2 skipped
• .env_example consistency tests: 4 passed
• Documentation validation: passed
• YAML validation: passed
• Ruff formatting and lint: passed
• ty type checking: passed
• Async-suffix, whitespace, EOF, private-key, large-file, and documentation pre-commit hooks: passed
• Diff checks: passed

[Copilot]

Pull request overview

Adds Azure Key Vault-backed environment bootstrapping with recursive reference resolution, precedence handling, warnings, and documentation.

Changes:

• Resolves env:, kv:, aliases, and escaped literals.
• Adds environment-source validation and AKV/local-file precedence.
• Expands tests and configuration documentation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
pyrit/setup/initialization.py	Implements AKV loading and reference resolution.
tests/unit/setup/test_initialization.py	Tests environment initialization behavior.
doc/getting_started/pyrit_conf.md	Documents loading precedence and AKV references.
.pyrit_conf_example	Updates example AKV configuration guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

tests/unit/setup/test_initialization.py:380

• This patch target is no longer called by initialize_pyrit_async, leaving the output assertion dependent on any real default environment files. Patch _resolve_environment_files instead so unrelated local files cannot add output or trigger reference resolution.

    @mock.patch("pyrit.setup.initialization._load_environment_files", return_value=True)

pyrit/setup/initialization.py:537

• Direct callers using the former list-shaped env_akv_ref reach .strip() here and get AttributeError, rather than the deliberate ValueError used by ConfigurationLoader. Validate the runtime type before calling string methods so this public API rejects legacy values consistently.

        if not env_akv_ref.strip():

pyrit/setup/initialization.py:439

• This merge is case-sensitive even on Windows. For example, ambient Path=old plus a winning PATH=new leaves both keys, so env:Path returns the ambient exact match and violates the documented merged-source precedence. Normalize keys on Windows while applying values last.

    reference_environment = {**ambient_environment, **values}

tests/unit/setup/test_initialization.py:372

• initialize_pyrit_async no longer calls _load_environment_files, so this patch is inert and the test can read real ~/.pyrit files (and even resolve their Key Vault references). Patch the resolver now used by initialization to keep the unit test isolated.

This issue also appears on line 380 of the same file.

    @mock.patch("pyrit.setup.initialization._load_environment_files", return_value=True)

doc/getting_started/pyrit_conf.md:174

• This row contradicts both the implementation and the earlier AKV precedence section: when env_files is omitted, initialization loads both .env and .env.local after the bootstrap. Remove the claim that only .env.local is loaded.

| Omitted or `null` | Load default `~/.pyrit/.env` and `~/.pyrit/.env.local`, or only `.env.local` after an AKV root |

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

.env_example:294

• This Azure OpenAI TTS base URL is missing the required /openai segment. OpenAITTSTarget._get_provider_examples() expects https://{resource}.openai.azure.com/openai/v1, so copying this example will fail endpoint validation or send requests to the wrong path.

AZURE_OPENAI_TTS_ENDPOINT2 = "<https://xxxxx.openai.azure.com/v1>"

pyrit/setup/initialization.py:208

• This still accepts malformed/nonexistent Azure vault hosts such as one-character names, leading/trailing hyphens, and consecutive hyphens. Azure vault names are 3–24 characters, start with a letter, end with an alphanumeric character, and cannot contain consecutive hyphens; without those checks, URLs that should fail preflight proceed to credential/client creation.

    valid_vault_name = 1 <= len(vault_name) <= 63 and all(
        char.isascii() and (char.isalnum() or char == "-") for char in vault_name
    )

.env_example:55

• Dotenv preserves the angle brackets inside these quoted values, so this becomes the literal endpoint <https://api.openai.com/v1>, which is not a valid URL. The same pattern now appears on every endpoint assignment in this file; remove the < and > from all actual URL values.

PLATFORM_OPENAI_CHAT_ENDPOINT="<https://api.openai.com/v1>"

[Richard Lundeen (rlundeen2)]

There's a lot to read through, so apologies if this has been addressed.

But I want to make sure we don't lose debuggability and that I can easily tell which targets we have. Right now, I use .env ~weekly to help debug myself and others.

E.g. what is the default adversarial model? What is configured? Where is the default open ai target referencing? Where is X pointing at? Etc

Right now I do a lot of that with .env. In theory we could download a .env and I could use it the same way and it could reference key vault secrets. I'm worried if all of .env is obstructed, I won't be able to see what's configured. e.g. what is the adversarial model? Or how do I configure for another target when not in the GUI?

There might be answers to this. But if they're aren't, we may want to download a .env to help even see which targets are available (and have that be able to reference keyvault secrets)

[Victor Valbuena (ValbuenaVC)]

There's a lot to read through, so apologies if this has been addressed.

But I want to make sure we don't lose debuggability and that I can easily tell which targets we have. Right now, I use .env ~weekly to help debug myself and others.

E.g. what is the default adversarial model? What is configured? Where is the default open ai target referencing? Where is X pointing at? Etc

Right now I do a lot of that with .env. In theory we could download a .env and I could use it the same way and it could reference key vault secrets. I'm worried if all of .env is obstructed, I won't be able to see what's configured. e.g. what is the adversarial model? Or how do I configure for another target when not in the GUI?

There might be answers to this. But if they're aren't, we may want to download a .env to help even see which targets are available (and have that be able to reference keyvault secrets)

Fwiw, one of the goals of this PR is to avoid having users keep an .env on disk. This feels like a possible anti-pattern when we have registries and CLI utilities to find components, but feel free to disagree.

I see what you mean though and I think we can fix this by adding a save to disk flag that saves the new .env (without actual secrets, just references to kv secrets) to disk. This seems like the best of both worlds and for debugging lets you investigate line-by-line.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

.env_example:191

• This primary TTS section still defines the old OPENAI_TTS_* names, while both TTS target registrations now read AZURE_OPENAI_TTS_* (pyrit/setup/initializers/targets.py:360-371). Filling this section therefore does not configure either registered TTS target; reconcile these names and the duplicate Azure definitions later in the file.

OPENAI_TTS_ENDPOINT1 = "<https://xxxxx.openai.azure.com/openai/v1>"
OPENAI_TTS_MODEL1 = "tts"
OPENAI_TTS_UNDERLYING_MODEL1 = "tts"

OPENAI_TTS_ENDPOINT2 = "<https://xxxxx.openai.azure.com/v1>"

pyrit/setup/initialization.py:106

• The PR description promises that bootstrap documents remain in memory and are never written to disk, but this public option intentionally persists them, including any literal credentials in the bootstrap document. Either remove the write-to-disk feature or update the stated contract and scope so this security-sensitive behavior is explicitly reviewed.

        env_akv_write_env (bool): If True, save fetched bootstrap documents with unresolved
            child references to ``~/.pyrit/.env``. Defaults to False.

.env_example:32

• The angle brackets are literal dotenv value characters, not Markdown delimiters, so copying this example produces endpoints such as <https://...> that URL clients will reject. This pattern occurs throughout the newly updated endpoint values; remove the < and > wrappers everywhere in this file.

AZURE_OPENAI_GPT4O_ENDPOINT="<https://xxxx.openai.azure.com/openai/v1>"

.env_example:140

• These aliases are interpolated before PLATFORM_OPENAI_CHAT_ENDPOINT and PLATFORM_OPENAI_CHAT_MODEL are assigned at lines 318–320. Because python-dotenv resolves in assignment order and does not revisit earlier values, both aliases become empty when this file is loaded. Move source definitions before their aliases (also for the response, realtime, image, and TTS forward references below) or move the aliases after the sources.

OPENAI_CHAT_ENDPOINT=${PLATFORM_OPENAI_CHAT_ENDPOINT}
OPENAI_CHAT_MODEL=${PLATFORM_OPENAI_CHAT_MODEL}

.env_example:172

• This primary image section still defines the old OPENAI_IMAGE_*2 names, while TargetConfig now reads AZURE_OPENAI_IMAGE_*2 (pyrit/setup/initializers/targets.py:349-352). A user following the file's instruction to fill only this section will not configure openai_image_platform; reconcile these names and the duplicate Azure definitions later in the file.

This issue also appears on line 187 of the same file.

OPENAI_IMAGE_ENDPOINT2 = "<https://xxxxx.openai.azure.com/openai/v1>"
OPENAI_IMAGE_MODEL2 = "dall-e-3"
OPENAI_IMAGE_UNDERLYING_MODEL2 = "dall-e-3"

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated no new comments.

Suppressed comments (9)

pyrit/setup/initializers/targets.py:371

• This preserves the openai_tts_platform registry name but changes all of its inputs to Azure-only variables. The former endpoint-2 example was the OpenAI platform endpoint, so users now get an Azure target under the platform registry name and there is no PLATFORM_OPENAI_TTS_* path anywhere in the repository. Keep a distinct platform variable set here, or rename this registration as a second Azure target and add the actual platform registration.

        endpoint_var="AZURE_OPENAI_TTS_ENDPOINT2",
        key_var="AZURE_OPENAI_TTS_KEY2",
        model_var="AZURE_OPENAI_TTS_MODEL2",
        underlying_model_var="AZURE_OPENAI_TTS_UNDERLYING_MODEL2",

.env_example:293

• These platform source assignments occur after OPENAI_CHAT_* and OPENAI_RESPONSES_* interpolate them at lines 128-137. Since python-dotenv resolves in assignment order, copying this file into a clean environment leaves those generic endpoint/model/key aliases empty. Move the platform source block before its aliases (the same ordering contract is documented in this PR).

PLATFORM_OPENAI_CHAT_ENDPOINT="<https://api.openai.com/v1>"
PLATFORM_OPENAI_CHAT_KEY="sk-xxxxx"
PLATFORM_OPENAI_CHAT_MODEL="gpt-4o"
PLATFORM_OPENAI_RESPONSES_ENDPOINT="<https://api.openai.com/v1>"
PLATFORM_OPENAI_RESPONSES_KEY="sk-xxxxx"

.env_example:314

• OPENAI_REALTIME_ENDPOINT and OPENAI_REALTIME_MODEL interpolate these names at lines 149-150, before these assignments are parsed. In a standalone copy of .env_example, both aliases therefore become empty. Define the platform realtime values before the generic aliases.

PLATFORM_OPENAI_REALTIME_ENDPOINT="wss://api.openai.com/v1"
PLATFORM_OPENAI_REALTIME_KEY="sk-xxxxx"
PLATFORM_OPENAI_REALTIME_MODEL="gpt-4o-realtime-preview"

.env_example:360

• The generic image aliases at lines 159-160 reference AZURE_OPENAI_IMAGE_ENDPOINT2 and AZURE_OPENAI_IMAGE_MODEL2 before this block defines them. Python-dotenv does not resolve references retroactively, so those generic values are empty when users copy this example into a clean environment. Move these primary assignments before the alias block.

AZURE_OPENAI_IMAGE_ENDPOINT1  = "<https://xxxxx.openai.azure.com/openai/v1>"
AZURE_OPENAI_IMAGE_API_KEY1 = "xxxxxx"
AZURE_OPENAI_IMAGE_MODEL1 = "deployment-name"
AZURE_OPENAI_IMAGE_UNDERLYING_MODEL1 = "dall-e-3"
AZURE_OPENAI_IMAGE_ENDPOINT2 = "<https://xxxxx.openai.azure.com/openai/v1>"

.env_example:374

• The generic TTS aliases at lines 171-172 interpolate the endpoint-2/model-2 names before this source block is reached, leaving both values empty in a clean environment. Place the primary TTS assignments before those aliases so the documented assignment-order semantics produce usable values.

AZURE_OPENAI_TTS_ENDPOINT1 = "<https://xxxxx.openai.azure.com/openai/v1>"
AZURE_OPENAI_TTS_KEY1 = "xxxxxxx"
AZURE_OPENAI_TTS_MODEL1 = "tts"
AZURE_OPENAI_TTS_UNDERLYING_MODEL1 = "tts"
AZURE_OPENAI_TTS_ENDPOINT2 = "<https://xxxxx.openai.azure.com/v1>"

pyrit/setup/akv_initialization.py:496

• write_text creates/truncates the potentially secret-bearing file under the process umask before permissions are restricted. During that window it may be readable by other users, and if chmod fails the code merely warns and leaves the exposed file in place. Create the file securely with mode 0600 before writing (and fail/clean up if permissions cannot be guaranteed).

    env_file.write_text(content, encoding="utf-8")
    try:
        env_file.chmod(0o600)
    except OSError:
        logger.warning("Could not restrict permissions on written AKV environment file: %s", env_file)

pyrit/setup/initializers/targets.py:352

• openai_image_platform now reads the Azure deployment variables, while the updated example defines the actual platform values as PLATFORM_OPENAI_IMAGE_ENDPOINT/KEY/MODEL (.env_example:300-302). Consequently, configuring those documented platform values will not register this target, and the Azure endpoint-2 configuration is mislabeled as the platform target. Wire this entry to the PLATFORM_OPENAI_IMAGE_* variables and update the corresponding initializer test.

This issue also appears on line 368 of the same file.

        endpoint_var="AZURE_OPENAI_IMAGE_ENDPOINT2",
        key_var="AZURE_OPENAI_IMAGE_API_KEY2",
        model_var="AZURE_OPENAI_IMAGE_MODEL2",
        underlying_model_var="AZURE_OPENAI_IMAGE_UNDERLYING_MODEL2",

.env_example:32

• These angle brackets are stored literally by python-dotenv, so the copied endpoint becomes <https://...> rather than a valid URL. The same Markdown-style wrapping appears in 52 URL assignments throughout this dotenv file; remove the angle brackets from all of them.

This issue also appears in the following locations of the same file:

• line 289
• line 312
• line 356
• line 370

AZURE_OPENAI_GPT4O_ENDPOINT="<https://xxxx.openai.azure.com/openai/v1>"

pyrit/setup/akv_initialization.py:422

• The PR description states that bootstrap documents remain in memory and are never written to disk, but this new public option writes them to ~/.pyrit/.env (including any literal secrets in the bootstrap). Either remove this disk-writing path to preserve the stated guarantee or update the PR's security contract and description explicitly.

    env_akv_write_env: bool = False,