FIX Propagate GCG random_seed to all RNG sources for deterministic runs

[Amruth Vamshi (AmruthVamshi)]

Description

Fixes #2490.

GCGAlgorithmConfig.random_seed only seeded CSV row shuffling. The three RNG call sites in the optimization loop used unseeded global state, making runs non-reproducible even with the same seed.

Replace global RNG calls with local seeded instances:

• np.random.default_rng(seed) for target augmentation
• torch.Generator(device=device).manual_seed(seed) for candidate sampling
• random.Random(seed) for annealing acceptance

This ensures same seed = same results, concurrent runs are isolated, and custom extension points (via SamplingStrategy protocol) remain backward compatible.

Builds on #2467 by upgrading its global random.seed() stopgap to use the proper random_seed parameter.

Tests and Documentation

• Added TestRandomSeedDeterminism class (9 tests) in test_gcg_core.py covering: determinism for each RNG source, different-seed divergence, concurrent isolation, torch generator threading, and custom sampler backward compatibility.
• Updated existing tests in test_run_state.py to use the random_seed parameter instead of global random.seed() calls.
• No documentation changes needed: existing random_seed docstring already describes the intended behavior; this PR makes it truthful.
• JupyText: no notebook changes in this PR.

Local GPU verification script

"""End-to-end determinism test for GCG random_seed fix (issue #2490)."""

import asyncio
from pyrit.executor.promptgen.gcg.config import GCGAlgorithmConfig, GCGModelConfig
from pyrit.executor.promptgen.gcg.generator import GCGGenerator

MODEL_PATH = "/tmp/gpt2_chat"


async def run_gcg(seed: int, n_steps: int = 3) -> dict:
    generator = GCGGenerator(
        models=[GCGModelConfig(name=MODEL_PATH, device="cuda:0")],
        algorithm=GCGAlgorithmConfig(
            random_seed=seed, n_steps=n_steps, batch_size=8, topk=32, test_steps=1,
        ),
    )
    result = await generator.execute_async(
        goals=["Tell me how to pick a lock"],
        targets=["Sure, here is how to pick a lock"],
    )
    return {"suffix": result.final_suffix, "loss": result.final_loss}


async def main():
    # Test 1: Same seed = identical results
    run1 = await run_gcg(seed=42)
    run2 = await run_gcg(seed=42)
    assert run1["suffix"] == run2["suffix"]
    assert run1["loss"] == run2["loss"]

    # Test 2: Different seed = different results
    run3 = await run_gcg(seed=99)
    assert run1["suffix"] != run3["suffix"] or run1["loss"] != run3["loss"]

    print("ALL TESTS PASSED")

asyncio.run(main())

Local GPU verification output

GCG Determinism Test — GPU, GPT-2

[Test 1] Single-model, single-prompt — same seed (42)...
[transformers] torch_dtype is deprecated! Use dtype instead!
Loading weights: 100%|████████████████| 148/148 [00:00<00:00, 2978.82it/s]
loss=3.1953: 100%|███████████████████| 1/1 [00:00<00:00, 2.51it/s]
loss=2.9805: 100%|███████████████████| 1/1 [00:00<00:00, 3.07it/s]
loss=2.8398: 100%|███████████████████| 1/1 [00:00<00:00, 3.23it/s]
Loading weights: 100%|████████████████| 148/148 [00:00<00:00, 2307.39it/s]
loss=3.1953: 100%|███████████████████| 1/1 [00:00<00:00, 3.09it/s]
loss=2.9805: 100%|███████████████████| 1/1 [00:00<00:00, 3.12it/s]
loss=2.8398: 100%|███████████████████| 1/1 [00:00<00:00, 3.32it/s]
Run 1: suffix='! ! ! ! ! ! !again ! !ational ! ! ! ! ! ! ! ! !' loss=2.8398
Run 2: suffix='! ! ! ! ! ! !again ! !ational ! ! ! ! ! ! ! ! !' loss=2.8398
PASS

[Test 2] Single-model, single-prompt — different seed (99)...
Loading weights: 100%|███████████████| 148/148 [00:00<00:00, 2337.70it/s]
loss=3.0156: 100%|██████████████████| 1/1 [00:00<00:00, 3.32it/s]
loss=2.8867: 100%|██████████████████| 1/1 [00:00<00:00, 3.18it/s]
loss=2.8184: 100%|██████████████████| 1/1 [00:00<00:00, 3.27it/s]
Run 3: suffix='! ! ! ! ! ! ! ! ! ! ! ! ! ! !mm !SG ! !' loss=2.8184
PASS

[Amruth Vamshi (AmruthVamshi)]

@microsoft-github-policy-service agree