Add support for "benchmarking scenarios"

[sjmonson]

This PR adds support for "scenarios" that allow specifying benchmark argument in a file / as a single Pydantic object. CLI argument defaults are loaded from the scenario object defaults to give benchmark-as-code users the same defaults as CLI. Argument values in the CLI follow the following precedence: Scenario (class defaults) < Scenario (CLI provided Scenario) < CLI Arguments.

Closes:

• Base enablement to enable a dataset configuration file to be passed in rather than standard args, allow alias's for the folder structure #81

[markurtz]

@sjmonson I like the direction of this. A few quick thoughts from my side:

• I think using Pydantic's built-in support for dotenv files along the lines of what we're doing for the general settings might work well here, and we can store the different scenarios then as just .env files and also enable potentially a default .env file: https://docs.pydantic.dev/latest/concepts/pydantic_settings/#dotenv-env-support
• I'd like to see if we could get a unified setup for validation of values, types, and auto conversion that click is handling currently standardized through this, so all entry point pathways go through the same validations and fail early
• It also would be great if we could get the Python API entrypoint and the CLI then pulling from the same defaults and able to support the same pathways

Not sure if there's something out there already to automatically handle the last two points, but if so, that would be a great inclusion

[sjmonson]

• I think using Pydantic's built-in support for dotenv files along the lines of what we're doing for the general settings might work well here, and we can store the different scenarios then as just .env files and also enable potentially a default .env file: https://docs.pydantic.dev/latest/concepts/pydantic_settings/#dotenv-env-support

My plan was to rely on the base class json and yaml loaders since they are cleaner for nested structures, such as synthetic dataset args. I can definitely try pydantic-settings since that does have the advantage of allowing us to unify all GuideLLM options under one file.

• I'd like to see if we could get a unified setup for validation of values, types, and auto conversion that click is handling currently standardized through this, so all entry point pathways go through the same validations and fail early

By default pydantic will attempt type coercion during the validation so its probably as simple as disabling the type handling in click and raising a click.BadParameter on model validation errors.

• It also would be great if we could get the Python API entrypoint and the CLI then pulling from the same defaults and able to support the same pathways

They already do in this patch as I have set every click default to pull from the scenario model, unless you mean something else?

[markurtz]

@sjmonson, for the last one, yes, something else. Specifically the entrypoint for benchmarking Python API here: https://github.com/neuralmagic/guidellm/blob/main/src/guidellm/benchmark/entrypoints.py#L22. Towards @anmarques's previous issues with needing to set all argument values when not using the CLI. Also would help towards suporting both CLI and Python API for the scenarios

[sjmonson]

Ah I see. The workflow I imagined for @anmarques's use-case is to call the higher-level entrypoint benchmark_with_scenario since it handles the defaults on scenario instantiation so a call like the following would be the same as calling the CLI with only the required args:

result = await benchmark_with_scenario(
    GenerativeTextScenario(
        target="http://localhost:8000",
        data={
            "prompt_tokens": 128,
            "output_tokens": 128,
        },
        rate_type="sweep",
    ),
    output_path="output.json",
)

[Copilot]

Pull Request Overview

This pull request adds support for benchmarking scenarios by introducing a unified way to specify benchmark arguments via Pydantic objects and JSON/YAML scenario files, ensuring that CLI and code-based defaults remain consistent.

• Introduces utility functions for JSON parsing and default option handling in CLI
• Adds Pydantic helper methods to load configuration from files
• Implements scenario-based configuration with new benchmark scenario files and updated entrypoints

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/guidellm/utils/cli.py	Added JSON parsing helper, default-setting function, and custom union parameter type
src/guidellm/objects/pydantic.py	Added get_default and from_file methods to streamline model instantiation
src/guidellm/benchmark/scenarios/rag.json	New benchmark scenario configuration in JSON
src/guidellm/benchmark/scenarios/chat.json	New benchmark scenario configuration in JSON
src/guidellm/benchmark/scenario.py	Introduced Scenario and GenerativeTextScenario classes along with a float-list parser
src/guidellm/benchmark/entrypoints.py	Added benchmark_with_scenario to enable scenario-driven execution
src/guidellm/main.py	Updated CLI to support scenario files and default values from scenario models

Comments suppressed due to low confidence (2)

src/guidellm/utils/cli.py:29

• [nitpick] The class name 'Union' may be confused with the built-in union type operator used in type hints. Consider renaming it to a more explicit name (e.g. 'UnionParamType') for clarity.

class Union(click.ParamType):

src/guidellm/benchmark/scenario.py:28

• [nitpick] The docstring for parse_float_list could be improved by mentioning that any whitespace around comma-separated numbers will not be trimmed automatically. Consider adding a note or trimming whitespace before conversion.

def parse_float_list(value: Union[str, float, list[float]]) -> list[float]: