Add benchmark for InferencePipeline model and workflow runs

docs/inference_helpers/cli_commands/benchmark.md

@@ -1,7 +1,7 @@
 # Benchmarking `inference`
 
 `inference benchmark` offers you an easy way to check the performance of `inference` in your setup. The command 
-is capable of benchmarking both `inference` server and `inference` Python package.
+is capable of benchmarking `inference` server, `inference` Python package, and InferencePipeline for both models and workflows on video stream processing.
 
 !!! Tip "Discovering command capabilities"
 
@@ -49,15 +49,39 @@ parameter, throughput, latency, errors and platform details) in pointed director
     ```bash
     inference server start
     ```
-Basic benchmark can be run using the following command: 
+
+### Model Benchmarking
+
+Basic model benchmark can be run using the following command: 
 
 ```bash
 inference benchmark api-speed \
   -m {your_model_id} \
   -d {pre-configured dataset name or path to directory with images} \
   -o {output_directory}  
 ```
-Command runs specified number of inferences using pointed model and saves statistics (including benchmark 
+
+### Workflow Benchmarking
+
+You can also benchmark workflows through the API:
+
+```bash
+# Remote workflow
+inference benchmark api-speed \
+  --workflow-id {workflow_id} \
+  --workspace-name {workspace_name} \
+  -d {dataset} \
+  -o {output_directory}
+
+# Local workflow specification
+inference benchmark api-speed \
+  --workflow-specification '{...}' \
+  --workflow-parameters '{"param1": "value1"}' \
+  -d {dataset} \
+  -o {output_directory}
+```
+
+Command runs specified number of inferences using pointed model or workflow and saves statistics (including benchmark 
 parameter, throughput, latency, errors and platform details) in pointed directory.
 
 This benchmark has more configuration options to support different ways HTTP API profiling. In default mode,
@@ -69,3 +93,96 @@ production environment where multiple clients are sending requests concurrently)
 option can be used (and `-c` will be ignored). Value provided in `--rps` option specifies how many requests 
 are to be spawned **each second** without waiting for previous requests to be handled. In I/O intensive benchmark 
 scenarios - we suggest running command from multiple separate processes and possibly multiple hosts.
+
+## Benchmarking InferencePipeline
+
+!!! note "InferencePipeline benchmarking"
+
+    InferencePipeline is designed for continuous video stream processing and provides optimized 
+    performance for real-time computer vision applications. It supports both model inference and 
+    workflow execution benchmarking.
+
+### Model Pipeline Benchmarking
+
+Basic model pipeline benchmark can be run using the following command:
+
+```bash
+inference benchmark pipeline-speed \
+  -m {your_model_id} \
+  -v {video_source} \
+  -o {output_directory}  
+```
+
+### Workflow Pipeline Benchmarking
+
+InferencePipeline now supports benchmarking workflows for video processing:
+
+```bash
+# Remote workflow
+inference benchmark pipeline-speed \
+  --workflow-id {workflow_id} \
+  --workspace-name {workspace_name} \
+  -v {video_source} \
+  -o {output_directory}
+
+# Local workflow specification
+inference benchmark pipeline-speed \
+  --workflow-specification '{...}' \
+  --workflow-parameters '{"param1": "value1"}' \
+  -v {video_source} \
+  -o {output_directory}
+```
+
+!!! important "Model vs Workflow Parameters"
+
+    You must specify either `--model_id` OR workflow parameters (`--workflow-id` with `--workspace-name`, 
+    or `--workflow-specification`), but not both. The benchmark will automatically use the appropriate
+    InferencePipeline initialization method.
+
+The pipeline benchmark is specifically designed to test video stream processing performance with features like:
+- Multiple concurrent pipelines processing
+- Various video sources (files, streams, cameras)
+- Configurable FPS limits and processing modes
+- Real-time performance metrics
+- Support for both models and workflows
+
+### Advanced Pipeline Benchmarking
+
+You can benchmark multiple pipelines concurrently for both models and workflows:
+
+```bash
+# Model: Run 4 concurrent pipelines
+inference benchmark pipeline-speed \
+  -m yolov8n-640 \
+  -v rtsp://camera.local/stream \
+  --pipelines 4 \
+  --duration 120 \
+  --max_fps 30
+
+# Workflow: Run 2 concurrent pipelines with parameters
+inference benchmark pipeline-speed \
+  --workflow-id object-tracking-workflow \
+  --workspace-name my-workspace \
+  --workflow-parameters '{"confidence": 0.7, "iou_threshold": 0.5}' \
+  -v video.mp4 \
+  --pipelines 2 \
+  --duration 300
+```
+
+Pipeline-specific options:
+- `--model_id/-m`: Model ID for model benchmarking (mutually exclusive with workflow options)
+- `--workflow-id/-wid`: Workflow ID for workflow benchmarking
+- `--workspace-name/-wn`: Workspace name (required with --workflow-id)
+- `--workflow-specification/-ws`: Local workflow specification JSON
+- `--workflow-parameters/-wp`: Additional workflow parameters JSON
+- `--pipelines/-p`: Number of concurrent pipelines to run (default: 1)
+- `--duration/-t`: Benchmark duration in seconds (default: 60)
+- `--max_fps/-fps`: Maximum FPS limit for each pipeline
+- `--video_reference/-v`: Video source (file path, RTSP URL, or camera index)
+
+The benchmark will report:
+- Benchmark type (Model or Workflow)
+- Per-pipeline metrics (FPS, frames processed, errors)
+- Aggregate performance across all pipelines
+- Resource utilization patterns
+- Frame processing latencies

inference_cli/benchmark.py

@@ -1,5 +1,6 @@
 import json
 from typing import Optional
+import sys
 
 import typer
 from typing_extensions import Annotated
@@ -9,6 +10,7 @@
     run_infer_api_speed_benchmark,
     run_python_package_speed_benchmark,
     run_workflow_api_speed_benchmark,
+    run_pipeline_speed_benchmark,
 )
 
 benchmark_app = typer.Typer(help="Commands for running inference benchmarks.")
@@ -269,5 +271,133 @@ def python_package_speed(
         raise typer.Exit(code=1)
 
 
+@benchmark_app.command()
+def pipeline_speed(
+    model_id: Annotated[
+        Optional[str],
+        typer.Option(
+            "--model_id",
+            "-m",
+            help="Model ID in format project/version.",
+        ),
+    ] = None,
+    workflow_id: Annotated[
+        Optional[str],
+        typer.Option(
+            "--workflow-id",
+            "-wid",
+            help="Workflow ID.",
+        ),
+    ] = None,
+    workspace_name: Annotated[
+        Optional[str],
+        typer.Option(
+            "--workspace-name",
+            "-wn",
+            help="Workspace Name.",
+        ),
+    ] = None,
+    workflow_specification: Annotated[
+        Optional[str],
+        typer.Option(
+            "--workflow-specification",
+            "-ws",
+            help="Workflow specification JSON.",
+        ),
+    ] = None,
+    workflow_parameters: Annotated[
+        Optional[str],
+        typer.Option(
+            "--workflow-parameters",
+            "-wp",
+            help="Workflow parameters JSON.",
+        ),
+    ] = None,
+    video_reference: Annotated[
+        str,
+        typer.Option(
+            "--video_reference",
+            "-v",
+            help="Video source - can be video file path, stream URL, or camera ID",
+        ),
+    ] = "0",
+    duration_seconds: Annotated[
+        int,
+        typer.Option("--duration", "-t", help="Benchmark duration in seconds"),
+    ] = 60,
+    max_fps: Annotated[
+        Optional[float],
+        typer.Option("--max_fps", "-fps", help="Maximum FPS for pipeline"),
+    ] = None,
+    api_key: Annotated[
+        Optional[str],
+        typer.Option(
+            "--api-key",
+            "-a",
+            help="Roboflow API key for your workspace. If not given - env variable `ROBOFLOW_API_KEY` will be used",
+        ),
+    ] = None,
+    model_configuration: Annotated[
+        Optional[str],
+        typer.Option(
+            "--model_config", "-mc", help="Location of yaml file with model config"
+        ),
+    ] = None,
+    output_location: Annotated[
+        Optional[str],
+        typer.Option(
+            "--output_location",
+            "-o",
+            help="Location where to save the result (path to file or directory)",
+        ),
+    ] = None,
+    num_pipelines: Annotated[
+        int,
+        typer.Option(
+            "--pipelines",
+            "-p",
+            help="Number of concurrent pipelines to benchmark",
+        ),
+    ] = 1,
+):
+    if model_id and (workflow_id or workflow_specification):
+        typer.echo(
+            "Error: Cannot specify both --model_id and workflow parameters (--workflow-id or --workflow-specification)"
+        )
+        raise typer.Exit(code=1)
+    if not model_id and not workflow_id and not workflow_specification:
+        typer.echo(
+            "Error: Must specify either --model_id or workflow parameters (--workflow-id with --workspace-name, or --workflow-specification)"
+        )
+        raise typer.Exit(code=1)
+    if workflow_id and not workspace_name:
+        typer.echo("Error: --workspace-name is required when using --workflow-id")
+        raise typer.Exit(code=1)
+
+    try:
+        run_pipeline_speed_benchmark(
+            video_reference=video_reference,
+            model_id=model_id,
+            workflow_id=workflow_id,
+            workspace_name=workspace_name,
+            workflow_specification=workflow_specification,
+            workflow_parameters=workflow_parameters,
+            duration_seconds=duration_seconds,
+            max_fps=max_fps,
+            api_key=api_key,
+            model_configuration=model_configuration,
+            output_location=output_location,
+            num_pipelines=num_pipelines,
+        )
+
+    except KeyboardInterrupt:
+        print("\nBenchmark interrupted.")
+        return
+
+    except Exception as error:
+        typer.echo(f"Command failed. Cause: {error}")
+        raise typer.Exit(code=1)
+
+
 if __name__ == "__main__":
     benchmark_app()