diff --git a/notebooks/LLM_TCO_Calculator.xlsx b/notebooks/LLM_TCO_Calculator.xlsx new file mode 100644 index 000000000..1c8eb31fa Binary files /dev/null and b/notebooks/LLM_TCO_Calculator.xlsx differ diff --git a/notebooks/README.md b/notebooks/README.md new file mode 100644 index 000000000..61f8a3863 --- /dev/null +++ b/notebooks/README.md @@ -0,0 +1,9 @@ + +# AIPerf Utility Notebooks + +This folder contains the various utility notebooks for AIPerf. + +1. [TCO_calculator.ipynb](TCO_calculator.ipynb): This notebook allows user to benchmark a NIM LLM deployment, then export the data to the NIM total cost of ownership (TCO) calculator. diff --git a/notebooks/TCO_calculator.ipynb b/notebooks/TCO_calculator.ipynb new file mode 100644 index 000000000..8d548d7ce --- /dev/null +++ b/notebooks/TCO_calculator.ipynb @@ -0,0 +1,579 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "id": "f3e4c571-624d-4db6-b4d9-ae912879967b", + "metadata": {}, + "source": [ + "# AIPerf -> LLM TCO Calculator Data Connector\n", + "\n", + "This notebook shows you how to do LLM performance benchmarking with the NVIDIA AIPerf tool and then export the data to a TCO (Total Cost of Ownership) calculator in [Excel spreadsheet format](./LLM_TCO_Calculator.xlsx).\n", + "\n", + "\n", + "To execute this notebook, you can use the NVIDIA Triton server container:\n", + "```\n", + "docker run --gpus=all --ipc=host --net=host --rm -it -v $PWD:/myworkspace nvcr.io/nvidia/tritonserver:25.09-py3-sdk bash \n", + "```\n", + "\n", + "Then from within the docker interactive session:\n", + "```\n", + "pip install jupyterlab\n", + "jupyter lab --ip 0.0.0.0 --port=8888 --allow-root --notebook-dir=/myworkspace\n", + "```\n", + "\n", + "First, we define some metadata fields describing the deployment environment.\n", + "\n", + "**Notes:**\n", + "- NIM engine ID provides both the backend type (e.g. TensorRT-LLM, vLLM or SGlang) and precision. You can find this information when the NIM container starts.\n", + "\n", + "- This notebook collects data corresponding to a single deployment environment described by the metadata field. In this tutorial, we will make use of the `Meta-Llama-3-8B-Instruct` model. Note that NVIDIA NGC and HuggingFace model hub use slightly different identifier for this model." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "93c18473-09ea-4a6f-87fa-d67fa3f7daa5", + "metadata": {}, + "outputs": [], + "source": [ + "meta_field = {\n", + " 'Model': \"meta-llama/Meta-Llama-3-8B-Instruct\",\n", + " 'GPU Type': \"H100_80GB\",\n", + " 'number_of_gpus': 1,\n", + " 'Precision': \"BF16\",\n", + " 'Execution Mode': \"NIM-TRTLLM\",\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "70b3df53-c103-4de2-81f5-419aa4d65f83", + "metadata": {}, + "source": [ + "## Pre-requisite\n", + "\n", + "First, we install the AIPerf tool in the Pytorch container. \n", + "As a client-side LLM-focused benchmarking tool, NVIDIA AIPerf provides key metrics such as time to first token (TTFT), inter-token latency (ITL), tokens per second (TPS), requests per second (RPS) and more. AIPerf also supports any LLM inference service conforming to the OpenAI API specification, a widely accepted de facto standard in the industry. For this benchmarking guide, we’ll use NVIDIA NIM, a collection of inference microservices that offer high-throughput and low-latency inference for both base and fine-tuned LLMs. NIM features ease-of-use and enterprise-grade security and manageability. \n", + "\n", + "### Install AIPerf tool" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ad5de6fe-8547-4259-956a-980aa8b71dce", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "pip install aiperf" + ] + }, + { + "cell_type": "markdown", + "id": "9e6351a6-a5a3-4067-831e-abe26ae53969", + "metadata": {}, + "source": [ + "### Setting up a NIM LLM server (optional)\n", + "\n", + "If you don't already have a target for benchmarking, like an OpenAI compatible LLM service, let's setup one. \n", + "\n", + "NVIDIA NIM provides the easiest and quickest way to put LLMs and other AI foundation models into production. Read [A Simple Guide to Deploying Generative AI with NVIDIA NIM](https://developer.nvidia.com/blog/a-simple-guide-to-deploying-generative-ai-with-nvidia-nim/) or consult the latest [NIM LLM documentation](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html) to get started, which will walk you through hardware requirements and prerequisites, including NVIDIA NGC API keys.\n", + "\n", + "For convenience, the following commands have been provided for deploying NIM and executing inference from the [Getting Started Guide](https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html): \n", + "\n", + " \n", + "```\n", + "export NGC_API_KEY= \n", + "export LOCAL_NIM_CACHE=~/.cache/nim\n", + "\n", + "mkdir -p \"$LOCAL_NIM_CACHE\"\n", + "\n", + "docker run -it --rm \\\n", + " --gpus all \\\n", + " --shm-size=16GB \\\n", + " -e NGC_API_KEY \\\n", + " -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n", + " -u $(id -u) \\\n", + " -p 8000:8000 \\\n", + " nvcr.io/nim/meta/llama3-8b-instruct:latest\n", + "```\n", + "\n", + "\n", + "## Performance benchmarking script\n", + "\n", + "The next step is to define the use cases (i.e. input/output sequence length scenarios) and carry out the benchmarking." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "e8395733-ce18-4447-845c-b3579acc2067", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Overwriting benchmark.sh\n" + ] + } + ], + "source": [ + "%%writefile benchmark.sh\n", + "#!/usr/bin/env bash\n", + "\n", + "declare -A useCases\n", + "\n", + "# Populate the array with use case descriptions and their specified input/output lengths\n", + "useCases[\"Translation\"]=\"200/200\"\n", + "useCases[\"Text classification\"]=\"200/5\"\n", + "useCases[\"Text summary\"]=\"1000/200\"\n", + "\n", + "# Function to execute AIPerf with the input/output lengths as arguments\n", + "runBenchmark() {\n", + " local description=\"$1\"\n", + " local lengths=\"${useCases[$description]}\"\n", + " IFS='/' read -r inputLength outputLength <<< \"$lengths\"\n", + "\n", + " echo \"Running AIPerf for $description with input length $inputLength and output length $outputLength\"\n", + " #Runs\n", + " for concurrency in 1 2 5 10 50 100 250; do\n", + "\n", + " local INPUT_SEQUENCE_LENGTH=$inputLength\n", + " local INPUT_SEQUENCE_STD=0\n", + " local OUTPUT_SEQUENCE_LENGTH=$outputLength\n", + " local CONCURRENCY=$concurrency\n", + " local REQUEST_COUNT=$(($CONCURRENCY * 3))\n", + " local MODEL=meta/llama3-8b-instruct\n", + "\n", + " aiperf profile \\\n", + " -m $MODEL \\\n", + " --endpoint-type chat \\\n", + " --streaming \\\n", + " -u localhost:8000 \\\n", + " --synthetic-input-tokens-mean $INPUT_SEQUENCE_LENGTH \\\n", + " --synthetic-input-tokens-stddev $INPUT_SEQUENCE_STD \\\n", + " --concurrency $CONCURRENCY \\\n", + " --request-count $REQUEST_COUNT \\\n", + " --output-tokens-mean $OUTPUT_SEQUENCE_LENGTH \\\n", + " --extra-inputs min_tokens:$OUTPUT_SEQUENCE_LENGTH \\\n", + " --extra-inputs ignore_eos:true \\\n", + " --tokenizer meta-llama/Meta-Llama-3-8B-Instruct \\\n", + " --artifact-dir artifact/ISL${INPUT_SEQUENCE_LENGTH}_OSL${OUTPUT_SEQUENCE_LENGTH}/CON${CONCURRENCY}\n", + "\n", + " done\n", + "}\n", + "\n", + "# Iterate over all defined use cases and run the benchmark script for each\n", + "for description in \"${!useCases[@]}\"; do\n", + " runBenchmark \"$description\"\n", + "done\n" + ] + }, + { + "cell_type": "markdown", + "id": "603f1941-5206-4bca-a547-028e0ea50f21", + "metadata": {}, + "source": [ + "This test will use the llama-3 tokenizer from HuggingFace, which is a guarded repository [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct). You will need to apply for access, then login with your HF credential.\n", + "\n", + "Open a terminal in you Jupyter lab interface, then login to HF:\n", + "```\n", + " pip install huggingface_hub\n", + " huggingface-cli login\n", + "```\n", + "\n", + "Next, we execute the bash script, which will carry out the defined benchmarking scenarios and gather the data in a default directory named `artifacts` under the current working directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6cbfacd3-5755-4c0b-ae23-3abffceebbdb", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "%%bash\n", + "bash benchmark.sh" + ] + }, + { + "cell_type": "markdown", + "id": "c480b28d-6816-4c84-9124-bdc56fc81f41", + "metadata": {}, + "source": [ + "## Reading AIPerf data\n", + "\n", + "Once performance benchmarking is done, we read and collect the results in a single data frame." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "ff69c986-0c9b-46a9-8f28-800cd61ab24d", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import json\n", + "import pandas as pd\n", + "\n", + "ISL_OSL_LIST = [\"200_5\", \"200_200\", \"1000_200\"]\n", + "CONCURRENCIES = [1, 2, 5, 10, 50, 100, 250]\n", + "df = pd.DataFrame()\n", + "\n", + "for concurrency in CONCURRENCIES :\n", + " for isl_osl in ISL_OSL_LIST:\n", + " ISL=isl_osl.split(\"_\")[0]\n", + " OSL=isl_osl.split(\"_\")[1]\n", + " \n", + " with open(f'./artifact/ISL{ISL}_OSL{OSL}/CON{concurrency}/profile_export_aiperf.json', 'r') as f:\n", + " data = json.load(f)\n", + " \n", + " row = {\n", + " 'Inter Token 90th Percentile Latency (ms)': data[\"records\"][\"inter_token_latency\"][\"p90\"],\n", + " 'Inter Token 99th Percentile Latency (ms)': data[\"records\"][\"inter_token_latency\"][\"p99\"],\n", + " 'Inter Token Average Latency (ms)': data[\"records\"][\"inter_token_latency\"][\"avg\"],\n", + " 'Time to First Token 90th Percentile Latency (ms)': data[\"records\"][\"ttft\"][\"p90\"],\n", + " 'Time to First Token 99th Percentile Latency (ms)': data[\"records\"][\"ttft\"][\"p99\"],\n", + " 'Time to First Token Average Latency (ms)': data[\"records\"][\"ttft\"][\"avg\"],\n", + " 'Request 90th Percentile Latency (ms)': data[\"records\"][\"request_latency\"][\"p90\"],\n", + " 'Request 99th Percentile Latency (ms)': data[\"records\"][\"request_latency\"][\"p99\"],\n", + " 'Request Latency (ms)': data[\"records\"][\"request_latency\"][\"avg\"],\n", + " 'Requests per Second': data[\"records\"][\"request_throughput\"][\"avg\"],\n", + " 'Tokens per Second': data[\"records\"][\"output_token_throughput\"][\"avg\"],\n", + " 'Seq Length (ISL/OSL)': isl_osl,\n", + " 'Concurrency': concurrency\n", + " } \n", + " \n", + " row = meta_field | row\n", + " \n", + " df = pd.concat([df, pd.DataFrame([row])], ignore_index=True)" + ] + }, + { + "cell_type": "markdown", + "id": "3a997b59-3d4e-4877-953c-088563aa8998", + "metadata": {}, + "source": [ + "## Exporting data to excel format\n", + "\n", + "We next export the benchmarking data to a NIM TCO Calculator compatible format, which comprises both metadata fields as well as performance metric fields." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "f39710a9-882c-44aa-b428-d7ed2976eb23", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
ModelGPU Typenumber_of_gpusPrecisionExecution ModeInter Token 90th Percentile Latency (ms)Inter Token 99th Percentile Latency (ms)Inter Token Average Latency (ms)Time to First Token 90th Percentile Latency (ms)Time to First Token 99th Percentile Latency (ms)Time to First Token Average Latency (ms)Request 90th Percentile Latency (ms)Request 99th Percentile Latency (ms)Request Latency (ms)Requests per SecondTokens per SecondSeq Length (ISL/OSL)Concurrency
0meta-llama/Meta-Llama-3-8B-InstructH100_80GB1BF16NIM-TRTLLM4.9493454.9670924.86787120.53295620.98871818.88936439.75327340.06144838.36085025.419948127.099742200_51
1meta-llama/Meta-Llama-3-8B-InstructH100_80GB1BF16NIM-TRTLLM4.8732484.8743384.86967920.86083821.59170518.887994990.637183991.584959987.9540301.010877202.175430200_2001
2meta-llama/Meta-Llama-3-8B-InstructH100_80GB1BF16NIM-TRTLLM4.7084174.7115564.70026346.37349846.79023245.210619981.959344982.167204980.5629751.018414203.6828551000_2001
3meta-llama/Meta-Llama-3-8B-InstructH100_80GB1BF16NIM-TRTLLM6.2573296.3406665.83873329.91147833.78747822.65207853.72580957.13734546.00701141.722412208.612062200_52
4meta-llama/Meta-Llama-3-8B-InstructH100_80GB1BF16NIM-TRTLLM4.9204284.9216034.91261429.58008133.86831422.3793201007.4998421011.103446999.9895051.995674399.134707200_2002
\n", + "
" + ], + "text/plain": [ + " Model GPU Type number_of_gpus Precision \\\n", + "0 meta-llama/Meta-Llama-3-8B-Instruct H100_80GB 1 BF16 \n", + "1 meta-llama/Meta-Llama-3-8B-Instruct H100_80GB 1 BF16 \n", + "2 meta-llama/Meta-Llama-3-8B-Instruct H100_80GB 1 BF16 \n", + "3 meta-llama/Meta-Llama-3-8B-Instruct H100_80GB 1 BF16 \n", + "4 meta-llama/Meta-Llama-3-8B-Instruct H100_80GB 1 BF16 \n", + "\n", + " Execution Mode Inter Token 90th Percentile Latency (ms) \\\n", + "0 NIM-TRTLLM 4.949345 \n", + "1 NIM-TRTLLM 4.873248 \n", + "2 NIM-TRTLLM 4.708417 \n", + "3 NIM-TRTLLM 6.257329 \n", + "4 NIM-TRTLLM 4.920428 \n", + "\n", + " Inter Token 99th Percentile Latency (ms) Inter Token Average Latency (ms) \\\n", + "0 4.967092 4.867871 \n", + "1 4.874338 4.869679 \n", + "2 4.711556 4.700263 \n", + "3 6.340666 5.838733 \n", + "4 4.921603 4.912614 \n", + "\n", + " Time to First Token 90th Percentile Latency (ms) \\\n", + "0 20.532956 \n", + "1 20.860838 \n", + "2 46.373498 \n", + "3 29.911478 \n", + "4 29.580081 \n", + "\n", + " Time to First Token 99th Percentile Latency (ms) \\\n", + "0 20.988718 \n", + "1 21.591705 \n", + "2 46.790232 \n", + "3 33.787478 \n", + "4 33.868314 \n", + "\n", + " Time to First Token Average Latency (ms) \\\n", + "0 18.889364 \n", + "1 18.887994 \n", + "2 45.210619 \n", + "3 22.652078 \n", + "4 22.379320 \n", + "\n", + " Request 90th Percentile Latency (ms) Request 99th Percentile Latency (ms) \\\n", + "0 39.753273 40.061448 \n", + "1 990.637183 991.584959 \n", + "2 981.959344 982.167204 \n", + "3 53.725809 57.137345 \n", + "4 1007.499842 1011.103446 \n", + "\n", + " Request Latency (ms) Requests per Second Tokens per Second \\\n", + "0 38.360850 25.419948 127.099742 \n", + "1 987.954030 1.010877 202.175430 \n", + "2 980.562975 1.018414 203.682855 \n", + "3 46.007011 41.722412 208.612062 \n", + "4 999.989505 1.995674 399.134707 \n", + "\n", + " Seq Length (ISL/OSL) Concurrency \n", + "0 200_5 1 \n", + "1 200_200 1 \n", + "2 1000_200 1 \n", + "3 200_5 2 \n", + "4 200_200 2 " + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5baf8e86-c8d1-42fc-94d3-15b592a5adc9", + "metadata": {}, + "outputs": [], + "source": [ + "!pip install openpyxl" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "125f78e6-cc51-4091-bb16-9a1d8403d6cf", + "metadata": {}, + "outputs": [], + "source": [ + "columns = [\n", + " 'Model',\n", + " 'GPU Type',\n", + " 'Seq Length (ISL/OSL)',\n", + " 'number_of_gpus',\n", + " 'Concurrency',\n", + " 'Precision',\n", + " 'Execution Mode',\n", + " 'Inter Token 90th Percentile Latency (ms)',\n", + " 'Inter Token 99th Percentile Latency (ms)',\n", + " 'Inter Token Average Latency (ms)',\n", + " 'Time to First Token 90th Percentile Latency (ms)',\n", + " 'Time to First Token 99th Percentile Latency (ms)',\n", + " 'Time to First Token Average Latency (ms)',\n", + " 'Request 90th Percentile Latency (ms)',\n", + " 'Request 99th Percentile Latency (ms)',\n", + " 'Request Latency (ms)',\n", + " 'Requests per Second',\n", + " 'Tokens per Second'\n", + " ]\n", + "df[columns].to_excel('data.xlsx', index=False)\n" + ] + }, + { + "cell_type": "markdown", + "id": "becc138b-6d92-49aa-a9a6-3ad31ad75c87", + "metadata": {}, + "source": [ + "## Importing the data to the TCO calculator\n", + "\n", + "The [NIM TCO calculator tool](LLM_TCO_Calculator.xlsx) is implemented as an Excel spreadsheet. You can use MS Excel spreadsheet to open the excel file above, then simply copy the data rows into the \"data\" subsheet of the TCO calculator. That will complete the import phase and make the new data available in the TCO calculator." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b1645aa-28e9-45d1-9db9-b86af239e627", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}