Skip to content

Commit 4e9a590

Browse files
committed
Merge branch 'develop' of github.com:NVIDIA/AIQToolkit into feature/workflow-tool-authentication
Signed-off-by: Eric Evans <[email protected]>
2 parents 1e08f93 + 10d007e commit 4e9a590

File tree

3 files changed

+25
-1
lines changed

3 files changed

+25
-1
lines changed

docs/source/workflows/evaluate.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,9 +142,18 @@ eval:
142142

143143
A judge LLM is used to evaluate the trajectory produced by the workflow, taking into account the tools available during execution. It returns a floating-point score between 0 and 1, where 1.0 indicates a perfect trajectory.
144144

145+
To configure the judge LLM, define it in the `llms` section of the configuration file, and reference it in the evaluator configuration using the `llm_name` key.
146+
145147
It is recommended to set `max_tokens` to 1024 for the judge LLM to ensure sufficient context for evaluation.
146148

147-
To configure the judge LLM, define it in the `llms` section of the configuration file, and reference it in the evaluator configuration using the `llm_name` key.
149+
Note: Trajectory evaluation may result in frequent LLM API calls. If you encounter rate-limiting errors (such as `[429] Too Many Requests` error), you can reduce the number of concurrent requests by adjusting the `max_concurrency` parameter in your config. For example:
150+
151+
```yaml
152+
eval:
153+
general:
154+
max_concurrency: 2
155+
```
156+
This setting reduces the number of concurrent requests to avoid overwhelming the LLM endpoint.
148157

149158
## Workflow Output
150159
The `aiq eval` command runs the workflow on all the entries in the `dataset`. The output of these runs is stored in a file named `workflow_output.json` under the `output_dir` specified in the configuration file.

src/aiq/authentication/__init__.py

Whitespace-only changes.

src/aiq/authentication/utils.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,18 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
116
import logging
217

318
import httpx

0 commit comments

Comments
 (0)