Skip to content

[Multi Async Eval Configs] - Docs update #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SamstyleGhost wants to merge 1 commit into
base: 10-28-_live_trends_docs_creation
Choose a base branch
from
Open

[Multi Async Eval Configs] - Docs update #160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified images/docs/configure_eval.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 18 additions & 1 deletion online-evals/via-ui/set-up-auto-evaluation-on-logs.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ Navigate to the repository where you want to evaluate your logs.
Click on `Configure evaluation` in the top right corner of the page. This will open up the evaluation configuration sheet.

![Screenshot of the repository page with the configure evaluation button highlighted](/images/docs/configure_eval.png)
<Note>
You can create multiple configurations for auto evaluation on logs. Each configuration can be used to evaluate logs based on different criteria.
Explained in the [Multiple configurations for auto evaluations](#multiple-configurations-for-auto-evaluations) section.
</Note>
</Step>
<Step title="Understand evaluation levels">
The `Auto Evaluation` section allows you to configure evaluators at different levels of granularity:
Expand Down Expand Up @@ -199,6 +203,19 @@ When you click on any node in the tree view, the right panel displays the evalua

Learn more about evaluating individual components in [Node Level Evaluation](/online-evals/via-sdk/node-level-evaluation).

## Multiple configurations for auto evaluations

You can create multiple configurations on a single log repository for auto evaluations. Each configuration can be used to evaluate logs based on different criteria.

<Steps>
<Step title="Add configuration">
Click on the `Add configuration` button in the top right corner of the sheet. This will add a new configuration to the list.
</Step>
<Step title="Modify configuration">
You can add or remove evaluators, filters, and sampling rate for each configuration. Check the [Setting Up Auto Evaluation on Logs](#setting-up-auto-evaluation) section for more details on setting up a single configuration.
</Step>
</Steps>

## Dataset Curation

Once you have logs and evaluations in Maxim, you can easily curate datasets by filtering and selecting logs based on different criteria.
Expand All @@ -216,4 +233,4 @@ Once you have logs and evaluations in Maxim, you can easily curate datasets by f
<Step title="Choose to add logs to an existing dataset or create a new dataset. Map the columns and click `Add entries`">
![Add to dataset dialog](/images/docs/online-evals/dataset-curation-add-to-dataset-dialog.png)
</Step>
</Steps>
</Steps>