Skip to content

Commit

Permalink
Fix issue with fallbacks and the initial client (#1089)
Browse files Browse the repository at this point in the history
<!-- ELLIPSIS_HIDDEN -->



> [!IMPORTANT]
> Added retry policies to `CustomGPT4oMini` and `CustomHaiku`, corrected
strategies for round-robin and fallback clients in `clients.baml`.
> 
>   - **Retry Policies**:
>     - Added `retry_policy Exponential` to `CustomGPT4oMini`.
>     - Added `retry_policy Constant` to `CustomHaiku`.
>   - **Strategies**:
> - Corrected round-robin strategy in `CustomFast` to use
`CustomGPT4oMini` and `CustomHaiku`.
> - Corrected fallback strategy in `OpenaiFallback` to use
`CustomGPT4oMini` twice.
>   - **Retry Policy Definitions**:
> - Defined `retry_policy Constant` with `max_retries 3`, `delay_ms
200`.
> - Defined `retry_policy Exponential` with `max_retries 2`, `delay_ms
300`, `multiplier 1.5`, `max_delay_ms 10000`.
> 
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup>
for 8b092f3. It will automatically
update as commits are pushed.</sup>

<!-- ELLIPSIS_HIDDEN -->
  • Loading branch information
hellovai authored Oct 23, 2024
1 parent fa200b2 commit 682dd66
Showing 1 changed file with 27 additions and 2 deletions.
29 changes: 27 additions & 2 deletions engine/baml-runtime/src/cli/initial_project/baml_src/clients.baml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ client<llm> CustomGPT4o {

client<llm> CustomGPT4oMini {
provider openai
retry_policy Exponential
options {
model "gpt-4o-mini"
api_key env.OPENAI_API_KEY
Expand All @@ -27,24 +28,48 @@ client<llm> CustomSonnet {

client<llm> CustomHaiku {
provider anthropic
retry_policy Constant
options {
model "claude-3-haiku-20240307"
api_key env.ANTHROPIC_API_KEY
}
}

// https://docs.boundaryml.com/docs/snippets/clients/round-robin
client<llm> CustomFast {
provider round-robin
options {
// This will alternate between the two clients
strategy [GPT4oMini, Haiku]
strategy [CustomGPT4oMini, CustomHaiku]
}
}

// https://docs.boundaryml.com/docs/snippets/clients/fallback
client<llm> OpenaiFallback {
provider fallback
options {
// This will try the clients in order until one succeeds
strategy [GPT4o, GPT4oMini]
strategy [CustomGPT4oMini, CustomGPT4oMini]
}
}

// https://docs.boundaryml.com/docs/snippets/clients/retry
retry_policy Constant {
max_retries 3
// Strategy is optional
strategy {
type constant_delay
delay_ms 200
}
}

retry_policy Exponential {
max_retries 2
// Strategy is optional
strategy {
type exponential_backoff
delay_ms 300
mutliplier 1.5
max_delay_ms 10000
}
}

0 comments on commit 682dd66

Please sign in to comment.