Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for dynamic types tests #1307

Open
wants to merge 3 commits into
base: canary
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion fern/01-guide/04-baml-basics/testing-functions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -313,4 +313,15 @@ test MyTest {
the remainder of the checks and asserts in this particular test.

For more information about the syntax used inside `@@check` and `@@assert`
attributes, see [Checks and Asserts](/guide/baml-advanced/checks-and-asserts)
attributes, see [Checks and Asserts](/guide/baml-advanced/checks-and-asserts)

## Dynamic Types Tests

Classes and enums marked with the [`@@dynamic`](/ref/baml-client/type-builder)
attribute can be modified in tests using the `type_builder` and `dynamic`
blocks.

<Markdown src="../../snippets/dynamic-class-test.mdx" />

The `type_builder` block can contain new types scoped to the parent `test` block
and also `dynamic` blocks that act as modifiers for dynamic classes or enums.
171 changes: 169 additions & 2 deletions fern/01-guide/05-baml-advanced/dynamic-types.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,173 @@ puts res
</CodeBlocks>

### Testing dynamic types in BAML
This feature is coming soon! Let us know if you're interested in testing it out!

You can still write tests in Python, TypeScript, Ruby, etc in the meantime.
When testing dynamic types there are two different cases:
1. Injecting properties into dynamic types returned by the tested function.
2. Injecting values into dynamic types received as arguments by the tested function.

The first case requires using the `type_builder` and `dynamic` blocks in the
test, whereas the second case only requires specifying the values in the `args`
block.

#### Testing return types

##### Dynamic classes

Suppose we have a dynamic class `Resume` and we want to add a property that
stores the user's work experience when we testing a specific function. We can
do that by specifying the types and properties that we need in the
`type_builder` block.


```baml {4, 14-27}
class Resume {
name string
skills string[]
@@dynamic // Marked as @@dynamic.
}

// Function that returns a dynamic class.
function ExtractResume(from_text: string) -> Resume {
// Prompt
}

test ReturnDynamicClassTest {
functions [ExtractResume]
type_builder {
// Defines a new type available only within this test block.
class Experience {
title string
company string
start_date string
end_date string
}

// Injects new properties into the `@@dynamic` part of the Resume class.
dynamic Resume {
experience Experience[]
}
}
args {
from_text #"
John Doe

Experience
- Software Engineer, Boundary, Sep 2022 - Sep 2023

Skills
- Python
- Java
"#
}
}
```

The rendered prompt for `ExtractResume` will now include the `experience` field
defined in the `dynamic` block and the LLM will correctly extract the experience
in the input text.

##### Dynamic enums

Dynamic enums can be included in the `type_builder` block just like classes. The
only difference is that we inject new variants in the `dynamic` block instead of
properties.

```baml {7, 17-22}
enum Category {
Refund
CancelOrder
TechnicalSupport
AccountIssue
Question
@@dynamic // Marked as @@dynamic.
}

// Function that returns a dynamic enum.
function ClassifyMessage(message: string) -> Category {
// Prompt
}

test ReturnDynamicEnumTest {
functions [ClassifyMessage]
type_builder {
// Injects new variants into the `@@dynamic` part of the Category enum.
dynamic Category {
Feedback
}
}
args {
message "I think the product is great!"
}
}
```

The `Feedback` variant will be rendered in the prompt for `ClassifyMessage`
during the test execution.

#### Testing parameter types

When a dynamic type is used as an input parameter of a function, we can simply
pass any value in the `args` block of the test and the value will be rendered in
the prompt.

##### Dynamic classes

```baml {4, 17-24}
class Resume {
name string
skills string[]
@@dynamic // Marked as @@dynamic.
}

function WriteResume(resume: Resume) -> string {
// Prompt
}

test DynamicClassAsInputTest {
functions [WriteResume]
args {
resume {
name "John Doe"
skills ["C++", "Java"]
experience [
{
title "Software Engineer"
company "Boundary"
start_date "2023-09-01"
end_date "2024-09-01"
}
]
}
}
}
```

##### Dynamic enums

Enums work the same way, any variant defined in the `args` block will be
rendered normally.

```baml {7, 17}
enum Category {
Refund
CancelOrder
TechnicalSupport
AccountIssue
Question
@@dynamic // Marked as @@dynamic.
}

function WriteCustomerMessage(category: Category) -> string {
// Prompt
}

test DynamicEnumAsInputTest {
functions [WriteCustomerMessage]
args {
category Feedback // The enum is dynamic so it accepts a new variant.
}
}
```

For more information about dynamic types, see [Type Builder](/ref/baml-client/type-builder).
8 changes: 8 additions & 0 deletions fern/03-reference/baml/attributes/dynamic.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,11 @@ def run
end
```

## Testing Dynamic Types

Dynamic classes and enums can be modified in tests using the `type_builder` and
`dynamic` blocks. All properties added in the `dynamic` block will be available
during the test execution.

<Markdown src="../../../snippets/dynamic-class-test.mdx" />

34 changes: 33 additions & 1 deletion fern/03-reference/baml/test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ A BAML test consists of:
- Input arguments
- Optional testing configuration
- Optional assertions
- Optional type builders

```baml
test TestName {
Expand All @@ -20,7 +21,7 @@ test TestName {

## Test Declaration

### Syntax
### Basic Syntax

```baml
test name {
Expand All @@ -31,11 +32,36 @@ test name {
}
```

### Optional Features

```baml {3-11, 15, 16}
test name {
functions [function_list]
type_builder {
class NewType {
// Props
}
dynamic ExistingDynamicType {
new_prop NewType
// Inject Props Here
}
}
args {
parameter_assignments
}
@@check( check_length, {{ this.prop|length > 0 }} )
@@assert( {{ this.prop|length < 255 }})
}
```

### Components

- `name`: Test identifier (unique per function)
- `functions`: List of functions to test
- `args`: Input parameters for the test case
- `type_builder`: Block used to inject values into dynamic types
- `@@check`: Conditional check for test validity
- `@@assert`: Assertion for test result

## Input Types

Expand Down Expand Up @@ -212,6 +238,12 @@ test EndToEndFlow {
}
```

## Testing Dynamic Types

Dynamic types can be tested using `type_builder` and `dynamic` blocks:

<Markdown src="../../snippets/dynamic-class-test.mdx" />

## Integration with Development Tools

### VSCode Integration
Expand Down
9 changes: 8 additions & 1 deletion fern/03-reference/baml_client/typebuilder.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,12 @@ tb.User.add_property("middle_name", tb.string.optional)
All types added through TypeBuilder must be connected to the return type of your BAML function. Standalone types that aren't referenced won't affect the output schema.
</Warning>

## Testing Dynamic Types

See the [advanced dynamic types tests guide](/guide/baml-advanced/dynamic-runtime-types#testing-dynamic-types-in-baml)
for examples of testing functions that use dynamic types. See also the
[reference](/ref/baml/test) for syntax.

## Future Features

We're working on additional features for TypeBuilder:
Expand All @@ -335,4 +341,5 @@ We're working on additional features for TypeBuilder:
- OpenAPI schema integration
- Pydantic model support

If you're interested in these features, please join the discussion in our GitHub issues.
If you're interested in these features, please join the discussion in our GitHub
issues.
22 changes: 22 additions & 0 deletions fern/snippets/dynamic-class-test.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
```baml {3, 12-16}
class DynamicClass {
static_prop string
@@dynamic
}

function ReturnDynamicClass(input: string) -> DynamicClass {
// ...
}

test DynamicClassTest {
functions [ReturnDynamicClass]
type_builder {
dynamic DynamicClass {
new_prop_here string
}
}
args {
input "test data"
}
}
```
Loading