Skip to content

Commit 6a95883

Browse files
committed
Update docstring
1 parent 30404e9 commit 6a95883

File tree

1 file changed

+33
-43
lines changed

1 file changed

+33
-43
lines changed

chatlas/_chat.py

Lines changed: 33 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -809,69 +809,59 @@ def console(
809809

810810
def to_solver(self):
811811
"""
812-
Return an InspectAI solver that proxies prompts through this chat.
812+
Create an InspectAI solver from this chat.
813813
814-
This method creates an InspectAI solver that uses this Chat instance to
815-
generate responses. The solver automatically translates the chat's
816-
conversation history (including system prompt, previous turns, and rich
817-
content) into InspectAI's message format.
814+
Translates this Chat instance into an InspectAI solver function that can
815+
be used with InspectAI's evaluation framework. This solver will capture
816+
(and translate) important state from the chat, including the model,
817+
system prompt, previous turns, registered tools, model parameters, etc.
818818
819819
Returns
820820
-------
821-
Solver
822-
An InspectAI solver function that can be used with InspectAI's
823-
evaluation framework.
824-
825-
For more information on InspectAI solvers, see the
826-
[Solvers documentation](https://inspect.ai-safety-institute.org.uk/solvers.html).
821+
An [InspectAI solver](https://inspect.ai-safety-institute.org.uk/solvers.html)
822+
function that can be used with InspectAI's evaluation framework.
827823
828824
Examples
829825
--------
830-
First, put this code in a python script, perhaps named `eval_chat.py`
831-
832-
```python
833-
from chatlas import ChatOpenAI
834-
import inspect_ai as iai
835-
from inspect_ai.dataset import Sample
836-
from inspect_ai.scorer import model_graded_fact
837-
838-
chat = ChatOpenAI(
839-
system_prompt="You are a helpful assistant. Be concise.",
840-
model="gpt-5-nano-2025-08-07",
841-
)
842-
843-
# Convert to InspectAI solver
844-
solver = chat.to_solver()
845-
846-
# InspectAI evaluation task
847-
task = iai.Task(
848-
dataset=[Sample(input="What is 2+2?", target="4")],
849-
solver=solver,
850-
scorer=model_graded_fact()
851-
)
826+
First, put this code in a python script, perhaps named `eval_chat.py`
852827
853-
iai.eval(task, model="openai/gpt-5-nano-2025-08-07")
854-
# Or, if running interactively in Jupyter/Positron
855-
# await iai.eval_async(task, model="openai/gpt-5-nano-2025-08-07")
856-
857-
```
828+
```{.python filename="eval_chat.py"}
829+
from chatlas import ChatOpenAI
830+
from inspect_ai import Task, task
831+
from inspect_ai.dataset import csv_dataset
832+
from inspect_ai.scorer import model_graded_qa
833+
834+
chat = ChatOpenAI(system_prompt="You are a helpful assistant.")
835+
836+
@task
837+
def my_eval(grader_model: str = "openai/gpt-4o"):
838+
return Task(
839+
dataset=csv_dataset("my_eval_dataset.csv"),
840+
solver=chat.to_solver(),
841+
scorer=model_graded_qa(model=grader_model)
842+
)
843+
```
858844
859-
Now, from a terminal, run the evaluation and view the results
845+
Then run the evaluation with InspectAI's CLI:
860846
861847
```bash
862-
python eval_chat.py
863-
inspect view
848+
inspect eval eval_chat.py -T --grader-model openai/gpt-4o
864849
```
850+
851+
Note
852+
----
853+
Learn more about this method and InspectAI's evaluation framework
854+
in the [Chatlas documentation](https://posit-dev.github.io/chatlas/misc/evals.html).
865855
"""
866856

867857
try:
868858
import inspect_ai.model as imodel
869859
import inspect_ai.solver as isolver
870-
except ImportError as exc: # pragma: no cover - optional dependency
860+
except ImportError as e: # pragma: no cover - optional dependency
871861
raise ImportError(
872862
"Chat.to_solver() requires the optional dependency `inspect-ai`. "
873863
"Install it with `pip install inspect-ai`."
874-
) from exc
864+
) from e
875865

876866
from ._inspect import content_to_chatlas, turn_as_messages
877867

0 commit comments

Comments
 (0)