Description
[X] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question
I constructed a simple RAG with LLM for the evaluation. Got some NaN and many 0.0 in the results. Also saw this error during the evaluation:
ERROR:ragas.executor:Exception raised in Job[0]: AttributeError('StringIO' object has no attribute 'sentences')
Code Examples
Here is the sample code:
ragas_data = [
{
"user_input": entry["user_input"],
"response": entry["response"],
"retrieved_contexts": entry["retrieved_contexts"],
"reference": entry["reference"]
}
for entry in sample_dataset
]
evaluation_dataset = EvaluationDataset.from_list(ragas_data)
evaluator_llm = LangchainLLMWrapper(ChatCohere(cohere_api_key=COHERE_API_KEY, model="command-r-plus-08-2024"))
results = evaluate(
dataset = dataset,
metrics=[
Faithfulness(llm=evaluator_llm),
LLMContextRecall(llm=evaluator_llm),
LLMContextPrecisionWithReference(llm=evaluator_llm),
FactualCorrectness(llm=evaluator_llm)
],
llm = evaluator_llm
)
df = results.to_pandas()
The output of the df shows faithfulness as Nan, the llm_context_precision_with_reference with 0.0, while the context_recall is 0.5, and factual_correctness 0.89.
During the evaluation, it printed the error:
ERROR:ragas.executor:Exception raised in Job[0]: AttributeError('StringIO' object has no attribute 'sentences')
This community speaks code. Share your code snippets to help us understand your question better.
Additional context
Anything else you want to share with us?