AttributeError: object has no attribute 'sentences'

[William8Work]

[X] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
I constructed a simple RAG with LLM for the evaluation. Got some NaN and many 0.0 in the results. Also saw this error during the evaluation:

ERROR:ragas.executor:Exception raised in Job[0]: AttributeError('StringIO' object has no attribute 'sentences')

Code Examples
Here is the sample code:

ragas_data = [
    {
        "user_input": entry["user_input"],
        "response": entry["response"],
        "retrieved_contexts": entry["retrieved_contexts"],
        "reference": entry["reference"]
    }
    for entry in sample_dataset
]

evaluation_dataset = EvaluationDataset.from_list(ragas_data)

evaluator_llm = LangchainLLMWrapper(ChatCohere(cohere_api_key=COHERE_API_KEY, model="command-r-plus-08-2024"))
    results = evaluate(
        dataset = dataset,
        metrics=[
            Faithfulness(llm=evaluator_llm),
            LLMContextRecall(llm=evaluator_llm),
            LLMContextPrecisionWithReference(llm=evaluator_llm),
            FactualCorrectness(llm=evaluator_llm)
        ],
        llm = evaluator_llm
    )

    df = results.to_pandas()

The output of the df shows faithfulness as Nan, the llm_context_precision_with_reference with 0.0, while the context_recall is 0.5, and factual_correctness 0.89.

During the evaluation, it printed the error:
ERROR:ragas.executor:Exception raised in Job[0]: AttributeError('StringIO' object has no attribute 'sentences')

This community speaks code. Share your code snippets to help us understand your question better.

Additional context
Anything else you want to share with us?

[jjmachan]

closed with #1864 will be released with v0.2.12 🙂

I'm closing this for now but if the issue is still persisting, please do let me know - really sorry about the delay