Skip to content

NameError when evaluating the llamaindex query engine #2330

@hzhaoy

Description

@hzhaoy

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When evaluating the llamaindex query engine, I get a NameError: name 'EvaluationResult' is not defined.

Ragas version: 0.3.5
Python version: 3.13.3

Code to Reproduce
https://docs.ragas.io/en/stable/howtos/integrations/_llamaindex/

Error trace

/Users/admin/Documents/Projects/AgenticSystem/.venv/lib/python3.13/site-packages/pydantic/_internal/_generate_schema.py:2225: UnsupportedFieldAttributeWarning: The 'validate_default' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'validate_default' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
  warnings.warn(
Running Query Engine: 100%|███████████████████████████████████████████████████| 5/5 [00:05<00:00,  1.10s/it]
Evaluating:   0%|                                                                    | 0/20 [00:00<?, ?it/s]n values greater than 1 not support for LlamaIndex LLMs
n values greater than 1 not support for LlamaIndex LLMs
n values greater than 1 not support for LlamaIndex LLMs
n values greater than 1 not support for LlamaIndex LLMs
Exception raised in Job[9]: IndexError(list index out of range)
Exception raised in Job[13]: IndexError(list index out of range)
n values greater than 1 not support for LlamaIndex LLMs
Evaluating:   5%|███                                                         | 1/20 [00:04<01:24,  4.46s/it]Exception raised in Job[5]: IndexError(list index out of range)
Evaluating:  15%|█████████                                                   | 3/20 [00:12<01:08,  4.05s/it]Exception raised in Job[1]: IndexError(list index out of range)
Evaluating:  40%|████████████████████████                                    | 8/20 [00:13<00:15,  1.32s/it]Exception raised in Job[17]: IndexError(list index out of range)
Evaluating: 100%|███████████████████████████████████████████████████████████| 20/20 [00:26<00:00,  1.31s/it]
Traceback (most recent call last):
  File "/Users/admin/Documents/Projects/AgenticSystem/src/evaluation/ragas_test.py", line 99, in <module>
    result = evaluate(
        query_engine=query_engine,
    ...<2 lines>...
        run_config=RunConfig(timeout=6000),
    )
  File "/Users/admin/Documents/Projects/AgenticSystem/.venv/lib/python3.13/site-packages/ragas/integrations/llama_index.py", line 107, in evaluate
    return t.cast(EvaluationResult, results)
                  ^^^^^^^^^^^^^^^^
NameError: name 'EvaluationResult' is not defined. Did you mean: 'EvaluationDataset'?

Expected behavior
Successfully execute the evaluation process.

Another question
In addition, I see some warnings and exceptions in the program output. Will this affect the evaluation results?

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingquestionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions