Instability in answer relevance score

[laughinghugs]

For evaluating my RAG pipeline, in real time, I am implementing 'Answer Relevance' score. But what I found that for same set of question, answer, and context it is generating different scores. And these scores are varying by 10-15%. I understand as it is a probabilistic measure (using LLM, re-engineering questions from answer etc.), the score may not be the same always. But varying 10-15% is a trust issue. Any solution to this?

[jjmachan]

hey @laughinghugs this is a problem we are aware of and hoping to solve it with approaches like the once mentioned in https://blog.ragas.io/aligning-llm-as-judge-with-human-evaluators.

@shahules786 would you have any suggestions