Support multi language testset generation

[grauvictor]

It doesn't seem that testset generation supports several types of language.

One solution would be to use an adapt function to translate the generation scripts in the same way as is done for metrics:

ragas/src/ragas/metrics/_faithfulness.py

Lines 203 to 212 in 27e48b0

	def adapt(self, language: str, cache_dir: t.Optional[str] = None) -> None:
	assert self.llm is not None, "LLM is not set"
	logger.info(f"Adapting Faithfulness metric to {language}")
	self.long_form_answer_prompt = self.long_form_answer_prompt.adapt(
	language, self.llm, cache_dir
	)
	self.nli_statements_message = self.nli_statements_message.adapt(
	language, self.llm, cache_dir
	)

[jjmachan]

yes, we are going to bring that same functionality to testset generation too - I'm glad you brought it up 🙂.
we also had a few ideas in our mind about this as well, could we run them by you sometime your free?

[grauvictor]

yes, we are going to bring that same functionality to testset generation too - I'm glad you brought it up 🙂. we also had a few ideas in our mind about this as well, could we run them by you sometime your free?

Thx, of course :)

[Gr33nLight]

Hello! I stumbled into the same issue, are there any updates on this, maybe a workaround? Thanks!! @jjmachan @grauvictor
I'm specifically using the generate_with_langchain_docs to generate initial test data.

[drdsgvo]

I would also be interested in a multi-language feature, as I am working with German texts.
Is there any update on this issue?

[shahules786]

Hey guys, Yes now this is possible with language adaptation for test generation. Just follow the guide here and enjoy :) Install ragas from the source before doing so.
@Gr33nLight @drdsgvo @grauvictor

[drdsgvo]

Thank you a lot for your quick and helpful answer.
I tried it out with German language and German wikipedia articles.
From 6 answers generated, 3 are not usable:

1 answer to a generated conditional question is empty
1 answer to a generated conditional question is (in English!): "Sorry, I cannot translate a negative number. Please provide a valid input"
1 answer to a generated reasoning question is (in English!): "Sorry I cannot answer this question as the information provided in the context does not mention... < here comes a question-specific problem>"

The other 3 answers are in German and as expected. All questions are good.

But with 3 out of 6 answers generated being no-answers, the approach is not feasible as of now.
Any ideas?

[shahules786]

Hey @drdsgvo , which models are you using?

[drdsgvo]

Hey @drdsgvo , which models are you using?

Just the default ones. I did not change anything and copy pasted your example code from your guide with some minor adaptations (like changing the language to 'german' insteadt of 'hindi')

[shahules786]

Thanks, bro @drdsgvo , I will work on it. Would also love to chat with you to understand your application for it, if you're free sometime this week or later. calendly

[drdsgvo]

Thanks, bro @drdsgvo , I will work on it. Would also love to chat with you to understand your application for it, if you're free sometime this week or later. calendly

With the synthetic questions and answers generated we want to see if we can train an llm from scratch for question answering with given context.

[shahules786]

Hey, @drdsgvo just raised a PR for #599 fixing some evolution flows which solves this problem (90% filling rate). Here is a sample of Spanish data I generated from Wikipedia

[drdsgvo]

Hey, @drdsgvo just raised a PR for #599 fixing some evolution flows which solves this problem (90% filling rate). Here is a sample of Spanish data I generated from Wikipedia <

Thank you very much for your quick response and coding. I tried it (updated to latest ragas version, no modifications in my code).
An exception is raised in line 75 of fe0bcc4

Exception message:
The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()