feat(testset_generator): support error catching in generation process

[yuukidach]

I am using Azure OpenAI API to generate testset. However, Azure's API is sensitive to content and can easily trigger the following returns:

{'id': 'fake_id', 'choices': [{'finish_reason': 'content_filter', 'index': 0, 'message': {'content
': None, 'role': 'assistant', 'function_call': None, 'tool_calls': None}}], 'created': 1702175894, 'model': 'gpt-4-32k', 'object'
: 'chat.completion', 'system_fingerprint': None, 'usage': {'completion_tokens': 21, 'prompt_tokens': 4271, 'total_tokens': 4292}}

The finish_reason is content_filter and the content is None.

When conducting create_llm_result(), because the content is None, an error will be raised.

Error occurred: ('Fatal error occurred while running async tasks.', ValidationError(model='Generation', errors=[{'loc': ('text',)
, 'msg': 'none is not an allowed value', 'type': 'type_error.none.not_allowed'}]))

This error will interrupt the execution of the entire generation task. In fact, if such an error occurs, we can just skip it and moving on to generate another test item.

[jjmachan]

this PR looks good! Looping in @shahules786 for the final approval
Thanks @yuukidach for adding this PR :)

[jjmachan]

hey @yuukidach we are also working on refactoring the testset generator piece. I've shared the RFC here #380 and I would love to hear your feedback too. I can jump on a call and explain it more too if you want - let me know if you're interested :)

[shahules786]

LGTM

[jjmachan]

Hey @yuukidach

I must have sent this to you already but it is because we really mean it - Thank you so much for helping us improve ragas with your PR ❤️. Now since it's Christmas and all we wanted to send you a postcard and a couple of stickers as our way of saying thank you for your contribution. If you are interested could you shoot an email at [email protected] and I'll tell you more?

Cheers 🙂
Jithin

[yuukidach]

@jjmachan

Thank you so much for your kind message! I'm delighted to have been able to contribute to improving ragas, and receiving your appreciation means a lot.

I appreciate the offer of a postcard and stickers for Christmas, but I'm currently in another country, and I understand that it might be inconvenient for you to send them here. Nevertheless, I truly value your gesture, and knowing your goodwill has already made me happy.

I had a quick look at the refactoring of test_generator, and it seems quite meaningful. I'm curious whether you've considered introducing the concept of an agent, such as autogen. It appears to be a trend in the future applications of llm and aligns well with your ideas.

Wishing you a joyful Christmas season!

[jjmachan]

@yuukidach I hadn't considered it previously but it is interesting. I'm curious as to the concepts you linked in that if you can share