Async Executor/Runner slows to a halt with jobs that auto-retry with default (high) max_wait

[joy13975]

Describe the bug
Anything that a) has a throttle limit and b) uses Executor (and in turn, Runner), such as OpenAI's API, will slow to a halt if too many jobs are requested.

Specifically, in my case I was attempting to generate testset for ~1k documents.

testset = generator.generate_with_langchain_docs(documents, test_size=N, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

This refuses to even start if either there are too many documents (roughly >300) or too high test_size. Example with 500 docs stuck at 0% after 20min:

Reducing test_size didn't help with high document count because the place that got stuck was at docstore.add_documents. However, reducing both document count and test_size to < 100 did get it going at reasonable speed.

Ragas version: 0.1.2.dev8+gc18c7f4
Python version: 3.9.13

Code to Reproduce
Below is just one example that uses Executor & Runner with higher job count (i.e. 1,000). Document contens averaged ~800 characters. OpenAI API is being used as LLM.

testset = generator.generate_with_langchain_docs(documents, test_size=N, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

Error trace
No error, just gets stuck.

Expected behavior
Don't get stuck with high job count.

Additional context
Will add my findings in comments below. Related: #394

[joy13975]

Problem Investigation
The Executor async job code was initially introduced by bad3a8e and later separated into Runner. The problem is that Runner starts all jobs at once (here) and then later uses a asyncio.as_completed() (here) to wait for all jobs to be completed, but doesn't impose any max concurrency.

When too many jobs (i.e. >hundreds) are calling the same API, a throughput limit is hit and many of the jobs need to wait & retry. With the default RunConfig, wait time can be up to 60s 90s. I suspect this is the culrprit of the stuckness.

Proposal
The Executor/Runner should facilitate a maximum concurrency limit so as to avoid kicking all the remaining (and possibly on-the-fly?) requests into wait & retry loop once throughput limit gets hit. Even if that's not a good reason (for the admittedly lack of tracing down the actual reason of getting stuck), it's still a good knob to have for users who don't want to be constantly hitting throughput limits.

PR created: #643 please review.

[joy13975]

In case this doesn't get attention, excuse my mention @jjmachan

[wangyiran33]

Problem Investigation The Executor async job code was initially introduced by bad3a8e and later separated into Runner. The problem is that Runner starts all jobs at once (here) and then later uses a asyncio.as_completed() (here) to wait for all jobs to be completed, but doesn't impose any max concurrency.

When too many jobs (hundreds) are calling the same API, a throughput limit is hit and many of the jobs need to wait & retry. With the default RunConfig, wait time can be up to 60s. I suspect this is the culrprit of the stuckness.

Proposal The Executor/Runner should facilitate a maximum concurrency limit so as to avoid kicking all the remaining requests (once throughput limit gets hit) into wait & retry loop.

PR created: #643 please review.

same problem, when the size of document chunks > 500 the program will probably stuck. appreciate your PR.

[joy13975]

#486 seems to have added max_workers to the evaluate function but did not reference it. @jjmachan correct me if I'm wrong.

[jjmachan]

hey @joy13975 really sorry I have not been able to get to this - but for the last commend you are right. max_workers are an old arg we dont' use

as for your PR - really really appreciate it ❤️ , will get it reviewed both and respond tonight

thanks again for your help bringing this up 🙌🏽

[jjmachan]

thanks to @joy13975 we have fixed this guys 🙂