Add support for optional max concurrency

[joy13975]

Added optional Semaphore-based concurrency control for #642
As for the default value for max_concurrency, I don't know the ratio of API users vs. local LLM users, so the proposed default is an opinionated value of 16

• I think more people use OpenAI API for now vs. local LLMs, thus default is not -1 (no limit)
• 16 seems to be reasonably fast and doesn't seem to hit throughput limit in my experience

Tests
Embedding for 1k documents finished in <2min and subsequent Testset generation for test_size=1000 proceeding without getting stuck:

another 30s passes:

[joy13975]

Note: currently have test_size=500 done without problem (at 1k docs).
test_size=1000 still takes over an hour but does properly proceed rather than getting stuck at any %.

[wangyiran33]

Thanks! try to run but encounter a problemAs of 3.10, the *loop* parameter was removed from Semaphore() since it is no longer necessary

[jjmachan]

I love this PR - thank you so much @joy13975 ❤️ - really! thank you for sharing your fix.

I just have one commend though. So I was thinking about the RunConfig and wondering it would be a good place to add this one too. the goal of that abstraction is that anything a user wants to configure for running - they can configure it through that and it will be our responsibility to make sure it gets used in the right place.

so as a user would you prefer it to be in RunConfig is that an overkill?

That was the only question I had but otherwise, I'm merging this in!

[joy13975]

@jjmachan
Totally overlooked putting it in RunConfig, but I think that might be neater than this. I’ll sort that out and put up another PR.

@wangyiran33 thanks for the report and l’ll make it compatible with 3.10 as well in the next PR update.

[jjmachan]

makes perfect sense @joy13975 - there seems to be a small type error - if you fix that I'll merge this in

I can fix it too if you want but it will take some time for me to get to this sadly but the PR looks really good so would be awesome to merge it as soon as we can right.

[jjmachan]

or not - because if we don't have support for 3.10 that is a deal breaker - thanks for reporting that @wangyiran33
we might have to refer to the stuff they did

would love to hear your thoughts @joy13975

[joy13975]

@jjmachan yeah I’ll fix all the above points once I get time. Should be up this weekend.

[jjmachan]

sounds good - thanks a lot again
lets get this bug fixed 🔥

[joy13975]

@jjmachan Just pushed an update to address all the above. @wangyiran33 Can you please help test on Python 3.10+?
Won't be able to increase my OpenAI account usage limit for a week..

[wangyiran33]

@jjmachan Just pushed an update to address all the above. @wangyiran33 Can you please help test on Python 3.10+? Won't be able to increase my OpenAI account usage limit for a week..

of course! I have tested and it works for py3.10, really appreciate that. Love this PR! A big Thanks for you! @joy13975

[jjmachan]

awesome - finished my round of testing and it is a massive improvement ❤️

thanks again for this @joy13975 for helping us all and contributing this fix 🙂

going out in today's release