Skip to content

max_worker evaluate option not passed anywhere? #587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
stdweird opened this issue Feb 9, 2024 · 1 comment
Closed

max_worker evaluate option not passed anywhere? #587

stdweird opened this issue Feb 9, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@stdweird
Copy link

stdweird commented Feb 9, 2024

Describe the bug
the max_worker option is not passed anywhere when runing evaluate in sync mode (the default). the asyncio pool is way too large by default

Ragas version: latest
Python version: 3.11

Code to Reproduce

Error trace

Expected behavior
we can control the number of workers during sync evaluate

Additional context

@stdweird stdweird added the bug Something isn't working label Feb 9, 2024
@stdweird
Copy link
Author

will be fixed by #643, i had a monkey patch with that is based on similar approach

# monkeypatch concurrency into executor with a semaphore
from ragas.executor import Executor
import asyncio
import typing
def new_wrap_callable_with_index(self, callable: typing.Callable, counter):
    if not hasattr(self, 'sema'):
        self.sema = asyncio.Semaphore(1)

    async def wrapped_callable_async(*args, **kwargs):
        async with self.sema:
            return counter, await callable(*args, **kwargs)

    return wrapped_callable_async
Executor.wrap_callable_with_index = new_wrap_callable_with_index
# end monkeypatch

@stdweird stdweird closed this as completed Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants