Hotfix/benchmarking engine and backends#96
Hotfix/benchmarking engine and backends#96catebros wants to merge 3 commits intohotfix/benchmarking_redesignfrom
Conversation
catebros
commented
Mar 27, 2026
catebros
commented
- Academy support: added AcademyOrchestrationFramework adapter alongside AutoGen/LangGraph
- Parsl fix: fixed clear() not properly resetting the Parsl backend between runs
- AutoGen events: renamed event emission in the AutoGen to align with the new event model
- Dummy model: added calls_per_tool param to DummyAutoGenClient
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the AcademyOrchestrator to support the Academy agent framework, accompanied by a new example and Makefile target. It also enhances the AutoGenOrchestrator and dummy client with improved event tracking and tool call capabilities, while ensuring the Parsl backend clears its configuration before loading. Feedback includes a recommendation to use asyncio.gather for concurrent execution in the example and a suggestion to refactor duplicated orchestration logic into a common base class to improve maintainability.
| temperature = await self._fetch_temperature(location=location) | ||
| humidity = await self._fetch_humidity(location=location) |
There was a problem hiding this comment.
The _fetch_temperature and _fetch_humidity calls are independent and can be executed concurrently. Using asyncio.gather will run these operations in parallel, which can improve performance by reducing the total execution time from ~4 seconds to ~2 seconds in this example.
| temperature = await self._fetch_temperature(location=location) | |
| humidity = await self._fetch_humidity(location=location) | |
| temperature, humidity = await asyncio.gather( | |
| self._fetch_temperature(location=location), | |
| self._fetch_humidity(location=location) | |
| ) |
| class AcademyOrchestrator(AgentOrchestrator): | ||
| def __init__(self, engine: BaseEngine) -> None: | ||
| self.engine = engine | ||
|
|
||
| def hpc_task(self, func: Callable): | ||
| """ | ||
| Wraps a function to execute as an HPC task. | ||
| """ | ||
| task_name = getattr(func, "__name__", str(func)) | ||
| wrap_id = str(uuid.uuid4()) | ||
|
|
||
| self.engine.emit( | ||
| { | ||
| "event": "tool_wrap_start", | ||
| "ts": time.perf_counter(), | ||
| "tool_name": task_name, | ||
| "wrap_id": wrap_id, | ||
| } | ||
| ) | ||
|
|
||
| @wraps(func) | ||
| async def wrapper(*args, **kwargs): | ||
| invocation_id = str(uuid.uuid4()) | ||
| self.engine.emit( | ||
| { | ||
| "event": "tool_invoke_start", | ||
| "ts": time.perf_counter(), | ||
| "tool_name": task_name, | ||
| "invocation_id": invocation_id, | ||
| } | ||
| ) | ||
| result = await self.engine.execute_tool( | ||
| func, *args, invocation_id=invocation_id, **kwargs | ||
| ) | ||
| self.engine.emit( | ||
| { | ||
| "event": "tool_invoke_end", | ||
| "ts": time.perf_counter(), | ||
| "tool_name": task_name, | ||
| "invocation_id": invocation_id, | ||
| } | ||
| ) | ||
| return result | ||
|
|
||
| self.engine.emit( | ||
| { | ||
| "event": "tool_wrap_end", | ||
| "ts": time.perf_counter(), | ||
| "tool_name": task_name, | ||
| "wrap_id": wrap_id, | ||
| } | ||
| ) | ||
|
|
||
| return wrapper | ||
|
|
||
| def hpc_block(self, block_func: Callable): | ||
| """ | ||
| Wraps a function to execute as an HPC block. | ||
| """ | ||
| block_name = getattr(block_func, "__name__", str(block_func)) | ||
| wrap_id = str(uuid.uuid4()) | ||
|
|
||
| self.engine.emit( | ||
| { | ||
| "event": "block_wrap_start", | ||
| "ts": time.perf_counter(), | ||
| "block_name": block_name, | ||
| "wrap_id": wrap_id, | ||
| } | ||
| ) | ||
|
|
||
| wrapped_block = self.engine.wrap_node(block_func) | ||
|
|
||
| self.engine.emit( | ||
| { | ||
| "event": "block_wrap_end", | ||
| "ts": time.perf_counter(), | ||
| "block_name": block_name, | ||
| "wrap_id": wrap_id, | ||
| } | ||
| ) | ||
|
|
||
| return wrapped_block |
There was a problem hiding this comment.
The implementation of AcademyOrchestrator is nearly identical to the updated AutoGenOrchestrator. Both hpc_task and hpc_block methods contain the same logic for event emission and wrapping. To improve maintainability and follow the DRY (Don't Repeat Yourself) principle, this duplicated code should be moved to the AgentOrchestrator base class.
1 participant
