Question on leaderboard of ALE-bench

[Jasonwu-0803]

Hi, recently we are working on a evolution based llm coding system (similar to AlphaEvolve and ShinkaEvolve), and we are having a better result (Avg. Perf) on ALE-Bench lite. I noticed you just changed your leaderboard, and it focused more on model comparison instead of comparison between different llm system. I wonder if there is a leaderboard for llm system and what kind of information is required to be listed on the leaderboard.

[Yuki-Imajuku]

Thank you for your interest in ALE-Bench!!!
As you said, the current leaderboard intentionally focuses on model comparison because we first want to understand the base performance of all models and their test-time scaling (through simple self-refine).
I imagine agent systems vary greatly depending on the LLMs and scaffolding (e.g. memory, tools...) used and I think they should be compared against the base LLM's performance to isolate the contribution of the agent's architecture.
If we find more agent papers trying out ALE-Bench, we'd be happy to create a separate (table-like) leaderboard and receive results from the research community.

[Jasonwu-0803]

Totally agree with you. Model is definitely deterministic factor of solving problems like ALE. Very glad to see the comparisons of all models on this much harder problems involves both coding and reasoning.
Since we are also using Gemini-2.5-pro (same as ALE-Agent), we think our results do provide some insights. We'd love to share our results. Our paper will also release recently.

	overall	ahc008	ahc011	ahc015	ahc016	ahc024	ahc025	ahc026	ahc027	ahc039	ahc046
ALE-Agent	1879.3	1189	1652	2446	1457	1980	1331	1965	1740	2880	2153
FM-Agent	1976.3	1312	1662	2543	2148	2241	1588	2123	1799	2525	1815