Feature request: inference parallelisation

[hippalectryon-0]

As per the README, "The framework supports parallel evaluation of candidates". However, it seems that this only applies to the testing evaluation - not to the LLM querying itself.

Many problems, such as circle packing, have very small testing times compared to the querying times. I believe it would be highly beneficial to have some parallelization on the querying.

I expect this is not totally trivial - the current sequential approach ensures that the n+1th iteration can benefit from the n-th - but it should still be fairly easy to batch this logic.

Nb: I noticed some unused code e.g. AsyncLLMClient, was that an attempt in this direction ?

Best,

[RobertTLange]

Dear @hippalectryon-0,

Thank you so much! This is current work in progress and will likely be released in the next week. We are aware that while Shinka currently supports parallel program eval, for short program runtimes, the main bottleneck is LLM API querying. We are currently finalizing everything and will ship an AsyncEvolutionRunner asap. To share some current progress for a problem like circle packing the speed up is almost linear with respect to the resources (see image). Please note that the bottleneck shifts for longer running jobs though (e.g. the MoE loss optimization in our paper).

Will ping you again in this issue thread, once everything is ready and shipped.

[zia1138]

@RobertTLange I have a private fork of ShinkaEvolve that uses ray for parallelization. Happy to share if you are interested.