Allow iterative refinement of `program.md` based on experiment results alongside `train.py`

[morozow]

I would like to propose an additional research direction for autoresearch.

Right now the agent mainly improves the training loop by modifying train.py and validating the result through experiments. This is useful, but it keeps the optimization focused on the training code itself. A possible next step would be to also let the system iteratively refine its own research guidance in program.md.

The idea is simple:

• the agent runs experiments and collects outcomes;
• then, based on successful and failed changes, it updates the prompt and/or program that guides the next phase of research;
• the next iterations are therefore influenced not only by code changes in train.py, but also by improved search strategy, priorities, and heuristics in program.md.

Cause is effect here is direct:

• if the agent only edits train.py, it optimizes the object of research;
• if it can also refine program.md, it begins to optimize the method of research itself;
• this can reduce random local changes, improve search discipline, and make later iterations more informed.

In other words, the system could preserve the current train.py mutation loop, while adding a second loop where the agent improves the prompt – program.md – that drives future mutations. A constrained version may be especially practical: not rewriting the whole program freely, but updating only specific strategy sections such as hypothesis priorities, failed-pattern memory, search order, or mutation heuristics. One important detail here is that the current constraints are explicit for train.py, prepare.py, dependencies, and the evaluation harness, but there is no equally explicit prohibition against modifying program.md. In that sense, program.md is paradoxically not just the instruction layer of the system, but also a potential research object itself. If train.py is the object-level mutation target, then program.md may be viewed as the nlp-level mutation target, because it shapes how future mutations are generated and prioritized. This suggests a possible second optimization loop: not only improving the training code, but also iteratively refining the research program that guides the next experimental phase.

This looks interesting because it turns the process from "edit training code and test" into "edit training code, learn from outcomes, and improve the research strategy for the next round". I think this could make the frontier AI research stronger not only as an experiment runner, but as a system that incrementally improves its own research behaviour.

Would be interesting to hear whether this direction has already been considered.

[GiladLeef]

Why not rely on the results file directly instead of letting the agent rewrite program.md? The outcomes already provide a grounded signal for what works and what doesn’t, without introducing another layer of self-modification that could degrade over time.

I’m skeptical about letting the model guide the research instructions. These systems tend to have weak intuition when it comes to long-horizon research planning... they’re much stronger at local, incremental optimization than at deciding what to explore next.

[morozow]

@GiladLeef Thanks for the feedback. I studied the program.md breakdown in detail, and there are really only two places where we have any opportunity to modify something – to give potential for more productive agent activity. So we also have the results.tsv file, which is exactly where potential can be introduced through optimization of program.md.

Your concern about agent's program self-modification is valid. Expanding or transforming it further is probably premature and pointless right now. So this is exactly where the implementation fits. I extended results.tsv with a hypothesis column – not restating what was tried, but non-obvious combinations, conditions, and unexplored connections. The agent records these during experiments and can revisit them when stuck.

One more thing: the PR mentions A/B testing for positioning "gather hypotheses" at either the beginning or end of the "out of ideas" list. The point is that the agent modification I was talking about could be specifically in changing the order of the hypothesis instruction. The agent could allow itself to place it before or after depending on how relevant it is at the moment. This kind of policy could be the entry point for agent-driven transformation of program.md.

PR with implementation: #336