- 
                Notifications
    You must be signed in to change notification settings 
- Fork 2.4k
fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot #8909
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| @codex review | 
| ) | ||
|  | ||
| # Update max demos if specified | ||
| initial_max_bootstrapped_demos = self.max_bootstrapped_demos | 
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should avoid updating instance attributes as much as possible. Can we update methods that rely on these attributes (e.g., _bootstrap_fewshot_examples) to receive them as arguments?
…efore determining if the program was zero shot
9e6d5df    to
    2e898bc      
    Compare
  
    | Needs testing DNM | 
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
| @isaacbmiller feel free to merge once you run MIPRO with this code to confirm no regression! | 
| Verified via: import dspy
lm = dspy.LM("openai/gpt-5-nano", temperature=1.0, max_tokens=16000)
program = dspy.Predict("number_as_string: str -> answer_from_rule: float")
trainset = [
    dspy.Example(number_as_string="123", answer_from_rule=124).with_inputs("number_as_string"),
    dspy.Example(number_as_string="456", answer_from_rule=457).with_inputs("number_as_string"),
    dspy.Example(number_as_string="789", answer_from_rule=790).with_inputs("number_as_string"),
    dspy.Example(number_as_string="101112", answer_from_rule=101113).with_inputs("number_as_string"),
    dspy.Example(number_as_string="131415", answer_from_rule=131416).with_inputs("number_as_string"),
    dspy.Example(number_as_string="161718", answer_from_rule=161719).with_inputs("number_as_string"),
    dspy.Example(number_as_string="192021", answer_from_rule=192022).with_inputs("number_as_string"),
]
def metric_dspy(e, p, t=None):
    return 1 if p.answer_from_rule == e.answer_from_rule else 0
optimizer = dspy.MIPROv2(
    metric_dspy, 
    max_bootstrapped_demos = 1, 
    max_labeled_demos = 1, 
    prompt_model=lm, 
    task_model=lm,
    auto = "medium",
    num_threads=6)
program_optimized_no_demos = optimizer.compile(program, trainset=trainset, requires_permission_to_run = False, max_bootstrapped_demos=0, max_labeled_demos=0)
program_optimized_no_demos.save("program_optimized_no_demos.json")
program_optimized_with_demos = optimizer.compile(program, trainset=trainset, requires_permission_to_run = False)
program_optimized_with_demos.save("program_optimized_with_demos.json") | 
commit 056d54e Author: Isaac Miller <[email protected]> Date: Wed Oct 29 17:23:09 2025 +0100 fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot (stanfordnlp#8909) * fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot * remove extra logs * Remove log * Fix merge conflict * Remove extra whitespace commit da69f9d Author: TomuHirata <[email protected]> Date: Wed Oct 29 13:23:34 2025 +0900 Update anthropic model name (stanfordnlp#8992) Signed-off-by: TomuHirata <[email protected]> commit aaadf05 Author: Chen Qian <[email protected]> Date: Tue Oct 28 12:21:55 2025 -0700 lints (stanfordnlp#8987) commit e842ba1 Author: eramis73 <[email protected]> Date: Tue Oct 28 02:40:34 2025 +0300 [docs] Add Google-style docstrings for dspy/evaluate/metrics.py (stanfordnlp#8954) * docs(metrics): add Google-style docstrings for public metrics * docs(metrics): address review feedback (concise openings, mkdocs block examples); revert non-doc changes * fixes --------- Co-authored-by: chenmoneygithub <[email protected]> commit 6c43880 Author: TomuHirata <[email protected]> Date: Tue Oct 28 07:21:06 2025 +0900 Cache Ollama to speed up CI (stanfordnlp#8972) * Cache Ollama to speed up CI * fix permission commit 462baef Author: Copilot <[email protected]> Date: Mon Oct 27 11:57:27 2025 -0700 Fix TypeError when tracking usage with Anthropic models returning Pydantic objects (stanfordnlp#8978) * Initial plan * Fix TypeError when merging Anthropic CacheCreation objects in usage tracker Co-authored-by: TomeHirata <[email protected]> * Enhance _flatten_usage_entry to convert Pydantic models on first add Co-authored-by: TomeHirata <[email protected]> * Fix potential TypeError when both usage entries are None Co-authored-by: TomeHirata <[email protected]> * simplify * small fix * lint * robust version handling --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: TomeHirata <[email protected]> Co-authored-by: chenmoneygithub <[email protected]> commit 9b467b5 Author: Noah Ziems <[email protected]> Date: Mon Oct 27 13:32:07 2025 -0400 Add Disable Fallback Option in ChatAdapter (stanfordnlp#8984) commit bf022c7 Author: Lakshya A Agrawal <[email protected]> Date: Sat Oct 25 23:37:42 2025 +0530 Update gepa[dspy] dependency version to 0.0.18 (stanfordnlp#8969) * Update gepa[dspy] dependency version to 0.0.18 * Update pyproject.toml * fix test --------- Co-authored-by: TomuHirata <[email protected]>
commit 31b96af Author: Dushmanta <[email protected]> Date: Thu Oct 30 13:52:40 2025 +0530 fix: broken PyPI downloads badge from pepy.tech in README and docs home page (stanfordnlp#8995) * fix: update broken pypi download badge in readme * fix: update broken pypi download badge in docs home page commit 056d54e Author: Isaac Miller <[email protected]> Date: Wed Oct 29 17:23:09 2025 +0100 fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot (stanfordnlp#8909) * fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot * remove extra logs * Remove log * Fix merge conflict * Remove extra whitespace commit da69f9d Author: TomuHirata <[email protected]> Date: Wed Oct 29 13:23:34 2025 +0900 Update anthropic model name (stanfordnlp#8992) Signed-off-by: TomuHirata <[email protected]> commit aaadf05 Author: Chen Qian <[email protected]> Date: Tue Oct 28 12:21:55 2025 -0700 lints (stanfordnlp#8987) commit e842ba1 Author: eramis73 <[email protected]> Date: Tue Oct 28 02:40:34 2025 +0300 [docs] Add Google-style docstrings for dspy/evaluate/metrics.py (stanfordnlp#8954) * docs(metrics): add Google-style docstrings for public metrics * docs(metrics): address review feedback (concise openings, mkdocs block examples); revert non-doc changes * fixes --------- Co-authored-by: chenmoneygithub <[email protected]> commit 6c43880 Author: TomuHirata <[email protected]> Date: Tue Oct 28 07:21:06 2025 +0900 Cache Ollama to speed up CI (stanfordnlp#8972) * Cache Ollama to speed up CI * fix permission commit 462baef Author: Copilot <[email protected]> Date: Mon Oct 27 11:57:27 2025 -0700 Fix TypeError when tracking usage with Anthropic models returning Pydantic objects (stanfordnlp#8978) * Initial plan * Fix TypeError when merging Anthropic CacheCreation objects in usage tracker Co-authored-by: TomeHirata <[email protected]> * Enhance _flatten_usage_entry to convert Pydantic models on first add Co-authored-by: TomeHirata <[email protected]> * Fix potential TypeError when both usage entries are None Co-authored-by: TomeHirata <[email protected]> * simplify * small fix * lint * robust version handling --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: TomeHirata <[email protected]> Co-authored-by: chenmoneygithub <[email protected]> commit 9b467b5 Author: Noah Ziems <[email protected]> Date: Mon Oct 27 13:32:07 2025 -0400 Add Disable Fallback Option in ChatAdapter (stanfordnlp#8984) commit bf022c7 Author: Lakshya A Agrawal <[email protected]> Date: Sat Oct 25 23:37:42 2025 +0530 Update gepa[dspy] dependency version to 0.0.18 (stanfordnlp#8969) * Update gepa[dspy] dependency version to 0.0.18 * Update pyproject.toml * fix test --------- Co-authored-by: TomuHirata <[email protected]>
The
zeroshot_optoption, which determines if the proposer should use demos or not, was being set before applying the compile time arguments passed into the.compile functionFixes #8508