Skip to content

Conversation

@isaacbmiller
Copy link
Collaborator

The zeroshot_opt option, which determines if the proposer should use demos or not, was being set before applying the compile time arguments passed into the .compile function

Fixes #8508

@isaacbmiller
Copy link
Collaborator Author

@codex review

)

# Update max demos if specified
initial_max_bootstrapped_demos = self.max_bootstrapped_demos
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid updating instance attributes as much as possible. Can we update methods that rely on these attributes (e.g., _bootstrap_fewshot_examples) to receive them as arguments?

@isaacbmiller isaacbmiller force-pushed the isaac/mipro-few-shot-bug branch from 9e6d5df to 2e898bc Compare October 7, 2025 13:37
@isaacbmiller
Copy link
Collaborator Author

Needs testing DNM

Copy link
Collaborator

@TomeHirata TomeHirata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TomeHirata
Copy link
Collaborator

@isaacbmiller feel free to merge once you run MIPRO with this code to confirm no regression!

@isaacbmiller
Copy link
Collaborator Author

Verified via:

import dspy

lm = dspy.LM("openai/gpt-5-nano", temperature=1.0, max_tokens=16000)

program = dspy.Predict("number_as_string: str -> answer_from_rule: float")
trainset = [
    dspy.Example(number_as_string="123", answer_from_rule=124).with_inputs("number_as_string"),
    dspy.Example(number_as_string="456", answer_from_rule=457).with_inputs("number_as_string"),
    dspy.Example(number_as_string="789", answer_from_rule=790).with_inputs("number_as_string"),
    dspy.Example(number_as_string="101112", answer_from_rule=101113).with_inputs("number_as_string"),
    dspy.Example(number_as_string="131415", answer_from_rule=131416).with_inputs("number_as_string"),
    dspy.Example(number_as_string="161718", answer_from_rule=161719).with_inputs("number_as_string"),
    dspy.Example(number_as_string="192021", answer_from_rule=192022).with_inputs("number_as_string"),
]


def metric_dspy(e, p, t=None):
    return 1 if p.answer_from_rule == e.answer_from_rule else 0

optimizer = dspy.MIPROv2(
    metric_dspy, 
    max_bootstrapped_demos = 1, 
    max_labeled_demos = 1, 
    prompt_model=lm, 
    task_model=lm,
    auto = "medium",
    num_threads=6)

program_optimized_no_demos = optimizer.compile(program, trainset=trainset, requires_permission_to_run = False, max_bootstrapped_demos=0, max_labeled_demos=0)
program_optimized_no_demos.save("program_optimized_no_demos.json")
program_optimized_with_demos = optimizer.compile(program, trainset=trainset, requires_permission_to_run = False)
program_optimized_with_demos.save("program_optimized_with_demos.json")

@isaacbmiller isaacbmiller merged commit 056d54e into main Oct 29, 2025
10 checks passed
hironow added a commit to hironow/dspy that referenced this pull request Oct 30, 2025
commit 056d54e
Author: Isaac Miller <[email protected]>
Date:   Wed Oct 29 17:23:09 2025 +0100

    fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot (stanfordnlp#8909)

    * fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot

    * remove extra logs

    * Remove log

    * Fix merge conflict

    * Remove extra whitespace

commit da69f9d
Author: TomuHirata <[email protected]>
Date:   Wed Oct 29 13:23:34 2025 +0900

    Update anthropic model name (stanfordnlp#8992)

    Signed-off-by: TomuHirata <[email protected]>

commit aaadf05
Author: Chen Qian <[email protected]>
Date:   Tue Oct 28 12:21:55 2025 -0700

    lints (stanfordnlp#8987)

commit e842ba1
Author: eramis73 <[email protected]>
Date:   Tue Oct 28 02:40:34 2025 +0300

    [docs] Add Google-style docstrings for dspy/evaluate/metrics.py (stanfordnlp#8954)

    * docs(metrics): add Google-style docstrings for public metrics

    * docs(metrics): address review feedback (concise openings, mkdocs block examples); revert non-doc changes

    * fixes

    ---------

    Co-authored-by: chenmoneygithub <[email protected]>

commit 6c43880
Author: TomuHirata <[email protected]>
Date:   Tue Oct 28 07:21:06 2025 +0900

    Cache Ollama to speed up CI (stanfordnlp#8972)

    * Cache Ollama to speed up CI

    * fix permission

commit 462baef
Author: Copilot <[email protected]>
Date:   Mon Oct 27 11:57:27 2025 -0700

    Fix TypeError when tracking usage with Anthropic models returning Pydantic objects (stanfordnlp#8978)

    * Initial plan

    * Fix TypeError when merging Anthropic CacheCreation objects in usage tracker

    Co-authored-by: TomeHirata <[email protected]>

    * Enhance _flatten_usage_entry to convert Pydantic models on first add

    Co-authored-by: TomeHirata <[email protected]>

    * Fix potential TypeError when both usage entries are None

    Co-authored-by: TomeHirata <[email protected]>

    * simplify

    * small fix

    * lint

    * robust version handling

    ---------

    Co-authored-by: copilot-swe-agent[bot] <[email protected]>
    Co-authored-by: TomeHirata <[email protected]>
    Co-authored-by: chenmoneygithub <[email protected]>

commit 9b467b5
Author: Noah Ziems <[email protected]>
Date:   Mon Oct 27 13:32:07 2025 -0400

    Add Disable Fallback Option in ChatAdapter (stanfordnlp#8984)

commit bf022c7
Author: Lakshya A Agrawal <[email protected]>
Date:   Sat Oct 25 23:37:42 2025 +0530

    Update gepa[dspy] dependency version to 0.0.18 (stanfordnlp#8969)

    * Update gepa[dspy] dependency version to 0.0.18

    * Update pyproject.toml

    * fix test

    ---------

    Co-authored-by: TomuHirata <[email protected]>
hironow added a commit to hironow/dspy that referenced this pull request Oct 30, 2025
commit 31b96af
Author: Dushmanta <[email protected]>
Date:   Thu Oct 30 13:52:40 2025 +0530

    fix: broken PyPI downloads badge from pepy.tech in README and docs home page (stanfordnlp#8995)

    * fix: update broken pypi download badge in readme

    * fix: update broken pypi download badge in docs home page

commit 056d54e
Author: Isaac Miller <[email protected]>
Date:   Wed Oct 29 17:23:09 2025 +0100

    fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot (stanfordnlp#8909)

    * fix(MIPROv2): zero shot not taking .compile parameters into account before determining if the program was zero shot

    * remove extra logs

    * Remove log

    * Fix merge conflict

    * Remove extra whitespace

commit da69f9d
Author: TomuHirata <[email protected]>
Date:   Wed Oct 29 13:23:34 2025 +0900

    Update anthropic model name (stanfordnlp#8992)

    Signed-off-by: TomuHirata <[email protected]>

commit aaadf05
Author: Chen Qian <[email protected]>
Date:   Tue Oct 28 12:21:55 2025 -0700

    lints (stanfordnlp#8987)

commit e842ba1
Author: eramis73 <[email protected]>
Date:   Tue Oct 28 02:40:34 2025 +0300

    [docs] Add Google-style docstrings for dspy/evaluate/metrics.py (stanfordnlp#8954)

    * docs(metrics): add Google-style docstrings for public metrics

    * docs(metrics): address review feedback (concise openings, mkdocs block examples); revert non-doc changes

    * fixes

    ---------

    Co-authored-by: chenmoneygithub <[email protected]>

commit 6c43880
Author: TomuHirata <[email protected]>
Date:   Tue Oct 28 07:21:06 2025 +0900

    Cache Ollama to speed up CI (stanfordnlp#8972)

    * Cache Ollama to speed up CI

    * fix permission

commit 462baef
Author: Copilot <[email protected]>
Date:   Mon Oct 27 11:57:27 2025 -0700

    Fix TypeError when tracking usage with Anthropic models returning Pydantic objects (stanfordnlp#8978)

    * Initial plan

    * Fix TypeError when merging Anthropic CacheCreation objects in usage tracker

    Co-authored-by: TomeHirata <[email protected]>

    * Enhance _flatten_usage_entry to convert Pydantic models on first add

    Co-authored-by: TomeHirata <[email protected]>

    * Fix potential TypeError when both usage entries are None

    Co-authored-by: TomeHirata <[email protected]>

    * simplify

    * small fix

    * lint

    * robust version handling

    ---------

    Co-authored-by: copilot-swe-agent[bot] <[email protected]>
    Co-authored-by: TomeHirata <[email protected]>
    Co-authored-by: chenmoneygithub <[email protected]>

commit 9b467b5
Author: Noah Ziems <[email protected]>
Date:   Mon Oct 27 13:32:07 2025 -0400

    Add Disable Fallback Option in ChatAdapter (stanfordnlp#8984)

commit bf022c7
Author: Lakshya A Agrawal <[email protected]>
Date:   Sat Oct 25 23:37:42 2025 +0530

    Update gepa[dspy] dependency version to 0.0.18 (stanfordnlp#8969)

    * Update gepa[dspy] dependency version to 0.0.18

    * Update pyproject.toml

    * fix test

    ---------

    Co-authored-by: TomuHirata <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Setting max_bootstrapped_demos and max_labeled_demos to 0 still results in few shot examples being generated for MiproV2

3 participants