Skip to content

fix: Apply pre-commit formatting to test file #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: Apply pre-commit formatting to test file #31

wants to merge 3 commits into from

Conversation

@siddhantparadox
Copy link
Contributor

What does this PR do?

Adds detailed tracking and real-time display of the instruction-proposal phase during prompt optimization.

  • Telemetry – new InstructionProposalTracker in telemetry.py
  • Integration – hooked into DSPy’s MIPROv2 loop in BasicOptimizationStrategy
  • UX – single-line progress bar (%, avg time) after each candidate; no duplicate lines
  • Tests – new tests/unit/test_instruction_tracker.py (12 cases)
  • Docs – expanded docs/advanced/logging.md with tracker usage
  • Quality – all tests & pre-commit hooks pass

Closes #20

Test Results

Unit & Integration Suite

(.venv) PS D:\Meta\llama-prompt-ops> python -m pytest -v
==================================================================== test session starts =====================================================================
platform win32 -- Python 3.11.4, pytest-8.4.1, pluggy-1.6.0 -- D:\Meta\llama-prompt-ops\.venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: D:\Meta\llama-prompt-ops
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 76 items                                                                                                                                            

tests/integration/test_cli_integration.py::TestCLIIntegration::test_cli_migrate_command PASSED                                                          [  1%] 
tests/integration/test_cli_integration.py::TestCLIIntegration::test_cli_config_loading PASSED                                                           [  2%]
tests/integration/test_cli_integration.py::TestCLIIntegration::test_end_to_end_cli_flow PASSED                                                          [  3%] 
tests/integration/test_core_integration.py::test_dataset_adapter_loading PASSED                                                                         [  5%]
tests/integration/test_core_integration.py::test_dataset_to_model_flow PASSED                                                                           [  6%] 
tests/integration/test_core_integration.py::test_metric_evaluation PASSED                                                                               [  7%] 
tests/integration/test_core_integration.py::test_strategy_execution PASSED                                                                              [  9%] 
tests/integration/test_core_integration.py::test_config_loading PASSED                                                                                  [ 10%] 
tests/integration/test_core_integration.py::test_end_to_end_flow_with_mocks PASSED                                                                      [ 11%] 
tests/integration/test_optimization_integration.py::TestOptimizationIntegration::test_llama_strategy_run PASSED                                         [ 13%] 
tests/integration/test_optimization_integration.py::TestOptimizationIntegration::test_dataset_based_optimization PASSED                                 [ 14%]
tests/integration/test_optimization_integration.py::TestOptimizationIntegration::test_end_to_end_optimization_flow PASSED                               [ 15%] 
tests/integration/test_optimization_integration.py::TestOptimizationIntegration::test_pre_optimization_summary_creation PASSED                          [ 17%]
tests/unit/test_cli.py::test_cli_help PASSED                                                                                                            [ 18%] 
tests/unit/test_cli.py::test_cli_commands_exist PASSED                                                                                                  [ 19%]
tests/unit/test_datasets.py::test_load_data_simple_fields PASSED                                                                                        [ 21%] 
tests/unit/test_datasets.py::test_load_data_nested_fields PASSED                                                                                        [ 22%] 
tests/unit/test_datasets.py::test_transform_functions PASSED                                                                                            [ 23%] 
tests/unit/test_datasets.py::test_load_dataset_default_splits PASSED                                                                                    [ 25%] 
tests/unit/test_datasets.py::test_custom_split_ratios PASSED                                                                                            [ 26%] 
tests/unit/test_datasets.py::test_minimum_records_in_dataset PASSED                                                                                     [ 27%] 
tests/unit/test_instruction_tracker.py::TestCandidateInfo::test_candidate_info_creation PASSED                                                          [ 28%]
tests/unit/test_instruction_tracker.py::TestCandidateInfo::test_candidate_info_duration_calculation PASSED                                              [ 30%] 
tests/unit/test_instruction_tracker.py::TestCandidateInfo::test_candidate_info_duration_none_when_incomplete PASSED                                     [ 31%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_end_candidate PASSED                                                       [ 32%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_get_summary PASSED                                                         [ 34%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_multiple_candidates PASSED                                                 [ 35%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_progress_bar_visualization PASSED                                          [ 36%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_progress_display_format PASSED                                             [ 38%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_start_candidate PASSED                                                     [ 39%]
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_to_json PASSED                                                             [ 40%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_tracker_initialization PASSED                                              [ 42%] 
tests/unit/test_instruction_tracker.py::TestInstructionProposalTracker::test_zero_total_candidates PASSED                                               [ 43%] 
tests/unit/test_logging.py::test_get_logger_singleton PASSED                                                                                            [ 44%] 
tests/unit/test_logging.py::test_phase_timing PASSED                                                                                                    [ 46%]
tests/unit/test_logging.py::test_log_metric PASSED                                                                                                      [ 47%] 
tests/unit/test_logging.py::test_export_json PASSED                                                                                                     [ 48%]
tests/unit/test_metrics.py::test_exact_match_metric[True-True-Answer-Answer-1.0] PASSED                                                                 [ 50%] 
tests/unit/test_metrics.py::test_exact_match_metric[True-True-Answer-answer-0.0] PASSED                                                                 [ 51%] 
tests/unit/test_metrics.py::test_exact_match_metric[False-True-Answer-answer-1.0] PASSED                                                                [ 52%] 
tests/unit/test_metrics.py::test_exact_match_metric[True-True-Answer - Answer-1.0] PASSED                                                               [ 53%] 
tests/unit/test_metrics.py::test_exact_match_metric[True-False-Answer - Answer-0.0] PASSED                                                              [ 55%] 
tests/unit/test_metrics.py::test_exact_match_metric[False-False-Answer-ANSWER -0.0] PASSED                                                              [ 56%] 
tests/unit/test_metrics.py::test_exact_match_metric_with_objects PASSED                                                                                 [ 57%] 
tests/unit/test_metrics.py::test_json_evaluation_metric_exact_match PASSED                                                                              [ 59%] 
tests/unit/test_metrics.py::test_json_evaluation_metric_partial_match PASSED                                                                            [ 60%] 
tests/unit/test_metrics.py::test_json_evaluation_metric_invalid_json PASSED                                                                             [ 61%] 
tests/unit/test_metrics.py::test_standard_json_metric_exists PASSED                                                                                     [ 63%] 
tests/unit/test_metrics.py::test_standard_json_metric_has_required_methods PASSED                                                                       [ 64%] 
tests/unit/test_metrics.py::test_facility_metric PASSED                                                                                                 [ 65%] 
tests/unit/test_metrics.py::test_metric_base_extract_value PASSED                                                                                       [ 67%] 
tests/unit/test_migrator.py::test_load_prompt_from_json PASSED                                                                                          [ 68%] 
tests/unit/test_migrator.py::test_save_optimized_prompt PASSED                                                                                          [ 69%]
tests/unit/test_migrator.py::test_migrator_uses_utils_function PASSED                                                                                   [ 71%] 
tests/unit/test_migrator.py::test_convert_json_to_yaml PASSED                                                                                           [ 72%] 
tests/unit/test_migrator.py::test_json_to_yaml_file PASSED                                                                                              [ 73%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_to_pretty_basic PASSED                                                              [ 75%]
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_to_pretty_with_guidance PASSED                                                      [ 76%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_to_pretty_with_long_guidance PASSED                                                 [ 77%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_to_pretty_with_baseline_score PASSED                                                [ 78%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_to_json PASSED                                                                      [ 80%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_log PASSED                                                                          [ 81%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_mipro_params_json_formatting PASSED                                                 [ 82%] 
tests/unit/test_preopt_summary.py::TestPreOptimizationSummary::test_none_values_handling PASSED                                                         [ 84%] 
tests/unit/test_prompt_processors.py::TestPromptProcessor::test_instruction_preference PASSED                                                           [ 85%] 
tests/unit/test_prompt_processors.py::TestPromptProcessor::test_llama_formatting PASSED                                                                 [ 86%] 
tests/unit/test_prompt_processors.py::TestPromptProcessor::test_processor_chain PASSED                                                                  [ 88%] 
tests/unit/test_prompt_processors.py::TestLlamaProcessingChain::test_chain_with_disabled_formatting PASSED                                              [ 89%] 
tests/unit/test_prompt_processors.py::TestLlamaProcessingChain::test_chain_with_examples PASSED                                                         [ 90%] 
tests/unit/test_prompt_processors.py::TestLlamaProcessingChain::test_processing_chain PASSED                                                            [ 92%] 
tests/unit/test_summary_utils.py::TestSummaryUtils::test_create_pre_optimization_summary_basic PASSED                                                   [ 93%] 
tests/unit/test_summary_utils.py::TestSummaryUtils::test_create_pre_optimization_summary_with_guidance PASSED                                           [ 94%] 
tests/unit/test_summary_utils.py::TestSummaryUtils::test_create_pre_optimization_summary_with_baseline PASSED                                           [ 96%]
tests/unit/test_summary_utils.py::TestSummaryUtils::test_create_pre_optimization_summary_handles_missing_attributes PASSED                              [ 97%] 
tests/unit/test_summary_utils.py::TestSummaryUtils::test_create_and_display_summary PASSED                                                              [ 98%] 
tests/unit/test_summary_utils.py::TestSummaryUtils::test_create_and_display_summary_handles_errors PASSED                                               [100%] 

====================================================================== warnings summary ====================================================================== 
.venv\Lib\site-packages\pydantic\_internal\_config.py:323
  D:\Meta\llama-prompt-ops\.venv\Lib\site-packages\pydantic\_internal\_config.py:323: PydanticDeprecatedSince20: Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/    
    warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================== 76 passed, 1 warning in 3.74s ================================================================

All 76 tests pass across unit, integration, and functional suites.

Pre-commit Hooks

$ pre-commit run --all-files
trim trailing whitespace.........................................Passed
fix end of files.................................................Passed
check yaml......................................................Passed
check for added large files.....................................Passed
black...........................................................Passed
isort...........................................................Passed

No formatting issues; repository conforms to style guidelines.

Manual CLI Verification

$ llama-prompt-ops migrate --log-level INFO
…
⏳ Proposing instructions: |██████░░░░░░░░░░░░░░░░░░░░░░░░| 1/5 (20.0%) | avg: 1.24s
⏳ Proposing instructions: |████████████░░░░░░░░░░░░░░░░░░| 2/5 (40.0%) | avg: 1.18s
⏳ Proposing instructions: |██████████████████░░░░░░░░░░░░| 3/5 (60.0%) | avg: 1.15s
⏳ Proposing instructions: |████████████████████████░░░░░░| 4/5 (80.0%) | avg: 1.12s
⏳ Proposing instructions: |██████████████████████████████| 5/5 (100.0%) | avg: 1.10s

Progress appears exactly once per candidate; no duplicate 0 % line.

How to Re-run

  1. Install dev deps

    pip install -e ".[dev]"
pre-commit install

  2. Run tests

    python -m pytest -v

  3. Run pre-commit

    pre-commit run --all-files

  4. Run a quick migration

    llama-prompt-ops migrate --config my-project/config.yaml --log-level INFO

You should see the real-time progress bar during the “Propose instructions” step and the test/pre-commit outputs shown above.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 7, 2025
@heyjustinai
Copy link
Member

heyjustinai commented Jul 7, 2025 •
edited
Loading

hey @siddhantparadox! thanks for pushing this, just ran a test and it looks like the progress bar isn't updated real time but it's only printed when step 2 is done, so users still experience the same 'black box' waiting period during the actual proposal generation

Output:

2025/07/07 14:17:48 INFO dspy.teleprompt.mipro_optimizer_v2: 
Proposing instructions...

2025-07-07 14:17:48,954 - root - INFO - Starting custom_propose_instructions with enhanced error handling and tracking
2025-07-07 14:17:48,954 - root - INFO - Calling original propose_instructions_for_program
2025-07-07 14:19:04,060 | INFO    | ⏳ Proposing instructions: |██████░░░░░░░░░░░░░░░░░░░░░░░░| 1/5 (20.0%) | avg: 0.01s
2025-07-07 14:19:04,071 | INFO    | ⏳ Proposing instructions: |████████████░░░░░░░░░░░░░░░░░░| 2/5 (40.0%) | avg: 0.01s
2025-07-07 14:19:04,085 | INFO    | ⏳ Proposing instructions: |██████████████████░░░░░░░░░░░░| 3/5 (60.0%) | avg: 0.01s
2025-07-07 14:19:04,097 | INFO    | ⏳ Proposing instructions: |████████████████████████░░░░░░| 4/5 (80.0%) | avg: 0.01s
2025-07-07 14:19:04,110 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 5/5 (100.0%) | avg: 0.01s
2025-07-07 14:19:04,110 - root - INFO - Instruction proposer returned result with keys: dict_keys([0])
2025/07/07 14:19:04 INFO dspy.teleprompt.mipro_optimizer_v2: Proposed Instructions for Predictor 0:

2025/07/07 14:19:04 INFO dspy.teleprompt.mipro_optimizer_v2: 0: You are a helpful assistant. Extract and return a json with the following keys and values:
- "urgency" as one of `high`, `medium`, `low`
- "sentiment" as one of `negative`, `neutral`, `positive`
- "categories" Create a dictionary with categories as keys and boolean values (True/False), where the value indicates whether the category is one of the best matching support category tags from: `emergency_repair_services`, `routine_maintenance_requests`, `quality_and_safety_concerns`, `specialized_cleaning_services`, `general_inquiries`, `sustainability_and_environmental_practices`, `training_and_support_requests`, `cleaning_services_scheduling`, `customer_feedback_and_complaints`, `facility_management_issues`
Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.

...

2025/07/07 14:19:04 INFO dspy.teleprompt.mipro_optimizer_v2: 

2025/07/07 14:19:04 INFO dspy.teleprompt.mipro_optimizer_v2: ==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==
2025/07/07 14:19:04 INFO dspy.teleprompt.mipro_optimizer_v2: We will evaluate the program over a series of trials with different combinations of instructions and few-shot examples to find the optimal combination using Bayesian Optimization.

siddhantparadox and others added 2 commits July 22, 2025 21:14
@siddhantparadox
Merge branch 'meta-llama:main' into feat/instruction-proposal-tracker
2ec8d4f
@siddhantparadox
feat: Add real-time progress tracking for DSPy instruction proposals
80f3c0c 
- Implement dynamic LM wrapper that inherits from the wrapped LM's class
- Add real-time progress bar showing instruction generation status
- Cap progress display at 100% to handle extra LM calls gracefully
- Update telemetry system to show "+X extra" for calls beyond expected
- Add comprehensive unit tests for the tracking functionality
- Fix .gitignore to properly exclude my-project/ directory

This addresses the "black box" experience during DSPy's MIPROv2 optimization
by providing immediate visual feedback as each instruction is generated.
@siddhantparadox
Copy link
Contributor Author

siddhantparadox commented Jul 29, 2025 •
edited
Loading

PR Title

Add real-time progress tracking for DSPy instruction proposals

PR Description

This PR implements real-time progress tracking during DSPy's instruction proposal phase, addressing the issue where users had to wait 1-2 minutes without any feedback during optimization.

Problem:
Previously, during the MIPROv2 optimization process, users experienced a "black box" waiting period with no indication of progress while DSPy generated instruction candidates.

Solution:

  • Created a dynamic wrapper (create_realtime_lm_wrapper) that inherits from the DSPy LM's class to maintain type compatibility
  • Implemented RealtimeProposalInterceptor that patches DSPy's GroundedProposer to enable tracking
  • Added real-time progress bar that updates as each instruction is generated
  • Fixed UI issue where progress could show >100% by capping the visual bar and showing extra calls as "+X extra"

Why does it show 15 calls instead of 10?
DSPy's MIPROv2 optimizer doesn't make exactly one LM call per instruction candidate. The additional calls are for:

  1. Dataset Summary Generation - DSPy analyzes your dataset to understand patterns
  2. Program Code Summary - It examines the structure of your prompt program
  3. Prompting Tip Selection - DSPy randomly selects and applies prompting strategies
  4. Quality Filtering - Some candidates may be regenerated or filtered out
  5. Internal Processing - Additional analysis for optimization

This is why you might see 15 LM calls for 10 requested candidates, but only 4 final instructions in the output. DSPy is doing sophisticated optimization work beyond simple generation.

Technical Details:

  • The wrapper intercepts all LM calls during the proposal phase
  • Progress is displayed in real-time: ⏳ Proposing instructions: |██████░░░░| 6/10 (60.0%) | avg: 6.83s
  • When DSPy makes more calls than expected (e.g., 15 instead of 10), the display shows: 10/10 (+5 extra) (100.0%)
  • The tracking accurately reflects what DSPy is actually doing, providing full transparency

Testing:

  • Added comprehensive unit tests in tests/unit/test_realtime_tracking.py
  • All tests pass with proper mocking of DSPy components
  • Verified real-time tracking works in actual optimization runs

Additional Changes:

  • Fixed .gitignore to properly exclude my-project/ directory (was incorrectly using underscore)

This enhancement significantly improves the user experience by providing immediate feedback during the optimization process and helps users understand that the extra LM calls are a normal part of DSPy's sophisticated optimization strategy.

Sample output

(.venv) PS D:\Meta\llama-prompt-ops\my-project> llama-prompt-ops migrate --log-level INFO
Loaded environment variables from .env
Loaded configuration from config.yaml
2025-07-28 22:32:18,488 | INFO    |  Using model with DSPy: openrouter/meta-llama/llama-3.3-70b-instruct
Using the same model for task and proposer: openrouter/meta-llama/llama-3.3-70b-instruct
Using metric: FacilityMetric
Resolved relative dataset path to: D:\Meta\llama-prompt-ops\my-project\data/dataset.json
Using dataset adapter: ConfigurableJSONAdapter
2025-07-28 22:32:18,507 - root - WARNING - Model '' does not appear to be a Llama model. This library is optimized for Llama models.
Auto-detected BasicOptimizationStrategy for model:
2025-07-28 22:32:18,508 - root - WARNING - Model '<llama_prompt_ops.core.model.DSPyModelAdapter object at 0x0000028E1B835810>' does not appear to be a Llama model. This library is optimized for Llama models and may not work as expected.
2025-07-28 22:32:18,511 - root - INFO - Loaded 200 examples from D:\Meta\llama-prompt-ops\my-project\data\dataset.json
2025-07-28 22:32:18,512 - root - INFO - Created dataset splits:
2025-07-28 22:32:18,512 - root - INFO -   - Training:   50 examples (25.0% of total)
2025-07-28 22:32:18,512 - root - INFO -   - Validation: 50 examples (25.0% of total)
2025-07-28 22:32:18,512 - root - INFO -   - Testing:    100 examples (50.0% of total)
Loaded prompt from file: D:\Meta\llama-prompt-ops\my-project\prompts/prompt.txt
Using 'system_prompt' from config
Using config filename as output prefix: config
Starting prompt optimization...
2025-07-28 22:32:18,516 | INFO    | Applying BasicOptimizationStrategy to optimize prompt
2025-07-28 22:32:18,516 | INFO    | Training set size: 50
2025-07-28 22:32:18,516 | INFO    | Validation set size: 50
2025-07-28 22:32:18,516 | INFO    | Test set size: 100

Computing baseline score on 100 test examples using 18 threads...
Average Metric: 78.67 / 100 (78.7%): 100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:28<00:00,  3.52it/s]
2025/07/28 22:32:46 INFO dspy.evaluate.evaluate: Average Metric: 78.66666666666671 / 100 (78.7%)
✅ Baseline Score: 78.670 in 33.94s

2025-07-28 22:32:52,456 | INFO    | === Pre-Optimization Summary ===
    Task Model       : openrouter/meta-llama/llama-3.3-70b-instruct
    Proposer Model   : openrouter/meta-llama/llama-3.3-70b-instruct
    Metric           : <llama_prompt_ops.core.metrics.FacilityMetric object at 0x0000028E1B835D10>
    Train / Val size : 50 / 50
    MIPRO Params     : {"auto_user":"basic","auto_dspy":"light","max_labeled_demos":5,"max_bootstrapped_demos":4,"num_candidates":10,"num_threads":18,"init_temperature":0.5,"seed":9}
    Baseline score   : 78.6700
2025-07-28 22:32:52,457 - root - INFO - Optimization strategy using 5 labeled demos, 4 bootstrapped demos with 18 threads
2025-07-28 22:32:52,457 - root - INFO - Compiling program with 50 training examples, 50 validation examples, and 100 test examples
2025-07-28 22:32:52,458 - root - INFO - Enabled real-time instruction proposal tracking for 10 candidates
2025-07-28 22:32:52,460 - root - WARNING - Debug module not available, continuing without enhanced debugging
2025-07-28 22:32:52,460 - root - INFO - Starting DSPy optimization with enhanced debugging
2025-07-28 22:32:52,460 - root - INFO - Program type: <class 'dspy.predict.predict.Predict'>
2025-07-28 22:32:52,460 - root - INFO - Trainset size: 50
2025-07-28 22:32:52,460 - root - INFO - Valset size: 50
2025-07-28 22:32:52,460 - root - INFO - First trainset example structure: <class 'dspy.primitives.example.Example'>
2025-07-28 22:32:52,461 - root - WARNING - Example missing required 'inputs' or 'outputs' attributes
2025-07-28 22:32:52,461 - root - WARNING - Example attributes: ['__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_demos', '_input_keys', '_output_keys', '_store', 'copy', 'get', 'inputs', 'items', 'keys', 'labels', 'toDict', 'values', 'with_inputs', 'without']
2025-07-28 22:32:52,461 - root - INFO - Calling optimizer.compile
2025/07/28 22:32:52 INFO dspy.teleprompt.mipro_optimizer_v2:
RUNNING WITH THE FOLLOWING LIGHT AUTO RUN SETTINGS:
num_trials: 7
minibatch: False
num_candidates: 5
valset size: 50

2025/07/28 22:32:52 INFO dspy.teleprompt.mipro_optimizer_v2:
==> STEP 1: BOOTSTRAP FEWSHOT EXAMPLES <==
2025/07/28 22:32:52 INFO dspy.teleprompt.mipro_optimizer_v2: These will be used as few-shot example candidates for our program and for creating instructions.

2025/07/28 22:32:52 INFO dspy.teleprompt.mipro_optimizer_v2: Bootstrapping N=5 sets of demonstrations...
Bootstrapping set 1/5
Bootstrapping set 2/5
Bootstrapping set 3/5
  8%|█████████▍                                                                                                            | 4/50 [00:25<04:57,  6.47s/it] 
Bootstrapped 4 full traces after 4 examples for up to 1 rounds, amounting to 4 attempts.
Bootstrapping set 4/5
  6%|███████                                                                                                               | 3/50 [00:13<03:28,  4.44s/it] 
Bootstrapped 3 full traces after 3 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 5/5
  4%|████▋                                                                                                                 | 2/50 [00:08<03:18,  4.14s/it] 
Bootstrapped 2 full traces after 2 examples for up to 1 rounds, amounting to 2 attempts.
2025/07/28 22:33:39 INFO dspy.teleprompt.mipro_optimizer_v2:
==> STEP 2: PROPOSE INSTRUCTION CANDIDATES <==
2025/07/28 22:33:39 INFO dspy.teleprompt.mipro_optimizer_v2: We will use the few-shot examples from the previous step, a generated dataset summary, a summary of the program code, and a randomly selected prompting tip to propose instructions.
2025/07/28 22:35:48 INFO dspy.teleprompt.mipro_optimizer_v2: 
Proposing instructions...

2025-07-28 22:35:48,653 | INFO    | Enabled real-time instruction proposal tracking
2025-07-28 22:35:55,405 | INFO    | ⏳ Proposing instructions: |███░░░░░░░░░░░░░░░░░░░░░░░░░░░| 1/10 (10.0%) | avg: 6.75s
2025-07-28 22:36:00,675 | INFO    | ⏳ Proposing instructions: |██████░░░░░░░░░░░░░░░░░░░░░░░░| 2/10 (20.0%) | avg: 6.01s
2025-07-28 22:36:09,868 | INFO    | ⏳ Proposing instructions: |█████████░░░░░░░░░░░░░░░░░░░░░| 3/10 (30.0%) | avg: 7.07s
2025-07-28 22:36:15,298 | INFO    | ⏳ Proposing instructions: |████████████░░░░░░░░░░░░░░░░░░| 4/10 (40.0%) | avg: 6.66s
2025-07-28 22:36:20,914 | INFO    | ⏳ Proposing instructions: |███████████████░░░░░░░░░░░░░░░| 5/10 (50.0%) | avg: 6.45s
2025-07-28 22:36:28,271 | INFO    | ⏳ Proposing instructions: |██████████████████░░░░░░░░░░░░| 6/10 (60.0%) | avg: 6.60s
2025-07-28 22:36:35,841 | INFO    | ⏳ Proposing instructions: |█████████████████████░░░░░░░░░| 7/10 (70.0%) | avg: 6.74s
2025-07-28 22:36:40,307 | INFO    | ⏳ Proposing instructions: |████████████████████████░░░░░░| 8/10 (80.0%) | avg: 6.45s
2025-07-28 22:36:45,463 | INFO    | ⏳ Proposing instructions: |███████████████████████████░░░| 9/10 (90.0%) | avg: 6.31s
2025-07-28 22:36:55,293 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 10/10 (100.0%) | avg: 6.66s
2025-07-28 22:36:59,402 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 10/10 (+1 extra) (100.0%) | avg: 6.43s
2025-07-28 22:37:07,501 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 10/10 (+2 extra) (100.0%) | avg: 6.57s
2025-07-28 22:37:14,562 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 10/10 (+3 extra) (100.0%) | avg: 6.61s
2025-07-28 22:37:22,019 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 10/10 (+4 extra) (100.0%) | avg: 6.67s
2025-07-28 22:37:32,951 | INFO    | ⏳ Proposing instructions: |██████████████████████████████| 10/10 (+5 extra) (100.0%) | avg: 6.95s
2025-07-28 22:37:32,951 | INFO    | Instruction proposal complete: 15/10 in 104.26s
2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: Proposed Instructions for Predictor 0:

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: 0: You are a helpful assistant. Extract and return a json with the following keys and values: 
- "urgency" as one of `high`, `medium`, `low`
- "sentiment" as one of `negative`, `neutral`, `positive`
- "categories" Create a dictionary with categories as keys and boolean values (True/False), where the value indicates whether the category is one of the best matching support category tags from: `emergency_repair_services`, `routine_maintenance_requests`, `quality_and_safety_concerns`, `specialized_cleaning_services`, `general_inquiries`, `sustainability_and_environmental_practices`, `training_and_support_requests`, `cleaning_services_scheduling`, `customer_feedback_and_complaints`, `facility_management_issues`
Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: 1: You are a customer support specialist for ProCare Facility Solutions, tasked with analyzing customer inquiries and providing timely, personalized responses. Extract and return a json with the following keys and values: "urgency" as one of `high`, `medium`, `low`, "sentiment" as one of `negative`, `neutral`, `positive`, and "categories" as a dictionary with categories as keys and boolean values (True/False), where the value indicates whether the category is one of the best matching support category tags from: `emergency_repair_services`, `routine_maintenance_requests`, `quality_and_safety_concerns`, `specialized_cleaning_services`, `general_inquiries`, `sustainability_and_environmental_practices`, `training_and_support_requests`, `cleaning_services_scheduling`, `customer_feedback_and_complaints`, `facility_management_issues`. Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above.

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: 2: Analyze the input question or inquiry from a customer regarding facility management and maintenance services, and generate a json response with the following keys: "urgency" as one of high, medium, low, "sentiment" as one of negative, neutral, positive, and "categories" as a dictionary with relevant category tags and boolean values indicating whether each category is a best match, ensuring the response is clear, concise, and directly readable as a json string.

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: 3: Given a customer inquiry related to facility management services, analyze the message to determine the urgency level as 'high', 'medium', or 'low', and the sentiment as 'negative', 'neutral', or 'positive'. Then, categorize the inquiry into one or more of the following categories: emergency_repair_services, routine_maintenance_requests, quality_and_safety_concerns, specialized_cleaning_services, general_inquiries, sustainability_and_environmental_practices, training_and_support_requests, cleaning_services_scheduling, customer_feedback_and_complaints, facility_management_issues, and return the result as a json string with the keys 'urgency', 'sentiment', and 'categories', where 'categories' is a dictionary with the category names as keys and boolean values indicating whether the category is a best match.

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: 4: You are a highly advanced language model designed to analyze customer inquiries related to facility management and maintenance services. Your task is to categorize the input question or inquiry, assess its sentiment, and determine its urgency. To accomplish this, you will utilize natural language processing (NLP) and machine learning algorithms to understand the context, intent, and content of the input question. You will then output a structured json string containing the following keys and values: "urgency" as one of `high`, `medium`, `low`, "sentiment" as one of `negative`, `neutral`, `positive`, and "categories" as a dictionary with categories as keys and boolean values (True/False), where the value indicates whether the category is one of the best matching support category tags from: `emergency_repair_services`, `routine_maintenance_requests`, `quality_and_safety_concerns`, `specialized_cleaning_services`, `general_inquiries`, `sustainability_and_environmental_practices`, `training_and_support_requests`, `cleaning_services_scheduling`, `customer_feedback_and_complaints`, `facility_management_issues`. Ensure your output is a valid json string that can be read directly, without newlines or unnecessary whitespaces.

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2:

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: ==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==
2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: We will evaluate the program over a series of trials with different combinations of instructions and few-shot examples to find the optimal combination using Bayesian Optimization.

2025/07/28 22:37:32 INFO dspy.teleprompt.mipro_optimizer_v2: == Trial 1 / 7 - Full Evaluation of Default Program ==
Average Metric: 38.73 / 50 (77.5%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:17<00:00,  2.93it/s]
2025/07/28 22:37:50 INFO dspy.evaluate.evaluate: Average Metric: 38.73333333333333 / 50 (77.5%)
2025/07/28 22:37:50 INFO dspy.teleprompt.mipro_optimizer_v2: Default program score: 77.47

D:\Meta\llama-prompt-ops\.venv\Lib\site-packages\optuna\_experimental.py:31: ExperimentalWarning: Argument ``multivariate`` is an experimental feature. The interface can change in the future.
  warnings.warn(
2025/07/28 22:37:50 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 2 / 7 =====
Average Metric: 42.27 / 50 (84.5%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:15<00:00,  3.18it/s]
2025/07/28 22:38:05 INFO dspy.evaluate.evaluate: Average Metric: 42.266666666666644 / 50 (84.5%)
2025/07/28 22:38:05 INFO dspy.teleprompt.mipro_optimizer_v2: Best full score so far! Score: 84.53
2025/07/28 22:38:05 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 84.53 with parameters ['Predictor 0: Instruction 1', 'Predictor 0: Few-Shot Set 1'].   
2025/07/28 22:38:05 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [77.47, 84.53]
2025/07/28 22:38:05 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 84.53
2025/07/28 22:38:05 INFO dspy.teleprompt.mipro_optimizer_v2: =======================


2025/07/28 22:38:05 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 3 / 7 =====
Average Metric: 41.10 / 50 (82.2%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:15<00:00,  3.16it/s]
2025/07/28 22:38:21 INFO dspy.evaluate.evaluate: Average Metric: 41.09999999999998 / 50 (82.2%)
2025/07/28 22:38:21 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 82.2 with parameters ['Predictor 0: Instruction 2', 'Predictor 0: Few-Shot Set 1'].    
2025/07/28 22:38:21 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [77.47, 84.53, 82.2]
2025/07/28 22:38:21 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 84.53
2025/07/28 22:38:21 INFO dspy.teleprompt.mipro_optimizer_v2: =======================


2025/07/28 22:38:21 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 4 / 7 =====
Average Metric: 42.07 / 50 (84.1%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:15<00:00,  3.16it/s]
2025/07/28 22:38:37 INFO dspy.evaluate.evaluate: Average Metric: 42.06666666666664 / 50 (84.1%)
2025/07/28 22:38:37 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 84.13 with parameters ['Predictor 0: Instruction 4', 'Predictor 0: Few-Shot Set 1'].   
2025/07/28 22:38:37 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [77.47, 84.53, 82.2, 84.13]
2025/07/28 22:38:37 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 84.53
2025/07/28 22:38:37 INFO dspy.teleprompt.mipro_optimizer_v2: =======================


2025/07/28 22:38:37 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 5 / 7 =====
Average Metric: 40.67 / 50 (81.3%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:23<00:00,  2.10it/s]
2025/07/28 22:39:01 INFO dspy.evaluate.evaluate: Average Metric: 40.66666666666665 / 50 (81.3%)
2025/07/28 22:39:01 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 81.33 with parameters ['Predictor 0: Instruction 2', 'Predictor 0: Few-Shot Set 1'].   
2025/07/28 22:39:01 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [77.47, 84.53, 82.2, 84.13, 81.33]
2025/07/28 22:39:01 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 84.53
2025/07/28 22:39:01 INFO dspy.teleprompt.mipro_optimizer_v2: =======================


2025/07/28 22:39:01 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 6 / 7 =====
Average Metric: 39.77 / 50 (79.5%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:18<00:00,  2.65it/s]
2025/07/28 22:39:20 INFO dspy.evaluate.evaluate: Average Metric: 39.76666666666665 / 50 (79.5%)
2025/07/28 22:39:20 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 79.53 with parameters ['Predictor 0: Instruction 4', 'Predictor 0: Few-Shot Set 3'].   
2025/07/28 22:39:20 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [77.47, 84.53, 82.2, 84.13, 81.33, 79.53]
2025/07/28 22:39:20 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 84.53
2025/07/28 22:39:20 INFO dspy.teleprompt.mipro_optimizer_v2: =======================


2025/07/28 22:39:20 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 7 / 7 =====
Average Metric: 41.93 / 50 (83.9%): 100%|█████████████████████████████████████████████████████████████████████████████████| 50/50 [00:23<00:00,  2.13it/s]
2025/07/28 22:39:43 INFO dspy.evaluate.evaluate: Average Metric: 41.933333333333316 / 50 (83.9%)
2025/07/28 22:39:43 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 83.87 with parameters ['Predictor 0: Instruction 0', 'Predictor 0: Few-Shot Set 1'].   
2025/07/28 22:39:43 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [77.47, 84.53, 82.2, 84.13, 81.33, 79.53, 83.87]
2025/07/28 22:39:43 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 84.53
2025/07/28 22:39:43 INFO dspy.teleprompt.mipro_optimizer_v2: =======================


2025/07/28 22:39:43 INFO dspy.teleprompt.mipro_optimizer_v2: Returning best identified program with score 84.53!
2025-07-28 22:39:43,854 - root - INFO - Optimizer.compile completed successfully
2025-07-28 22:39:43,854 - root - INFO - Restored original propose_instructions_for_program method
2025-07-28 22:39:43,854 - root - INFO - Optimized program type: <class 'dspy.predict.predict.Predict'>
2025-07-28 22:39:43,854 - root - INFO - Optimized program attributes: ['__call__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slotnames__', '__str__', '__subclasshook__', '__weakref__', '_base_init', 'batch', 'callbacks', 'candidate_programs', 'config', 'deepcopy', 'demos', 'dump_state', 'forward', 'get_config', 'get_lm', 'lm', 'load', 'load_state', 'map_named_predictors', 'mb_candidate_programs', 'model_family', 'named_parameters', 'named_predictors', 'named_sub_modules', 'parameters', 'predictors', 'prompt_model_total_calls', 'reset', 'reset_copy', 'save', 'score', 'set_lm', 'signature', 'stage', 'total_calls', 'traces', 'train', 'trial_logs', 'update_config']
2025-07-28 22:39:43,854 | INFO    | [Running optimization strategy] completed in 445.34s
2025-07-28 22:39:43,854 | INFO    | Optimized prompt:
2025-07-28 22:39:43,855 | INFO    | ----------------------------------------
2025-07-28 22:39:43,855 | INFO    | You are a customer support specialist for ProCare Facility Solutions, tasked with analyzing customer inquiries and providing timely, personalized responses. Extract and return a json with the following keys and values: "urgency" as one of `high`, `medium`, `low`, "sentiment" as one of `negative`, `neutral`, `positive`, and "categories" as a dictionary with categories as kcan be read directly and only contain the keys mentioned in the list above.
2025-07-28 22:39:43,855 | INFO    | ----------------------------------------
can be read directly and only contain the keys mentioned in the list above.
2025-07-28 22:39:43,855 | INFO    | ----------------------------------------
2025-07-28 22:39:43,856 | INFO    | Saved optimized prompt to results\config_20250728_223218.json
2025-07-28 22:39:43,873 | INFO    | Saved YAML prompt to results\config_20250728_223218.yaml
2025-07-28 22:39:43,873 | INFO    | [Saving optimized prompt] completed in 0.02s
Results saved to: D:\Meta\llama-prompt-ops\my-project\results\config_20250728_223218.json
Results also saved to: D:\Meta\llama-prompt-ops\my-project\results\config_20250728_223218.yaml
Results saved to: D:\Meta\llama-prompt-ops\my-project\results\config_20250728_223218.json
Results also saved to: D:\Meta\llama-prompt-ops\my-project\results\config_20250728_223218.yaml

Optimized prompt:
================================================================================
You are a customer support specialist for ProCare Facility Solutions, tasked with analyzing customer inquiries and providing timely, personalized responses. Extract and return a json with the following keys and values: "urgency" as one of `high`, `medium`, `low`, "sentiment" as one of `negative`, `neutral`, `positive`, and "categories" as a dictionary with categories as keys and boolean values (True/False), where the value indicates whether the category is one of the best matching support category tags from: `emergency_repair_services`, `routine_maintenance_requests`, `quality_and_safety_concerns`, `specialized_cleaning_services`, `general_inquiries`, `sustainability_and_environmental_practices`, `training_and_support_requests`, `cleaning_services_scheduling`, `customer_feedback_and_complaints`, `facility_management_issues`. Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above.
================================================================================
2025-07-28 22:39:43,879 | INFO    | === Timings summary ===
2025-07-28 22:39:43,880 | INFO    | Running optimization strategy 445.34s
2025-07-28 22:39:43,880 | INFO    | Saving optimized prompt     0.02s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement Instruction Proposal Phase Tracking

3 participants

@siddhantparadox @heyjustinai @facebook-github-bot