Repository navigation
fix: post-codex validation - research guard, memory unification, CLI learning-loop wiring, truthful docs - #11
Merged
Merged
Conversation
…coffee-roaster narrative; fix kernel-size and connector overclaims
A near-zero champion baseline produces unbounded negative improvement values (epoch foxf-fy2025-001 showed -18.85 for adjusted_ebitda_error, dragging weighted_improvement to -5.30). Introduce IMPROVEMENT_CLAMP = 1.0 applied after direction/zero-baseline logic and before weighting so no single metric can dominate the weighted objective. Bounded in [-1-complexity_penalty, 1].
…rics foxf_model.holdout_metrics reimplemented the relative-error formula that pyfpa.analysis.reconcile already computes as variance_pct. Replace the local _relative_error helper with _abs_variance_pct, which delegates to reconcile and takes abs(variance_pct). The formula is identical so committed epoch YAML numbers are unchanged by this fix alone.
Exports PROFILE_HEADINGS from onboarding.py and updates initialize_workspace to seed business-profile.md via render_business_profile with an empty Intake, eliminating the divergent hardcoded heading list in workspace.py. Init-then-onboard now produces no schema flip. Tests: added heading-identity assertion and memory-index completeness assertion to test_memory_workspace.py.
…ot field Adds snapshot: str | None to the Experiment model with a docstring identifying it as the label or relative path of the pyfpa.backtest Snapshot that produced the experiment's scored forecast. Defaults to None; absent experiments round-trip unchanged. Snapshot schema is untouched. Tests: two new round-trip assertions in test_memory_experiments.py.
… provided promote_challenger trusted the stored promotion_eligible flag, which a hand-edited epoch YAML could assert with garbage metrics. Add optional objective: ResearchObjective | None kwarg; when provided, rerun evaluate_challenger against the epoch's stored metrics and checks and raise ValueError if the recomputed result is not promotion_eligible. Without the kwarg the existing behavior (stored flag trusted) is preserved.
… fix Epoch foxf-fy2025-001: adjusted_ebitda_error improvement clamped from -18.85 to -1.0; weighted_improvement 0.056 (was -5.30); objective_gain 0.055 (was -5.30); status proposed (was discarded). Epoch foxf-fy2025-002 numbers unchanged. Artifacts generated from code via run_foxf.py, not hand-edited.
… a catastrophic single-metric regression; Fox epoch 001 discarded for the principled reason
…_command hints, add learning-loop subcommands
…earning-loop commands, align Fox MEMORY.md to generator, repo-wide em dash scrub
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Remediates the findings from a four-stream validation of the agent-native foundation drop (PR #10), against the project goal: a standout, CLI-native toolbelt for an AI agent doing FP&A.
Validation found, this branch fixes
Research loop (reviewed strong: real holdout, enforced human gate, byte-reproducible artifacts):
IMPROVEMENT_CLAMPbounds per-metric score contributions to [-1, +1] so a near-zero-baseline metric cannot swamp the weighted objective (epoch 001 previously showed -18.85 on one metric).max_metric_regressionhard guard inspects the RAW regression so the clamp cannot launder a catastrophic single-metric regression into a passable aggregate. Fox epoch 001 (which multiplies EBITDA error roughly 20x) is now discarded for the principled reason; epoch 002 unchanged. Artifacts regenerated via the code path, byte-reproducible.pyfpa.analysis.reconcile(one scoring seam, no parallel_relative_error).Memory unification (review: two systems bolted together):
initialize_workspacerenders via onboarding'srender_business_profile, so init and onboarding produce identical structure.Experiment.snapshotfield links experiment records to the backtest Snapshot evidence behind them.MEMORY.mdindex now describes the complete vault includingscorecard.mdandlearnings.md; both are seeded at init.CLI (was 1,221 lines with stranded logic; learning loop had no CLI surface):
cli.py1,221 -> 649 lines; classification moved tomemory/inspection.py, doctor validation tomemory/diagnostics.py(validate_workspace);WORKSPACE_DIRSdeduped.correction-record,correction-list,scorecard-render,experiment-list,context-pack,onboarding-render. Doctor now also validates corrections and snapshots.next_commandhints emit the deterministicpython3 -m pyfpa.cliform; README documents the source-checkout fallback.Docs truthfulness:
Verification
run_foxf.py.🤖 Generated with Claude Code