diff --git a/README.md b/README.md index eaf8516..2839003 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CI](https://github.com/JeffBrines/openfpa/actions/workflows/ci.yml/badge.svg)](https://github.com/JeffBrines/openfpa/actions/workflows/ci.yml) -**An AI-native FP&A toolkit.** Point an AI coding agent (Claude Code, Claude Cowork, Codex) at your numbers and it stands up a CFO-grade financial model — a 12-month P&L and cash-flow forecast, a 13-week cash-runway, and a board-ready briefing — in minutes. +**An AI-native FP&A toolkit.** Point an AI coding agent (Claude Code, Claude Cowork, Codex) at your numbers and it builds a working financial model — a 12-month P&L and cash-flow forecast, a 13-week cash runway, and a board-ready briefing. `openfpa` is a deliberately **lean Python forecast engine** plus a **progressive Claude skillset** that encodes the methodology and judgment of a real finance team. The engine is small on purpose: it's the substrate an AI extends per-business, not an off-the-shelf app you configure by hand. @@ -19,7 +19,7 @@ Those tools hand you connectors and a modeling layer, then leave the thinking to Two things underneath, rarely combined: - **Hundreds of hours of real FP&A engineering** — methodology distilled from production CFO work (a trucking fleet, a bicycle company, and more), not textbook finance. -- **A self-improving loop, inspired by [Karpathy's AutoResearch](https://github.com/karpathy/autoresearch)** — it scores itself against *your* actuals and gets measurably better over time. AutoResearch improves against validation loss; openfpa improves against reconciliation error on your own books. +- **A self-improving loop, inspired by [Karpathy's AutoResearch](https://github.com/karpathy/autoresearch)** — it scores itself against *your* actuals, aiming to get better over time. AutoResearch optimizes against validation loss; openfpa optimizes against reconciliation error on your own books. Self-hosted, auditable, yours — what it learns lives as plain files in your repo, not someone else's cloud. An open-source experiment from [Guiderail](https://guiderail.example); we'd love your help making it the FP&A tool we all wish existed. @@ -34,7 +34,18 @@ Fair question — Claude *can* write financial code from scratch. But every run - **Reproducible & auditable.** Config-driven, every figure source-traced to a filing, re-runnable — not a chat transcript you can't reproduce. - **Self-extension *with guardrails*.** The agent re-tools a *known, tested* structure per business (it generated a bespoke `segment-rollup` skill for Fox's segments) instead of emitting throwaway scripts. Template-grade rigor **and** bespoke-grade fit. -**Bare Claude is a brilliant analyst with a blank spreadsheet. openfpa is the firm's tested model engine, the encoded house methodology, and the review checklist — the rails the agent drives on, and the gauges that catch it when it's wrong.** +The short version: bare Claude is a capable analyst with a blank spreadsheet. openfpa adds the tested model engine, the encoded methodology, and a review checklist — rails to drive on, and gauges that catch the mistakes. + +--- + +## How it compounds + +openfpa is designed to get better the more you use it — two loops, both keeping your data in your own repo: + +- **Per client (Loop A).** Every close, it scores its last forecast against your actuals and proposes tweaks for you to accept or reject. Human corrections feed the same memory: the fixes you catch by eye, captured durably as plain markdown you own. +- **Across your book (Loop B).** For a fractional CFO with many clients, it looks for patterns that *generalize* — validated by leave-one-out cross-client backtesting (a pattern has to hold up on the clients it wasn't learned from) — and, with your sign-off, saves them to a local library that seeds the next client. + +It borrows [Karpathy's AutoResearch](https://github.com/karpathy/autoresearch) idea: a cheap, objective fitness metric — *reconciliation error against your own books* — for the loops to optimize against, at the client level and across your book. Nothing phones home; you ratify every change. --- @@ -88,13 +99,28 @@ It self-extends, too: Fox reports segment **Adjusted EBITDA** (ASU 2023-07), not --- +## Bring your numbers — any way you have them + +openfpa is **not married to a connector.** Everything it ingests normalizes to one shape — `{account: amount}` — so the engine never cares where the numbers came from. And where no built-in path exists, **the AI writes the ingestion**: the Fox example's [`pull_edgar.py`](examples/foxfactory/pull_edgar.py) is exactly that — a SEC-EDGAR adapter the agent built from scratch. + +So you connect however your numbers actually live: + +- **Public filings** — a 10-K / 10-Q (the Fox Factory example). +- **Your accounting system, live** — QuickBooks or NetSuite, via their **MCP servers** (the MCP server owns the auth — openfpa never touches your credentials) or their APIs. +- **D2C ops** — Shopify. +- **Or just the spreadsheets on your laptop** — P&L, balance sheet, AR/AP aging, inventory. CSV/Excel in, model out. (`read_pl_csv` reads any `Account, Amount` export; richer tables like aged AR or item-level inventory get parsed to what the model needs — DSO, DIO, DPO.) + +No data team, no implementation project, no "is my connector supported?" The agent meets your data where it is and builds the bridge. That's the whole point of a substrate an AI extends per-business. + +--- + ## What it does ``` config (YAML) ─▶ revenue ─▶ cogs ─▶ opex ─▶ working capital ─▶ debt ─▶ cashflow │ - trial balance / P&L CSV ─▶ (ingestion) ▼ - NetSuite · QuickBooks · Shopify ─▶ (adapters) 12-month P&L + cash flow + spreadsheets (P&L/BS/AR/AP/inventory) ─▶ (ingestion) ▼ + QuickBooks·NetSuite (MCP/API) · 10-K · Shopify ─▶ (adapters) 12-month P&L + cash flow │ scheduled weekly flows ─▶ 13-week direct-method cash ─▶ runway ▼ board-ready briefing (md / xlsx) @@ -102,8 +128,8 @@ config (YAML) ─▶ revenue ─▶ cogs ─▶ opex ─▶ working capital - **Monthly forecast engine** — config-driven (`EntityConfig`): revenue with seasonality + growth, channel-level COGS, fixed/variable OpEx, AR/AP/inventory working capital, multi-instrument debt, and an indirect-method cash flow with NOL-aware tax. - **13-week cash forecast** — schedule-driven direct method (`Cash13Config`): per-week receipts and disbursements, raw cash position (no auto-draws, so the liquidity gap is visible), and a runway summary (`min_cash`, `min_week`, `first_negative_week`). -- **IO layer** — read a QuickBooks-style P&L CSV, render a markdown CFO briefing, export to Excel. -- **Data-source adapters** — `from_netsuite` / `from_quickbooks` / `from_shopify` return a normalized `{account: amount}`. Fixture-backed out of the box; each documents its live path (SuiteQL/OAuth, QuickBooks Online, Shopify Admin API). Credentials come from your environment — never committed. +- **IO layer** — `read_pl_csv` reads any `Account, Amount` statement export (P&L, balance sheet, …) to `{account: amount}`; render a markdown CFO briefing; export to Excel. +- **Data-source adapters** — `from_netsuite` / `from_quickbooks` / `from_shopify` return a normalized `{account: amount}` (fixture-backed scaffolds; each documents its live path). But the toolkit isn't limited to these — connect QuickBooks/NetSuite **via MCP**, pull a 10-K, or point it at local spreadsheets, and the agent **builds the ingestion for that source** (see [`pull_edgar.py`](examples/foxfactory/pull_edgar.py)). Credentials come from the host/MCP server — never committed. ## Use it as a library @@ -132,19 +158,22 @@ print(to_briefing_md(monthly, title="My Company", runway=runway)) | Component | Status | |---|---| -| Monthly forecast engine (`pyfpa`) | ✅ Built | -| 13-week cash engine (`pyfpa.cash13`) | ✅ Built | -| IO layer + data-source adapters (`pyfpa.io`) | ✅ Built | -| Runnable demo (`examples/ridgeline`) | ✅ Built | -| Real public-company proof (`examples/foxfactory`) | ✅ Built | -| **Claude skillset** (the hero — see below) | ✅ Built | +| Monthly forecast engine (`pyfpa`) | Built | +| 13-week cash engine (`pyfpa.cash13`) | Built | +| IO layer + data-source adapters (`pyfpa.io`) | Built | +| Runnable demo (`examples/ridgeline`) | Built | +| Real public-company proof (`examples/foxfactory`) | Built | +| Claude skillset (see below) | Built | +| Self-improving backtest loop (`pyfpa.backtest` + `fpa-backtest-learn`) | Built | +| Human corrections + vault memory (`pyfpa.memory` + `fpa-capture-correction`) | Built | +| Cross-client portfolio learning (`pyfpa.portfolio` + `fpa-portfolio-learn`) | Built | **The skillset is the point.** The forecast engine is the substrate; the headline feature is a progressive Claude skillset (in [`skills/`](skills/), installable as a Claude plugin) that drives it across the lifecycle: 1. **`fpa-learn-business`** — interview + financials → a durable business profile, and *generate bespoke skills/agents* for that company (the self-extending part). 2. **`fpa-scaffold-model`** — build a runnable model from a trial balance. -3. **`fpa-configure-actuals`** — wire real numbers / connect a data source (NetSuite · QuickBooks · Shopify). -4. **Operate** — `fpa-monthly-close`, `fpa-cash-runway`, `fpa-board-briefing` — guided throughout by **`fpa-cfo-judgment`**, the encoded gotchas a real finance team knows (pre-close margins lie, D&A is a real expense — not a cash-flow freebie, a goodwill impairment is non-cash — bridge it, raw cash ≠ insolvency). +3. **`fpa-configure-actuals`** — wire in real numbers from wherever they live: local spreadsheets, QuickBooks/NetSuite via MCP or API, a 10-K, and so on. +4. **Operate** — `fpa-monthly-close`, `fpa-cash-runway`, `fpa-board-briefing`, **`fpa-backtest-learn`** (scores past forecasts against your actuals and proposes ratified improvements — the self-improving loop), **`fpa-capture-correction`** (turns a human's "that's off because X" into durable memory that grounds every future forecast), and **`fpa-portfolio-learn`** (distills what generalizes across your whole book into a reusable library that seeds new clients — cross-client learning) — guided throughout by **`fpa-cfo-judgment`**, the encoded gotchas a real finance team knows (pre-close margins lie, D&A is a real expense — not a cash-flow freebie, a goodwill impairment is non-cash — bridge it, raw cash ≠ insolvency). See [`docs/blog/launch.md`](docs/blog/launch.md) for the story — a cold AI agent building a coffee-roaster forecast from a 10-minute intake and writing its own bespoke skill, *and* the same toolkit reconciling Fox Factory's real 10-K to the dollar. @@ -152,7 +181,7 @@ See [`docs/blog/launch.md`](docs/blog/launch.md) for the story — a cold AI age ```bash pip install -e ".[dev]" -pytest -q # 92 tests, all green +pytest -q # the full test suite ``` ## License diff --git a/docs/superpowers/plans/2026-06-08-backtest-learning-loop.md b/docs/superpowers/plans/2026-06-08-backtest-learning-loop.md new file mode 100644 index 0000000..5df991e --- /dev/null +++ b/docs/superpowers/plans/2026-06-08-backtest-learning-loop.md @@ -0,0 +1,848 @@ +# Self-Improving Backtest Loop — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the per-client self-improvement loop — snapshot forecasts, score them against the user's actuals (weighted MAPE), and surface ratified improvements — as a lean tested `pyfpa.backtest` package plus an `fpa-backtest-learn` skill. + +**Architecture:** Hybrid. `pyfpa/backtest/` owns the objective, deterministic mechanics (extract scored lines from a forecast, score vs actuals via the existing `reconcile`, run a holdout backtest on history, and the mechanizable guardrail helpers). The `fpa-backtest-learn` skill owns judgment (attribution, proposals, ratification) and the `.fpa/` memory workflow. All new code is pure, immutable, typed, unit-tested. + +**Tech Stack:** Python 3.11+, pydantic v2, pandas, PyYAML (existing deps). + +**Spec:** `docs/superpowers/specs/2026-06-08-backtest-learning-loop-design.md` + +--- + +## File Structure + +| Path | Responsibility | +| --- | --- | +| `pyfpa/backtest/__init__.py` (create) | Re-export the package's public names. | +| `pyfpa/backtest/score.py` (create) | `DEFAULT_WEIGHTS`, `DEFAULT_SCORE_LINES`, `extract_lines`, `aggregate_periods`, `ScoreResult`, `score_forecast`. | +| `pyfpa/backtest/snapshot.py` (create) | `Snapshot`, `snapshot_forecast`, `save_snapshot`, `load_snapshot`. | +| `pyfpa/backtest/holdout.py` (create) | `holdout_backtest`. | +| `pyfpa/backtest/learn.py` (create) | `magnitude_cap`, `persistent_miss`, `render_scorecard`. | +| `pyfpa/__init__.py` (modify) | Export the public backtest symbols. | +| `skills/fpa-backtest-learn/SKILL.md` (create) | The judgment + `.fpa/` workflow skill. | +| `skills/fpa-monthly-close/SKILL.md` (modify) | Add a "Next" pointer to `fpa-backtest-learn`. | +| `.claude-plugin/plugin.json` (modify) | Register the new skill. | +| `tests/test_backtest_score.py` (create) | Unit tests for scoring + extraction. | +| `tests/test_backtest_snapshot.py` (create) | Unit tests for snapshot round-trip. | +| `tests/test_backtest_holdout.py` (create) | Unit tests for the holdout harness (incl. discrimination). | +| `tests/test_backtest_learn.py` (create) | Unit tests for the guardrail helpers + scorecard render. | +| `tests/test_public_api.py` (modify) | Keep the `__all__` contract test in sync. | + +Conventions (from `~/.claude/rules`): immutability (return new objects), small focused files, clear names, pydantic v2, comprehensive correctness. Tests live FLAT in `tests/` (e.g. `tests/test_cogs.py`). Use `python3 -m pytest`. + +--- + +## Group 1 — Scoring core (`pyfpa/backtest/score.py`) + +### Task 1: `extract_lines` + `aggregate_periods` + +These turn a forecast DataFrame (and a list of per-period actual dicts) into the scored-line dict, handling that `ending_cash` is a stock (take the last), `gross_margin` is a ratio (gross_profit / revenue), and the rest are flows (sum). + +**Files:** +- Create: `pyfpa/backtest/__init__.py` (empty for now), `pyfpa/backtest/score.py` +- Test: `tests/test_backtest_score.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_backtest_score.py +import pandas as pd +import pytest +from pyfpa.backtest.score import extract_lines, aggregate_periods, DEFAULT_SCORE_LINES + + +def _forecast(): + idx = pd.period_range("2026-01", periods=3, freq="M") + return pd.DataFrame({ + "revenue": [100.0, 100.0, 100.0], + "gross_profit": [40.0, 40.0, 40.0], + "ebitda": [30.0, 30.0, 30.0], + "ending_cash": [50.0, 70.0, 90.0], + }, index=idx) + + +def test_extract_lines_flow_stock_ratio(): + out = extract_lines(_forecast(), DEFAULT_SCORE_LINES) + assert out["revenue"] == 300.0 # flow: sum + assert out["ebitda"] == 90.0 # flow: sum + assert out["ending_cash"] == 90.0 # stock: last + assert out["gross_margin"] == pytest.approx(120.0 / 300.0) # ratio: ΣGP/Σrev + + +def test_aggregate_periods_matches_extract(): + periods = [ + {"revenue": 100.0, "gross_profit": 40.0, "ebitda": 30.0, "ending_cash": 50.0}, + {"revenue": 100.0, "gross_profit": 40.0, "ebitda": 30.0, "ending_cash": 70.0}, + {"revenue": 100.0, "gross_profit": 40.0, "ebitda": 30.0, "ending_cash": 90.0}, + ] + out = aggregate_periods(periods, DEFAULT_SCORE_LINES) + assert out["revenue"] == 300.0 + assert out["ending_cash"] == 90.0 # last period's stock + assert out["gross_margin"] == pytest.approx(0.4) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_backtest_score.py -v` +Expected: FAIL (`ModuleNotFoundError: pyfpa.backtest`). + +- [ ] **Step 3: Implement** + +Create empty `pyfpa/backtest/__init__.py`. Then `pyfpa/backtest/score.py`: + +```python +from __future__ import annotations + +from collections.abc import Mapping, Sequence + +import pandas as pd + +DEFAULT_SCORE_LINES = ["ending_cash", "ebitda", "revenue", "gross_margin"] +DEFAULT_WEIGHTS = {"ending_cash": 0.4, "ebitda": 0.3, "revenue": 0.2, "gross_margin": 0.1} + + +def extract_lines(forecast_df: pd.DataFrame, score_lines: Sequence[str]) -> dict[str, float]: + """Reduce a monthly forecast to a {line: value} dict for scoring. + + `ending_cash` is a stock (take the last month); `gross_margin` is a ratio + (Σ gross_profit / Σ revenue); every other line is a flow (sum).""" + revenue = float(forecast_df["revenue"].sum()) + out: dict[str, float] = {} + for line in score_lines: + if line == "ending_cash": + out[line] = float(forecast_df["ending_cash"].iloc[-1]) + elif line == "gross_margin": + gp = float(forecast_df["gross_profit"].sum()) + out[line] = (gp / revenue) if revenue else 0.0 + else: + out[line] = float(forecast_df[line].sum()) + return out + + +def aggregate_periods( + period_dicts: Sequence[Mapping[str, float]], score_lines: Sequence[str] +) -> dict[str, float]: + """Aggregate a chronological list of per-period actual dicts the same way + `extract_lines` aggregates a forecast (flow=sum, stock=last, ratio=ΣGP/Σrev).""" + revenue = sum(float(d.get("revenue", 0.0)) for d in period_dicts) + out: dict[str, float] = {} + for line in score_lines: + if line == "ending_cash": + out[line] = float(period_dicts[-1].get("ending_cash", 0.0)) + elif line == "gross_margin": + gp = sum(float(d.get("gross_profit", 0.0)) for d in period_dicts) + out[line] = (gp / revenue) if revenue else 0.0 + else: + out[line] = sum(float(d.get(line, 0.0)) for d in period_dicts) + return out +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_backtest_score.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/backtest/__init__.py pyfpa/backtest/score.py tests/test_backtest_score.py +git commit -m "feat: backtest line extraction (flow/stock/ratio aggregation)" +``` + +### Task 2: `ScoreResult` + `score_forecast` + +**Files:** +- Modify: `pyfpa/backtest/score.py` +- Test: `tests/test_backtest_score.py` + +- [ ] **Step 1: Write the failing test** + +```python +# append to tests/test_backtest_score.py +from pyfpa.backtest.score import ScoreResult, score_forecast, DEFAULT_WEIGHTS + + +def test_score_forecast_weighted_mape(): + predicted = {"ending_cash": 110.0, "ebitda": 90.0, "revenue": 300.0, "gross_margin": 0.40} + actual = {"ending_cash": 100.0, "ebitda": 100.0, "revenue": 300.0, "gross_margin": 0.40} + res = score_forecast(predicted, actual) + # per-line signed error = predicted/actual - 1 + assert res.per_line["ending_cash"] == pytest.approx(0.10) + assert res.per_line["ebitda"] == pytest.approx(-0.10) + assert res.per_line["revenue"] == pytest.approx(0.0) + # weighted MAPE with DEFAULT_WEIGHTS (0.4,0.3,0.2,0.1): 0.4*0.1 + 0.3*0.1 = 0.07 + assert res.fitness == pytest.approx(0.07) + + +def test_score_forecast_skips_absent_and_zero_actual_lines(): + # 'revenue' missing from predicted, 'ebitda' has zero actual -> both skipped; + # only ending_cash scored -> weights renormalize to 1.0 on it. + res = score_forecast( + {"ending_cash": 90.0, "ebitda": 30.0}, + {"ending_cash": 100.0, "ebitda": 0.0}, + ) + assert set(res.per_line) == {"ending_cash"} + assert res.fitness == pytest.approx(0.10) # 1.0 * |0.9-1| +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_backtest_score.py::test_score_forecast_weighted_mape -v` +Expected: FAIL (`ImportError`). + +- [ ] **Step 3: Implement** + +Append to `pyfpa/backtest/score.py` (add `from pydantic import BaseModel` and `from pyfpa.analysis.reconcile import reconcile` at the top): + +```python +class ScoreResult(BaseModel): + fitness: float # weighted MAPE across scored lines; lower is better + per_line: dict[str, float] # signed error %, predicted/actual - 1 + weights: dict[str, float] # the (renormalized) weights actually used + + +def score_forecast( + predicted: Mapping[str, float], + actual: Mapping[str, float], + *, + weights: Mapping[str, float] | None = None, +) -> ScoreResult: + """Weighted MAPE of predicted vs actual over the scored lines present in both + (and with non-zero actual). Per-line error reuses `reconcile`'s variance_pct. + Weights are renormalized over the lines actually scored.""" + weights = dict(weights or DEFAULT_WEIGHTS) + lines = [l for l in weights if l in predicted and l in actual and actual[l] != 0] + if not lines: + return ScoreResult(fitness=0.0, per_line={}, weights={}) + rec = reconcile({l: predicted[l] for l in lines}, {l: actual[l] for l in lines}) + per_line = {l: float(rec.loc[l, "variance_pct"]) for l in lines} + total_w = sum(weights[l] for l in lines) + used = {l: weights[l] / total_w for l in lines} + fitness = sum(used[l] * abs(per_line[l]) for l in lines) + return ScoreResult(fitness=fitness, per_line=per_line, weights=used) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_backtest_score.py -v` +Expected: PASS (all four). + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/backtest/score.py tests/test_backtest_score.py +git commit -m "feat: weighted-MAPE forecast scoring on top of reconcile" +``` + +--- + +## Group 2 — Snapshot persistence (`pyfpa/backtest/snapshot.py`) + +### Task 3: `Snapshot` + `snapshot_forecast` + YAML round-trip + +**Files:** +- Create: `pyfpa/backtest/snapshot.py` +- Test: `tests/test_backtest_snapshot.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_backtest_snapshot.py +import pandas as pd +import pytest +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.snapshot import Snapshot, snapshot_forecast, save_snapshot, load_snapshot +from pyfpa.backtest.score import score_forecast + + +def _cfg(): + return EntityConfig.model_validate({ + "name": "T", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 0, "dpo_days": 0, "dio_days": 0}, + "opening_balances": {"cash": 0.0}, + }) + + +def test_snapshot_forecast_captures_assumptions_and_predictions(): + cfg = _cfg() + snap = snapshot_forecast(cfg, cashflow_from_config(cfg), + label="2026", created="2026-02-01") + assert snap.label == "2026" + assert snap.assumptions["channels"][0]["annual_revenue"] == 1_200_000.0 + assert snap.predicted["revenue"] == pytest.approx(1_200_000.0) + assert snap.score is None + + +def test_snapshot_round_trip(tmp_path): + cfg = _cfg() + snap = snapshot_forecast(cfg, cashflow_from_config(cfg), + label="2026", created="2026-02-01") + snap = snap.model_copy(update={"score": score_forecast(snap.predicted, snap.predicted)}) + p = tmp_path / "2026.snapshot.yaml" + save_snapshot(snap, p) + back = load_snapshot(p) + assert back.label == "2026" + assert back.predicted == snap.predicted + assert back.score.fitness == pytest.approx(0.0) + assert back.assumptions == snap.assumptions +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_backtest_snapshot.py -v` +Expected: FAIL (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** + +```python +# pyfpa/backtest/snapshot.py +from __future__ import annotations + +from collections.abc import Sequence +from pathlib import Path + +import pandas as pd +import yaml +from pydantic import BaseModel + +from pyfpa.config.schemas import EntityConfig +from pyfpa.backtest.score import DEFAULT_SCORE_LINES, ScoreResult, extract_lines + + +class Snapshot(BaseModel): + """The full record of one forecast: the assumptions it used, the lines it + predicted, and (once the period closes) the realized score.""" + label: str + created: str # caller-supplied date string; never generated here + assumptions: dict # serialized EntityConfig + predicted: dict[str, float] + score: ScoreResult | None = None + + +def snapshot_forecast( + cfg: EntityConfig, + forecast_df: pd.DataFrame, + *, + label: str, + created: str, + score_lines: Sequence[str] = DEFAULT_SCORE_LINES, +) -> Snapshot: + return Snapshot( + label=label, + created=created, + assumptions=cfg.model_dump(), + predicted=extract_lines(forecast_df, score_lines), + ) + + +def save_snapshot(snapshot: Snapshot, path: str | Path) -> None: + Path(path).write_text(yaml.safe_dump(snapshot.model_dump(), sort_keys=False)) + + +def load_snapshot(path: str | Path) -> Snapshot: + p = Path(path) + if not p.exists(): + raise FileNotFoundError(f"snapshot not found: {p}") + return Snapshot.model_validate(yaml.safe_load(p.read_text())) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_backtest_snapshot.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/backtest/snapshot.py tests/test_backtest_snapshot.py +git commit -m "feat: forecast snapshots with YAML round-trip" +``` + +--- + +## Group 3 — Holdout backtest (`pyfpa/backtest/holdout.py`) + +### Task 4: `holdout_backtest` + +**Files:** +- Create: `pyfpa/backtest/holdout.py` +- Test: `tests/test_backtest_holdout.py` + +The harness splits ordered periods into fit (all but the last `holdout`) and holdout (the last `holdout`); calls the caller's `build_cfg_fn(fit_actuals)` to get a config that forecasts the holdout window; runs the engine; and scores predicted vs the aggregated holdout actuals. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_backtest_holdout.py +import pytest +from pyfpa.config.schemas import EntityConfig +from pyfpa.backtest.holdout import holdout_backtest + + +def _actuals(): + # 6 months, revenue flat at 100/mo, gross_profit 40, ebitda 30, cash +20/mo + out = {} + cash = 0.0 + for i in range(6): + cash += 20.0 + out[f"2026-{i+1:02d}"] = {"revenue": 100.0, "gross_profit": 40.0, + "ebitda": 30.0, "ending_cash": cash} + return out + + +def _build_cfg(growth): + # build_cfg_fn factory: forecasts a 3-month holdout at the given monthly run-rate. + def _fn(fit_actuals): + n = len(fit_actuals) + last_rev = list(fit_actuals.values())[-1]["revenue"] + annual = last_rev * 12 * (1 + growth) + return EntityConfig.model_validate({ + "name": "h", "start_month": f"2026-{n+1:02d}", "horizon_months": 3, + "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": annual, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.6}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 0, "dpo_days": 0, "dio_days": 0}, + "opening_balances": {"cash": 80.0}, # fit-window ending cash (4 * 20) + }) + return _fn + + +def test_holdout_backtest_scores_holdout(): + res = holdout_backtest(_actuals(), _build_cfg(0.0), holdout=3, + score_lines=["revenue", "ebitda"]) + assert "revenue" in res.per_line + assert res.fitness >= 0.0 + + +def test_holdout_discriminates_better_assumption(): + # the flat (0% growth) model matches the flat actuals better than a +50% one + good = holdout_backtest(_actuals(), _build_cfg(0.0), holdout=3, score_lines=["revenue"]) + bad = holdout_backtest(_actuals(), _build_cfg(0.5), holdout=3, score_lines=["revenue"]) + assert good.fitness < bad.fitness + + +def test_holdout_rejects_too_few_periods(): + with pytest.raises(ValueError): + holdout_backtest({"2026-01": {"revenue": 1.0}}, _build_cfg(0.0), holdout=3, + score_lines=["revenue"]) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_backtest_holdout.py -v` +Expected: FAIL (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** + +```python +# pyfpa/backtest/holdout.py +from __future__ import annotations + +from collections.abc import Callable, Mapping, Sequence + +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.score import ( + DEFAULT_SCORE_LINES, ScoreResult, aggregate_periods, extract_lines, score_forecast, +) + +BuildCfgFn = Callable[[dict[str, Mapping[str, float]]], EntityConfig] + + +def holdout_backtest( + actuals_by_period: Mapping[str, Mapping[str, float]], + build_cfg_fn: BuildCfgFn, + *, + holdout: int, + score_lines: Sequence[str] = DEFAULT_SCORE_LINES, + weights: Mapping[str, float] | None = None, +) -> ScoreResult: + """Fit on all but the last `holdout` periods, predict the holdout, and score + predicted vs the held-out actuals. The business-specific `build_cfg_fn` + (fit actuals -> a config that forecasts the holdout window) is supplied by the + caller; this harness only owns the split and the scoring. Nothing is scored on + data it was fit on.""" + periods = list(actuals_by_period) + if len(periods) <= holdout: + raise ValueError(f"need more than {holdout} periods, got {len(periods)}") + fit_periods = periods[:-holdout] + holdout_periods = periods[-holdout:] + + fit_actuals = {p: dict(actuals_by_period[p]) for p in fit_periods} + cfg = build_cfg_fn(fit_actuals) + predicted = extract_lines(cashflow_from_config(cfg), score_lines) + actual = aggregate_periods([dict(actuals_by_period[p]) for p in holdout_periods], score_lines) + return score_forecast(predicted, actual, weights=weights) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_backtest_holdout.py -v` +Expected: PASS (all three — note the discrimination test is the core proof the metric works). + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/backtest/holdout.py tests/test_backtest_holdout.py +git commit -m "feat: holdout backtest harness (train/holdout split + score)" +``` + +--- + +## Group 4 — Guardrail + render helpers (`pyfpa/backtest/learn.py`) + +### Task 5: `magnitude_cap`, `persistent_miss`, `render_scorecard` + +**Files:** +- Create: `pyfpa/backtest/learn.py` +- Test: `tests/test_backtest_learn.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_backtest_learn.py +import pytest +from pyfpa.backtest.learn import magnitude_cap, persistent_miss, render_scorecard +from pyfpa.backtest.snapshot import Snapshot +from pyfpa.backtest.score import ScoreResult + + +def test_magnitude_cap_clamps_relative(): + assert magnitude_cap(100.0, 200.0, cap=0.25) == pytest.approx(125.0) # up-clamp + assert magnitude_cap(100.0, 10.0, cap=0.25) == pytest.approx(75.0) # down-clamp + assert magnitude_cap(100.0, 110.0, cap=0.25) == pytest.approx(110.0) # within cap + assert magnitude_cap(0.0, 5.0, cap=0.25) == 5.0 # zero base: pass + + +def test_persistent_miss_requires_k_same_signed(): + assert persistent_miss([0.1, 0.2], k=2) is True # two positive + assert persistent_miss([-0.1, -0.2], k=2) is True # two negative + assert persistent_miss([0.1, -0.2], k=2) is False # sign flip + assert persistent_miss([0.2], k=2) is False # too short + assert persistent_miss([0.001, 0.001], k=2, threshold=0.01) is False # under threshold + + +def test_render_scorecard_table(): + snaps = [ + Snapshot(label="2026-01", created="2026-02-01", assumptions={}, predicted={}, + score=ScoreResult(fitness=0.05, per_line={"revenue": 0.02}, weights={})), + Snapshot(label="2026-02", created="2026-03-01", assumptions={}, predicted={}, + score=ScoreResult(fitness=0.03, per_line={"revenue": -0.01}, weights={})), + ] + md = render_scorecard(snaps) + assert "| 2026-01 |" in md + assert "| 2026-02 |" in md + assert "fitness" in md.lower() +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_backtest_learn.py -v` +Expected: FAIL (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** + +```python +# pyfpa/backtest/learn.py +from __future__ import annotations + +from collections.abc import Sequence + +from pyfpa.backtest.snapshot import Snapshot + + +def magnitude_cap(old: float, new: float, *, cap: float = 0.25) -> float: + """Clamp `new` to within ±`cap` (relative) of `old`, preventing overcorrection. + A zero base has no defined relative bound, so `new` passes through.""" + if old == 0: + return new + lo, hi = sorted((old * (1 - cap), old * (1 + cap))) + return max(lo, min(new, hi)) + + +def persistent_miss(errors: Sequence[float], *, k: int = 2, threshold: float = 0.0) -> bool: + """True when the last `k` per-line errors are all the same sign and all exceed + `threshold` in magnitude — a repeated, directional miss, not noise.""" + if len(errors) < k: + return False + last = errors[-k:] + return (all(e > threshold for e in last)) or (all(e < -threshold for e in last)) + + +def render_scorecard(snapshots: Sequence[Snapshot]) -> str: + """Render scored snapshots as a markdown track record (chronological).""" + lines = ["# Forecast Scorecard", "", "| Period | Fitness (lower=better) | Per-line error |", + "|---|--:|---|"] + for s in snapshots: + if s.score is None: + continue + errs = ", ".join(f"{k} {v * 100:+.1f}%" for k, v in s.score.per_line.items()) + lines.append(f"| {s.label} | {s.score.fitness:.4f} | {errs} |") + return "\n".join(lines) + "\n" +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_backtest_learn.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/backtest/learn.py tests/test_backtest_learn.py +git commit -m "feat: guardrail helpers (magnitude cap, persistence) + scorecard render" +``` + +--- + +## Group 5 — Package wiring + +### Task 6: Re-export + public API + +**Files:** +- Modify: `pyfpa/backtest/__init__.py`, `pyfpa/__init__.py`, `tests/test_public_api.py` + +- [ ] **Step 1: Fill `pyfpa/backtest/__init__.py`** + +```python +from pyfpa.backtest.score import ( + DEFAULT_SCORE_LINES, DEFAULT_WEIGHTS, ScoreResult, + aggregate_periods, extract_lines, score_forecast, +) +from pyfpa.backtest.snapshot import Snapshot, snapshot_forecast, save_snapshot, load_snapshot +from pyfpa.backtest.holdout import holdout_backtest +from pyfpa.backtest.learn import magnitude_cap, persistent_miss, render_scorecard + +__all__ = [ + "DEFAULT_SCORE_LINES", "DEFAULT_WEIGHTS", "ScoreResult", "aggregate_periods", + "extract_lines", "score_forecast", "Snapshot", "snapshot_forecast", + "save_snapshot", "load_snapshot", "holdout_backtest", "magnitude_cap", + "persistent_miss", "render_scorecard", +] +``` + +- [ ] **Step 2: Export from `pyfpa/__init__.py`** — after the existing analysis imports add: + +```python +from pyfpa.backtest import ( + ScoreResult, score_forecast, Snapshot, snapshot_forecast, + save_snapshot, load_snapshot, holdout_backtest, + magnitude_cap, persistent_miss, render_scorecard, +) +``` + +And extend `pyfpa/__init__.py`'s `__all__` with those same names: + +```python + "ScoreResult", "score_forecast", "Snapshot", "snapshot_forecast", + "save_snapshot", "load_snapshot", "holdout_backtest", + "magnitude_cap", "persistent_miss", "render_scorecard", +``` + +- [ ] **Step 3: Sync the public-API contract test** — `tests/test_public_api.py` asserts `set(pyfpa.__all__)` equals an expected set. Add the same ten names above to that expected set. (Read the test first; match its exact assertion form.) + +- [ ] **Step 4: Verify + full suite** + +Run: `python3 -c "import pyfpa; print(pyfpa.score_forecast, pyfpa.holdout_backtest, pyfpa.Snapshot)"` +Expected: prints the three symbols. +Run: `python3 -m pytest -q` +Expected: all pass. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/backtest/__init__.py pyfpa/__init__.py tests/test_public_api.py +git commit -m "feat: export pyfpa.backtest public API" +``` + +--- + +## Group 6 — The skill + +### Task 7: `fpa-backtest-learn` skill + registration + +**Files:** +- Create: `skills/fpa-backtest-learn/SKILL.md` +- Modify: `skills/fpa-monthly-close/SKILL.md` (add a Next pointer), `.claude-plugin/plugin.json` + +- [ ] **Step 1: Write `skills/fpa-backtest-learn/SKILL.md`** with this content: + +```markdown +--- +name: fpa-backtest-learn +description: Use when you want the model to learn from how its past forecasts actually turned out — scoring forecasts against the company's real actuals, backtesting assumptions on history, and proposing ratified improvements. Runs at/after monthly close. +--- + +# Backtest & Learn (Operate) + +## Overview + +The model should get measurably better at this business over time. This skill +scores past forecasts against the company's actuals, surfaces what keeps missing, +and proposes improvements a human ratifies. The objective metric is reconciliation +error against the user's own books (`pyfpa.score_forecast`) — the FP&A analog of a +validation loss. + +**Core principle:** self-improving, but never self-ratifying. The loop proposes; +a human accepts. Everything it learns lives as plain files in `.fpa/` the user owns. + +## Memory (`.fpa/`) + +- `forecasts/.snapshot.yaml` — each forecast's assumptions + predictions, and (after close) its score. +- `scorecard.md` — the running track record (rendered, never hand-edited). +- `learnings.md` — every accepted change: what, the evidence, the backtest delta, the date. + +## Workflow + +1. **Snapshot every forecast.** When you produce a forecast, persist it: + `snapshot_forecast(cfg, forecast_df, label=, created=)` → + `save_snapshot(..., ".fpa/forecasts/.snapshot.yaml")`. +2. **Score at close.** When a period closes (actuals via **fpa-configure-actuals**), + load that period's snapshot, `score_forecast(snap.predicted, actuals)`, write the + score back into the snapshot, and re-render `scorecard.md` with `render_scorecard`. +3. **Attribute** each material per-line miss to a driver (volume / price / cost ratio + / working-capital timing). **Run the fpa-cfo-judgment one-time-item screen first** — + never blame the model for a one-off. +4. **Propose**, tagged by type: + - **Parametric** (an assumption change): re-score it with `holdout_backtest` on the + company's history. Surface it **only if it lowers holdout fitness** (not in-sample), + ranked by the delta. Clamp the proposed move with `magnitude_cap` (±25%/cycle). + - **Structural** (a methodology/skill change, e.g. a revenue-recognition lag): surface + **only** when `persistent_miss` is true for the line (same-signed across K≥2 closes) + and it survived the one-time screen. Hand it to **fpa-learn-business** to generate the + skill *on approval* — propose, don't auto-write. +5. **Ratify + log.** Present proposals; the human accepts/rejects. On accept, update the + config and append to `learnings.md` (what, evidence, holdout delta, date). Reversible. + +## Bootstrap (day one) + +If the company already has ~12+ months of actuals, you don't have to wait: run +`holdout_backtest` immediately (fit on the earlier months, hold out the recent ones) +to report current accuracy and the first round of parametric proposals. + +## Guardrails (always) + +- Never score on data the model was fit on (`holdout_backtest` enforces this). +- No proposal off a single period — require a persistent, same-signed miss. +- A parametric change must improve **holdout** fitness, not in-sample fit. +- Cap how far any assumption moves per cycle (`magnitude_cap`). +- Human ratifies everything; every change is logged and reversible. + +## Next + +Scored + learned → **fpa-board-briefing** (report the forecast and how it's tracking). +``` + +- [ ] **Step 2: Add a Next pointer in `skills/fpa-monthly-close/SKILL.md`** — its `## Next` section currently points to fpa-cash-runway and fpa-board-briefing. Append: ` and **fpa-backtest-learn** (score this close against the prior forecast and learn from the miss).` + +- [ ] **Step 3: Register the skill in `.claude-plugin/plugin.json`** — read the file, find the array/list of skills, and add `fpa-backtest-learn` in the same shape as the existing entries (match their exact format — name/path keys). + +- [ ] **Step 4: Verify the plugin manifest is valid JSON** + +Run: `python3 -c "import json; json.load(open('.claude-plugin/plugin.json')); print('ok')"` +Expected: `ok`. + +- [ ] **Step 5: Commit** + +```bash +git add skills/fpa-backtest-learn/ skills/fpa-monthly-close/SKILL.md .claude-plugin/plugin.json +git commit -m "feat: fpa-backtest-learn skill — score, backtest, propose, ratify" +``` + +--- + +## Group 7 — Scenario test + docs + +### Task 8: Structural-miss scenario test + +**Files:** +- Create: `tests/test_backtest_scenario.py` + +Proves the loop's structural trigger: a business whose cash conversion lags (predicted ending cash persistently overstates actual) produces a persistent, same-signed miss that `persistent_miss` flags. + +- [ ] **Step 1: Write the test** + +```python +# tests/test_backtest_scenario.py +from pyfpa.backtest.score import score_forecast +from pyfpa.backtest.learn import persistent_miss + + +def test_persistent_cash_overstatement_is_flagged(): + # Across three closes the model predicts ending cash above actual every time + # (a real collections lag the model doesn't capture) -> structural trigger. + closes = [ + ({"ending_cash": 100.0}, {"ending_cash": 90.0}), + ({"ending_cash": 120.0}, {"ending_cash": 108.0}), + ({"ending_cash": 140.0}, {"ending_cash": 126.0}), + ] + cash_errors = [score_forecast(p, a).per_line["ending_cash"] for p, a in closes] + assert all(e > 0 for e in cash_errors) # always overstated + assert persistent_miss(cash_errors, k=3, threshold=0.05) is True +``` + +- [ ] **Step 2: Run test to verify it passes** + +Run: `python3 -m pytest tests/test_backtest_scenario.py -v` +Expected: PASS (this exercises already-built functions, so it passes immediately — it's a behavior lock, not new code). + +- [ ] **Step 3: Run the full suite** + +Run: `python3 -m pytest -q` +Expected: all pass. + +- [ ] **Step 4: Commit** + +```bash +git add tests/test_backtest_scenario.py +git commit -m "test: structural-miss scenario (persistent cash overstatement flagged)" +``` + +### Task 9: README skillset + roadmap mention + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Add the skill to the skillset list** — in the "The skillset is the point" numbered list, the **Operate** item names `fpa-monthly-close`, `fpa-cash-runway`, `fpa-board-briefing`. Add `fpa-backtest-learn` to that Operate item with a short gloss: "scores past forecasts against your actuals and proposes ratified improvements — the self-improving loop." + +- [ ] **Step 2: Add a roadmap row** — in the "Project status & roadmap" table add: `| Self-improving backtest loop (\`pyfpa.backtest\` + \`fpa-backtest-learn\`) | ✅ Built |`. + +- [ ] **Step 3: Commit** + +```bash +git add README.md +git commit -m "docs: surface the self-improving backtest loop in the README" +``` + +### Task 10: Final verification + PR + +- [ ] **Step 1: Full suite** + +Run: `python3 -m pytest -q` +Expected: all green. + +- [ ] **Step 2: Open the PR** (stacked on `feat/foxfactory-example`, which carries `reconcile`) + +```bash +git push -u origin feat/backtest-learning-loop +gh pr create --base main --title "feat: self-improving backtest loop (pyfpa.backtest + fpa-backtest-learn)" \ + --body "Per-client self-improvement loop: snapshot forecasts, score them against the user's actuals (weighted MAPE on cash/EBITDA/revenue/margin via reconcile), holdout-backtest on history, and propose ratified parametric + structural improvements. Memory is transparent files in .fpa/. Lean tested pyfpa.backtest package + fpa-backtest-learn skill. Spec: docs/superpowers/specs/2026-06-08-backtest-learning-loop-design.md. Depends on reconcile from #1." +``` + +Expected: PR opened. Jeff reviews/merges. + +--- + +## Self-Review notes + +- **Spec coverage:** memory layout → Tasks 3,5,7. `score.py` (extract/score) → Tasks 1,2. `snapshot.py` → Task 3. `holdout.py` → Task 4. `learn.py` (cap/persistence/render) → Task 5. Skill responsibilities (snapshot/score/attribute/propose/ratify) → Task 7. Guardrails → Tasks 4 (holdout separation), 5 (cap, persistence), 7 (one-time screen, improve-out-of-sample, human ratify). Fitness metric → Task 2. Bootstrap holdout → Tasks 4, 7. Success criteria 1–7 → covered. Testing list → Tasks 1,2,3,4,5,8. +- **Signature consistency:** `extract_lines`, `aggregate_periods`, `ScoreResult(fitness, per_line, weights)`, `score_forecast(predicted, actual, *, weights)`, `Snapshot(label, created, assumptions, predicted, score)`, `snapshot_forecast(cfg, forecast_df, *, label, created, score_lines)`, `holdout_backtest(actuals_by_period, build_cfg_fn, *, holdout, score_lines, weights)`, `magnitude_cap(old, new, *, cap)`, `persistent_miss(errors, *, k, threshold)`, `render_scorecard(snapshots)` — used identically everywhere. +- **No autonomous apply / no cross-client library / no new ingestion** — honored (scope boundaries). diff --git a/docs/superpowers/plans/2026-06-08-memory-corrections.md b/docs/superpowers/plans/2026-06-08-memory-corrections.md new file mode 100644 index 0000000..0f57740 --- /dev/null +++ b/docs/superpowers/plans/2026-06-08-memory-corrections.md @@ -0,0 +1,607 @@ +# Human Corrections & Vault-Friendly Memory — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Capture unprompted human corrections as typed, vault-native markdown memory that grounds future forecasts — a lean tested `pyfpa.memory` package + an `fpa-capture-correction` skill + integration into the existing skills. + +**Architecture:** Hybrid. `pyfpa/memory/` owns the deterministic mechanics (a dotted-path config setter, the `Correction` model, markdown round-trip, and `apply_corrections`). The `fpa-capture-correction` skill owns judgment (classify the correction, draft the override, confirm interpretation). Corrections are plain markdown the user owns; Obsidian is an optional viewer, never required. All new code is pure, immutable, typed, unit-tested. + +**Tech Stack:** Python 3.11+, pydantic v2, PyYAML (existing deps). + +**Spec:** `docs/superpowers/specs/2026-06-08-memory-corrections-design.md` + +--- + +## File Structure + +| Path | Responsibility | +| --- | --- | +| `pyfpa/memory/__init__.py` (create) | Re-export the package's public names. | +| `pyfpa/memory/paths.py` (create) | `_set_by_path` — dotted-path setter with `[n]`/`[*]` support. | +| `pyfpa/memory/corrections.py` (create) | `Override`, `Correction`, `load_corrections`, `save_correction`, `apply_corrections`. | +| `pyfpa/__init__.py` (modify) | Export the public memory symbols. | +| `skills/fpa-capture-correction/SKILL.md` (create) | Conversational capture + the `.fpa/MEMORY.md` convention. | +| `skills/fpa-learn-business/SKILL.md` (modify) | Apply parametric corrections; route structural ones. | +| `skills/fpa-cfo-judgment/SKILL.md` (modify) | One-time screen reads `context` corrections. | +| `skills/fpa-backtest-learn/SKILL.md` (modify) | Monitor applied corrections; flag stale. | +| `README.md` (modify) | Surface the capture skill + memory in the skillset/roadmap. | +| `tests/test_memory_paths.py` (create) | Unit tests for the path setter. | +| `tests/test_memory_corrections.py` (create) | Unit tests for model, round-trip, apply. | +| `tests/test_public_api.py` (modify) | Keep the `__all__` contract test in sync. | + +Conventions: immutability (return new objects), small focused files, clear names, pydantic v2. Tests live FLAT in `tests/`. Use `python3 -m pytest`. + +--- + +## Group 1 — Dotted-path setter (`pyfpa/memory/paths.py`) + +### Task 1: `_set_by_path` + +Supports `name`, `name[n]` (list index), and `name[*]` (every list item). Used by `apply_corrections` to write a correction's override into a config dict. + +**Files:** +- Create: `pyfpa/memory/__init__.py` (empty for now), `pyfpa/memory/paths.py` +- Test: `tests/test_memory_paths.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_memory_paths.py +import pytest +from pyfpa.memory.paths import _set_by_path + + +def test_set_nested_scalar(): + data = {"working_capital": {"dio_days": 30.0}} + _set_by_path(data, "working_capital.dio_days", 45.0) + assert data["working_capital"]["dio_days"] == 45.0 + + +def test_set_list_index(): + data = {"channels": [{"seasonality": [1.0] * 12}]} + _set_by_path(data, "channels[0].seasonality[11]", 2.0) + assert data["channels"][0]["seasonality"][11] == 2.0 + + +def test_set_star_applies_to_all_list_items(): + data = {"channels": [{"cogs_pct": 0.5}, {"cogs_pct": 0.4}]} + _set_by_path(data, "channels[*].cogs_pct", 0.6) + assert [c["cogs_pct"] for c in data["channels"]] == [0.6, 0.6] + + +def test_bad_path_raises_valueerror(): + with pytest.raises(ValueError): + _set_by_path({"a": {}}, "a.missing.deep", 1.0) + with pytest.raises(ValueError): + _set_by_path({"a": 1}, "a[", 1.0) # malformed segment +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_memory_paths.py -v` +Expected: FAIL (`ModuleNotFoundError: pyfpa.memory`). + +- [ ] **Step 3: Implement** + +Create empty `pyfpa/memory/__init__.py`. Then `pyfpa/memory/paths.py`: + +```python +from __future__ import annotations + +import re + +_SEGMENT = re.compile(r"^(\w+)(?:\[(\*|\d+)\])?$") + + +def _parse_path(path: str) -> list[tuple[str, str | int | None]]: + parsed: list[tuple[str, str | int | None]] = [] + for segment in path.split("."): + match = _SEGMENT.match(segment) + if not match: + raise ValueError(f"malformed override path segment: {segment!r}") + key, index = match.group(1), match.group(2) + if index is None: + parsed.append((key, None)) + elif index == "*": + parsed.append((key, "*")) + else: + parsed.append((key, int(index))) + return parsed + + +def _set_segments(node, segments, value) -> None: + (key, index), rest = segments[0], segments[1:] + if not isinstance(node, dict) or key not in node: + raise ValueError(f"override path key not found: {key!r}") + if index is None: + if rest: + _set_segments(node[key], rest, value) + else: + node[key] = value + return + target = node[key] + items = range(len(target)) if index == "*" else [int(index)] + for i in items: + if rest: + _set_segments(target[i], rest, value) + else: + target[i] = value + + +def _set_by_path(data: dict, path: str, value: float) -> None: + """Set `value` at `path` in `data` (in place). Supports dotted keys, `[n]` + list indices, and `[*]` (every item of a list). Raises ValueError on any + unresolvable or malformed path.""" + try: + _set_segments(data, _parse_path(path), value) + except (KeyError, IndexError, TypeError) as exc: + raise ValueError(f"cannot apply override path {path!r}: {exc}") from exc +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_memory_paths.py -v` +Expected: PASS (all four). + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/memory/__init__.py pyfpa/memory/paths.py tests/test_memory_paths.py +git commit -m "feat: dotted-path config setter for corrections (name, [n], [*])" +``` + +--- + +## Group 2 — Correction model + markdown round-trip (`pyfpa/memory/corrections.py`) + +### Task 2: `Override` + `Correction` + `save_correction` / `load_corrections` + +**Files:** +- Create: `pyfpa/memory/corrections.py` +- Test: `tests/test_memory_corrections.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_memory_corrections.py +import pytest +from pyfpa.memory.corrections import ( + Override, Correction, save_correction, load_corrections, +) + + +def _correction(): + return Correction( + slug="2026-06-08-december-seasonality", + type="parametric", + target="channels[*].seasonality[11]", + status="applied", + date="2026-06-08", + override=Override(path="channels[*].seasonality[11]", value=2.0), + notes="**Was off:** even spread.\n\n**Why:** [[business-profile#seasonality]]", + ) + + +def test_correction_round_trip(tmp_path): + save_correction(_correction(), tmp_path) + loaded = load_corrections(tmp_path) + assert len(loaded) == 1 + c = loaded[0] + assert c.slug == "2026-06-08-december-seasonality" + assert c.type == "parametric" + assert c.status == "applied" + assert c.override == Override(path="channels[*].seasonality[11]", value=2.0) + assert "Was off" in c.notes + assert "[[business-profile#seasonality]]" in c.notes + + +def test_load_missing_dir_is_empty(tmp_path): + assert load_corrections(tmp_path / "nope") == [] + + +def test_structural_correction_has_no_override(tmp_path): + c = Correction(slug="defrev", type="structural", target="revenue", + status="open", date="2026-06-08", + notes="**Was off:** double-counting deferred revenue.") + save_correction(c, tmp_path) + back = load_corrections(tmp_path)[0] + assert back.type == "structural" + assert back.override is None +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_memory_corrections.py -v` +Expected: FAIL (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** + +```python +# pyfpa/memory/corrections.py +from __future__ import annotations + +from pathlib import Path +from typing import Literal + +import yaml +from pydantic import BaseModel + +CorrectionType = Literal["parametric", "structural", "context"] +CorrectionStatus = Literal["open", "applied", "superseded"] + + +class Override(BaseModel): + path: str + value: float + + +class Correction(BaseModel): + """One human correction. Frontmatter fields are the machine-readable contract; + `notes` is the human-readable markdown body.""" + slug: str + type: CorrectionType + target: str + status: CorrectionStatus = "open" + date: str + override: Override | None = None + notes: str = "" + + +def _split_frontmatter(text: str) -> tuple[dict, str]: + if text.startswith("---"): + _, frontmatter, body = text.split("---", 2) + return yaml.safe_load(frontmatter) or {}, body.strip() + return {}, text.strip() + + +def save_correction(correction: Correction, directory: str | Path) -> None: + """Write `.md` (YAML frontmatter + markdown body) into `directory`.""" + directory = Path(directory) + directory.mkdir(parents=True, exist_ok=True) + data = correction.model_dump(exclude_none=True) + body = data.pop("notes", "") + data.pop("slug") + text = "---\n" + yaml.safe_dump(data, sort_keys=False) + "---\n" + body + "\n" + (directory / f"{correction.slug}.md").write_text(text) + + +def load_corrections(directory: str | Path) -> list[Correction]: + """Load every `*.md` correction in `directory` (slug = filename stem). + A missing directory returns an empty list.""" + directory = Path(directory) + if not directory.exists(): + return [] + out: list[Correction] = [] + for path in sorted(directory.glob("*.md")): + frontmatter, body = _split_frontmatter(path.read_text()) + out.append(Correction.model_validate({**frontmatter, "slug": path.stem, "notes": body})) + return out +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_memory_corrections.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/memory/corrections.py tests/test_memory_corrections.py +git commit -m "feat: Correction model + vault-native markdown round-trip" +``` + +--- + +## Group 3 — `apply_corrections` + +### Task 3: apply parametric corrections to a config + +**Files:** +- Modify: `pyfpa/memory/corrections.py` +- Test: `tests/test_memory_corrections.py` + +- [ ] **Step 1: Write the failing test** + +```python +# append to tests/test_memory_corrections.py +from pyfpa.config.schemas import EntityConfig +from pyfpa.memory.corrections import apply_corrections + + +def _cfg(): + return EntityConfig.model_validate({ + "name": "T", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": 30.0}, + "opening_balances": {"cash": 0.0}, + }) + + +def test_apply_parametric_override_returns_new_cfg(): + cfg = _cfg() + corr = Correction(slug="dio", type="parametric", target="working_capital.dio_days", + status="applied", date="2026-06-08", + override=Override(path="working_capital.dio_days", value=45.0)) + out = apply_corrections(cfg, [corr]) + assert out.working_capital.dio_days == 45.0 + assert cfg.working_capital.dio_days == 30.0 # input unmutated + + +def test_apply_star_seasonality(): + cfg = _cfg() + corr = Correction(slug="dec", type="parametric", target="channels[*].seasonality[11]", + status="applied", date="2026-06-08", + override=Override(path="channels[*].seasonality[11]", value=2.0)) + out = apply_corrections(cfg, [corr]) + assert out.channels[0].seasonality[11] == 2.0 + + +def test_apply_ignores_open_structural_context(): + cfg = _cfg() + corrections = [ + Correction(slug="a", type="parametric", target="working_capital.dio_days", + status="open", date="2026-06-08", + override=Override(path="working_capital.dio_days", value=99.0)), + Correction(slug="b", type="structural", target="revenue", status="applied", + date="2026-06-08"), + Correction(slug="c", type="context", target="revenue", status="applied", + date="2026-06-08"), + ] + out = apply_corrections(cfg, corrections) + assert out.working_capital.dio_days == 30.0 # nothing applied +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m pytest tests/test_memory_corrections.py::test_apply_parametric_override_returns_new_cfg -v` +Expected: FAIL (`ImportError`). + +- [ ] **Step 3: Implement** + +Add to the top of `pyfpa/memory/corrections.py`: + +```python +from pyfpa.config.schemas import EntityConfig +from pyfpa.memory.paths import _set_by_path +``` + +Append: + +```python +def apply_corrections(cfg: EntityConfig, corrections: list[Correction]) -> EntityConfig: + """Return a NEW config with every `applied` + `parametric` correction's override + written in. `open`, `structural`, and `context` corrections are ignored (the + latter two are routed by the skill, not applied to the model). Input unmutated.""" + data = cfg.model_dump() + for correction in corrections: + if correction.status == "applied" and correction.type == "parametric" and correction.override: + _set_by_path(data, correction.override.path, correction.override.value) + return EntityConfig.model_validate(data) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `python3 -m pytest tests/test_memory_corrections.py -v` +Expected: PASS (all six). Then full suite `python3 -m pytest -q`, no regressions. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/memory/corrections.py tests/test_memory_corrections.py +git commit -m "feat: apply_corrections — parametric overrides into a new config" +``` + +--- + +## Group 4 — Package wiring + +### Task 4: Re-export + public API + +**Files:** +- Modify: `pyfpa/memory/__init__.py`, `pyfpa/__init__.py`, `tests/test_public_api.py` + +- [ ] **Step 1: Fill `pyfpa/memory/__init__.py`** + +```python +from pyfpa.memory.corrections import ( + Override, Correction, load_corrections, save_correction, apply_corrections, +) + +__all__ = [ + "Override", "Correction", "load_corrections", "save_correction", "apply_corrections", +] +``` + +- [ ] **Step 2: Export from `pyfpa/__init__.py`** — after the existing `from pyfpa.backtest import ...` block add: + +```python +from pyfpa.memory import ( + Override, Correction, load_corrections, save_correction, apply_corrections, +) +``` + +And extend `pyfpa/__init__.py`'s `__all__` with: + +```python + "Override", "Correction", "load_corrections", "save_correction", "apply_corrections", +``` + +- [ ] **Step 3: Sync the contract test** — `tests/test_public_api.py` asserts `set(pyfpa.__all__) == {expected}` (exact-set-equality). Add the same five names to that expected set. Read the test first; do NOT weaken the assertion. + +- [ ] **Step 4: Verify + full suite** + +Run: `python3 -c "import pyfpa; print(pyfpa.Correction, pyfpa.apply_corrections, pyfpa.load_corrections)"` +Expected: prints the three symbols. +Run: `python3 -m pytest -q` +Expected: all pass. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/memory/__init__.py pyfpa/__init__.py tests/test_public_api.py +git commit -m "feat: export pyfpa.memory public API" +``` + +--- + +## Group 5 — Capture skill + integrations + +### Task 5: `fpa-capture-correction` skill + the MEMORY.md convention + +**Files:** +- Create: `skills/fpa-capture-correction/SKILL.md` + +- [ ] **Step 1: Write `skills/fpa-capture-correction/SKILL.md`** with this content: + +```markdown +--- +name: fpa-capture-correction +description: Use when a human reviewing a forecast catches something off ("December always spikes", "you're double-counting deferred revenue", "that Q3 number was a one-time contract") — captures it as a durable, typed correction in the company's memory so future forecasts are grounded by it. +--- + +# Capture a Correction (Operate) + +## Overview + +A human reviewing a forecast is the highest-signal feedback there is — they catch +structural errors and domain knowledge the backtest can't see, and catch them *now*. +This skill turns that into durable memory: a typed correction in `.fpa/corrections/` +that grounds every future forecast. + +**Core principle:** the human is the authority; capture, confirm interpretation once, +then it persists. Everything is plain markdown the user owns. + +## The three correction types + +- **parametric** — a concrete driver fix ("December runs ~2× a normal month"). Becomes + an `override` (a config path + value) applied to every future forecast via + `pyfpa.apply_corrections`. +- **structural** — a methodology fix ("you're double-counting deferred revenue"). A + *pre-ratified* structural proposal (the human authored it) — route it to + **fpa-learn-business** to generate the skill/model change; do NOT wait for backtest misses. +- **context** — a one-time-item note ("that Q3 spike was a one-off contract"). Annotates so + **fpa-cfo-judgment**'s one-time screen keeps the backtest from "learning" a one-off. + +## Workflow + +1. **Classify** the correction (parametric / structural / context). +2. **Identify the target** — the driver path (e.g. `channels[*].seasonality[11]`, + `working_capital.dio_days`), line, or profile area. For parametric, draft the concrete + `override: {path, value}`. +3. **Write** the correction file `.fpa/corrections/-.md` + (`pyfpa.save_correction`) with frontmatter (`type`, `target`, `status`, `date`, + `override`) and a markdown body (`**Was off:** … **Correction:** … **Why:** [[…]]`). + Link the body to the assumption/profile it corrects with `[[wikilinks]]`. +4. **Confirm interpretation.** Echo back the concrete change ("I'll set December + seasonality to 2.0 on all channels — right?"). Only on confirmation set + `status: applied`. +5. **Keep `.fpa/MEMORY.md` current** — the vault index (see below). + +## Applying corrections + +When building any forecast, `pyfpa.apply_corrections(cfg, load_corrections(".fpa/corrections"))` +folds the applied parametric corrections into the config. The per-client loop refines from +there — corrections are *seeds*, not mandates. + +## The `.fpa/` vault (`MEMORY.md` index) + +Keep a `.fpa/MEMORY.md` that orients a human, Obsidian, or Claude: +- `business-profile.md` — what we know about the business. +- `corrections/` — human corrections (this skill). +- `forecasts/*.snapshot.yaml`, `scorecard.md` — forecast snapshots + backtest track record. +- `learnings.md` — accepted model changes. +All plain markdown — open it in Obsidian if you like, but never required. + +## Guardrails + +- Confirm interpretation before `applied`. Reversible via `status` (`open`/`applied`/`superseded`). +- The backtest *monitors* applied corrections and may flag a stale one — it never reverts; + the human decides. + +## Next + +Correction captured → **fpa-monthly-close** / **fpa-board-briefing** (re-run grounded by it). +``` + +- [ ] **Step 2: Commit** + +```bash +git add skills/fpa-capture-correction/ +git commit -m "feat: fpa-capture-correction skill + .fpa vault (MEMORY.md) convention" +``` + +### Task 6: Wire corrections into the existing skills + +**Files:** +- Modify: `skills/fpa-learn-business/SKILL.md`, `skills/fpa-cfo-judgment/SKILL.md`, `skills/fpa-backtest-learn/SKILL.md` + +- [ ] **Step 1: fpa-learn-business** — in its workflow (after a model/config is built or refreshed), add a step: + `Apply any existing corrections: `pyfpa.apply_corrections(cfg, pyfpa.load_corrections(".fpa/corrections"))` before forecasting; route `type: structural` corrections through this skill's generation path as pre-ratified proposals (the human already authored them).` + Place it as a numbered step or a clearly-labeled paragraph consistent with the file's existing style (read the file first). + +- [ ] **Step 2: fpa-cfo-judgment** — add a row to its judgment table (the markdown table of gotchas): + `| **Known one-offs are flagged** | Read `.fpa/corrections/` for `type: context` notes (e.g. "Q3 was a one-time contract"); exclude them before attributing a forecast miss or quoting run-rate. |` + +- [ ] **Step 2b: fpa-backtest-learn** — in its workflow (the score/attribute step), add a bullet: + `Monitor applied corrections: if a `type: parametric` correction's target line keeps missing, flag it as possibly stale (`applied → superseded`) for the human — never auto-revert.` + +- [ ] **Step 3: Verify the skills still parse** — each SKILL.md must keep valid YAML frontmatter: + +Run: `python3 -c "import pathlib,yaml; [yaml.safe_load(p.read_text().split('---')[1]) for p in pathlib.Path('skills').glob('*/SKILL.md')]; print('ok')"` +Expected: `ok`. + +- [ ] **Step 4: Commit** + +```bash +git add skills/fpa-learn-business/SKILL.md skills/fpa-cfo-judgment/SKILL.md skills/fpa-backtest-learn/SKILL.md +git commit -m "feat: wire corrections into learn-business, cfo-judgment, backtest-learn" +``` + +--- + +## Group 6 — Docs + final + +### Task 7: README surfacing + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Add the skill to the skillset list** — in the "The skillset is the point" numbered list, add `fpa-capture-correction` to the **Operate** item with a gloss: "and **`fpa-capture-correction`** — turns a human's 'that's off because X' into durable memory that grounds future forecasts." + +- [ ] **Step 2: Add a roadmap row** — in the "Project status & roadmap" table add: + `| Human corrections + vault memory (\`pyfpa.memory\` + \`fpa-capture-correction\`) | ✅ Built |` + +- [ ] **Step 3: Commit** + +```bash +git add README.md +git commit -m "docs: surface human corrections + vault memory in the README" +``` + +### Task 8: Final verification + PR + +- [ ] **Step 1: Full suite** + +Run: `python3 -m pytest -q` +Expected: all green. + +- [ ] **Step 2: Open the PR** (stacked on `feat/backtest-learning-loop`) + +```bash +git push -u origin feat/memory-corrections +gh pr create --base feat/backtest-learning-loop --head feat/memory-corrections \ + --title "feat: human corrections + vault-friendly memory (pyfpa.memory + fpa-capture-correction)" \ + --body "Captures unprompted human corrections as typed, vault-native markdown in .fpa/corrections/. Parametric corrections apply to every future forecast via apply_corrections; structural become pre-ratified skill proposals; context annotates the one-time screen. Lean tested pyfpa.memory + fpa-capture-correction skill, wired into learn-business/cfo-judgment/backtest-learn. Obsidian-friendly, never required. Spec: docs/superpowers/specs/2026-06-08-memory-corrections-design.md. Stacked on the backtest loop." +``` + +Expected: PR opened. Jeff reviews/merges. + +--- + +## Self-Review notes + +- **Spec coverage:** Correction model + file format → Task 2. Three types + routing → Tasks 2,3,5,6. `apply_corrections` (applied+parametric only) → Task 3. Path grammar → Task 1. Capture skill + confirm-interpretation → Task 5. Vault `MEMORY.md` → Task 5. Integrations (learn-business apply+structural, cfo-judgment context, backtest monitor) → Task 6. README → Task 7. Loop-back (backtest flags, never reverts) → Task 6 step 2b. Guardrails → Tasks 3 (applied+parametric only), 5 (confirm), 6 (flag-not-revert). Testing list → Tasks 1,2,3. +- **Signature consistency:** `_set_by_path(data, path, value)`, `Override(path, value)`, `Correction(slug, type, target, status, date, override, notes)`, `load_corrections(dir)`, `save_correction(correction, dir)`, `apply_corrections(cfg, corrections)` — used identically across tasks. +- **Scope:** no Obsidian plugin, no auto-generation of corrections, no Loop B mining, no blanket retrofit, override grammar stays `[n]`/`[*]` + scalar — all honored. diff --git a/docs/superpowers/plans/2026-06-08-portfolio-learning.md b/docs/superpowers/plans/2026-06-08-portfolio-learning.md new file mode 100644 index 0000000..7607c8e --- /dev/null +++ b/docs/superpowers/plans/2026-06-08-portfolio-learning.md @@ -0,0 +1,957 @@ +# Portfolio Learning (Loop B) — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Cross-client learning for a fractional CFO's own (local) book — mine patterns that generalize across same-type clients, validate them with leave-one-out cross-client backtesting, and promote ratified priors/skills into a local library that seeds new clients. + +**Architecture:** A tested `pyfpa.portfolio` package owns the mechanics (manifest, snapshot mining, leave-one-out validation reusing Loop A's scorer, the local library). An `fpa-portfolio-learn` skill owns judgment (present candidates, operator ratifies); `fpa-learn-business` seeds new clients from the library. All local, no network; pure, immutable, typed, unit-tested. + +**Tech Stack:** Python 3.11+, pydantic v2, PyYAML, stdlib `statistics`/`copy` (existing deps). + +**Spec:** `docs/superpowers/specs/2026-06-08-portfolio-learning-design.md` + +--- + +## File Structure + +| Path | Responsibility | +| --- | --- | +| `pyfpa/memory/paths.py` (modify) | Add public `apply_override` (wraps `_set_by_path`). | +| `pyfpa/memory/__init__.py`, `pyfpa/__init__.py` (modify) | Export `apply_override`. | +| `pyfpa/portfolio/__init__.py` (create) | Re-export the package's public names. | +| `pyfpa/portfolio/manifest.py` (create) | `ClientRef`, `Portfolio`, `load_portfolio`, `clients_of_type`. | +| `pyfpa/portfolio/recover.py` (create) | `recover_actuals`, `best_snapshot`. | +| `pyfpa/portfolio/mine.py` (create) | `MINEABLE_DRIVERS`, `PriorCandidate`, `mine_priors`, `SkillCandidate`, `find_recurring_skills`. | +| `pyfpa/portfolio/validate.py` (create) | `ValidationResult`, `validate_prior` (leave-one-out). | +| `pyfpa/portfolio/library.py` (create) | `load_library`, `promote_prior`, `promote_skill`, `seed_from_library`. | +| `pyfpa/__init__.py` (modify) | Export the portfolio public symbols. | +| `skills/fpa-portfolio-learn/SKILL.md` (create) | The judgment + workflow skill. | +| `skills/fpa-learn-business/SKILL.md` (modify) | Seed new clients from the library. | +| `README.md` (modify) | The "how it compounds" pass + skillset/roadmap rows. | +| `tests/test_memory_paths.py` (modify) | Test the new `apply_override`. | +| `tests/test_portfolio_manifest.py`, `test_portfolio_recover.py`, `test_portfolio_mine.py`, `test_portfolio_validate.py`, `test_portfolio_library.py` (create) | Unit tests. | +| `tests/test_public_api.py` (modify) | Keep the `__all__` contract test in sync. | + +Conventions: immutability, small focused files, clear names, pydantic v2. Tests FLAT in `tests/`. Use `python3 -m pytest`. **All file paths the engine touches are caller-supplied; no network anywhere.** + +--- + +## Group 1 — Public `apply_override` (memory touch-up) + +### Task 1: promote `_set_by_path` to public `apply_override` + +**Files:** +- Modify: `pyfpa/memory/paths.py`, `pyfpa/memory/__init__.py`, `pyfpa/__init__.py`, `tests/test_memory_paths.py`, `tests/test_public_api.py` + +- [ ] **Step 1: Add the failing test** (append to `tests/test_memory_paths.py`): + +```python +from pyfpa.memory.paths import apply_override + + +def test_apply_override_is_public_set_by_path(): + data = {"working_capital": {"dio_days": 30.0}} + apply_override(data, "working_capital.dio_days", 45.0) + assert data["working_capital"]["dio_days"] == 45.0 +``` + +- [ ] **Step 2: Run, confirm fail** + +Run: `python3 -m pytest tests/test_memory_paths.py::test_apply_override_is_public_set_by_path -v` +Expected: FAIL (`ImportError`). + +- [ ] **Step 3: Implement** — append to `pyfpa/memory/paths.py`: + +```python +def apply_override(data: dict, path: str, value: float) -> None: + """Set ``value`` at dotted ``path`` (supports ``name``, ``name[n]``, ``name[*]``) + in ``data``, in place. Public wrapper over the internal path setter.""" + _set_by_path(data, path, value) +``` + +In `pyfpa/memory/__init__.py`, add `apply_override` to the import from `pyfpa.memory.paths` (add a line `from pyfpa.memory.paths import apply_override`) and to `__all__`. +In `pyfpa/__init__.py`, add `apply_override` to the `from pyfpa.memory import (...)` block and to `__all__`. +In `tests/test_public_api.py`, add `"apply_override"` to the expected `__all__` set (keep exact-set-equality). + +- [ ] **Step 4: Run tests** + +Run: `python3 -m pytest -q` +Expected: all pass (incl. test_public_api). + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/memory/ pyfpa/__init__.py tests/test_memory_paths.py tests/test_public_api.py +git commit -m "feat: public apply_override in pyfpa.memory" +``` + +--- + +## Group 2 — Portfolio manifest (`pyfpa/portfolio/manifest.py`) + +### Task 2: `ClientRef` + `Portfolio` + `load_portfolio` + `clients_of_type` + +**Files:** +- Create: `pyfpa/portfolio/__init__.py` (empty for now), `pyfpa/portfolio/manifest.py` +- Test: `tests/test_portfolio_manifest.py` + +- [ ] **Step 1: Failing test** + +```python +# tests/test_portfolio_manifest.py +from pyfpa.portfolio.manifest import ClientRef, Portfolio, load_portfolio, clients_of_type + + +def test_load_portfolio_and_filter(tmp_path): + (tmp_path / "p.yaml").write_text( + "library: ~/.fpa/library\n" + "clients:\n" + " - { path: ~/clients/acme, type: d2c-inventory }\n" + " - { path: ~/clients/haul, type: trucking }\n" + " - { path: ~/clients/peak, type: d2c-inventory }\n" + ) + pf = load_portfolio(tmp_path / "p.yaml") + assert pf.library == "~/.fpa/library" + assert len(pf.clients) == 3 + d2c = clients_of_type(pf, "d2c-inventory") + assert [c.path for c in d2c] == ["~/clients/acme", "~/clients/peak"] + assert all(isinstance(c, ClientRef) for c in d2c) +``` + +- [ ] **Step 2: Run, confirm fail** (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** — empty `pyfpa/portfolio/__init__.py`, then `pyfpa/portfolio/manifest.py`: + +```python +from __future__ import annotations + +from pathlib import Path + +import yaml +from pydantic import BaseModel + + +class ClientRef(BaseModel): + path: str # client workspace root (contains .fpa/ and skills/) + type: str # business-type tag (the clustering key) + + +class Portfolio(BaseModel): + library: str + clients: list[ClientRef] + + +def load_portfolio(path: str | Path) -> Portfolio: + p = Path(path) + if not p.exists(): + raise FileNotFoundError(f"portfolio manifest not found: {p}") + return Portfolio.model_validate(yaml.safe_load(p.read_text())) + + +def clients_of_type(portfolio: Portfolio, business_type: str) -> list[ClientRef]: + return [c for c in portfolio.clients if c.type == business_type] +``` + +- [ ] **Step 4: Run** `python3 -m pytest tests/test_portfolio_manifest.py -v` → PASS. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/portfolio/__init__.py pyfpa/portfolio/manifest.py tests/test_portfolio_manifest.py +git commit -m "feat: portfolio manifest (ClientRef, Portfolio, load, clients_of_type)" +``` + +--- + +## Group 3 — Recover actuals + best snapshot (`pyfpa/portfolio/recover.py`) + +### Task 3: `recover_actuals` + `best_snapshot` + +**Files:** +- Create: `pyfpa/portfolio/recover.py` +- Test: `tests/test_portfolio_recover.py` + +- [ ] **Step 1: Failing test** + +```python +# tests/test_portfolio_recover.py +import pytest +from pyfpa.backtest.snapshot import Snapshot, save_snapshot +from pyfpa.backtest.score import ScoreResult +from pyfpa.portfolio.recover import recover_actuals, best_snapshot + + +def _snap(label, fitness, predicted, per_line): + return Snapshot(label=label, created="2026-01-01", assumptions={}, predicted=predicted, + score=ScoreResult(fitness=fitness, per_line=per_line, weights={})) + + +def test_recover_actuals_inverts_error(): + # per_line error = predicted/actual - 1 => actual = predicted / (1 + error) + snap = _snap("p", 0.1, {"revenue": 110.0, "ebitda": 90.0}, {"revenue": 0.10, "ebitda": -0.10}) + act = recover_actuals(snap) + assert act["revenue"] == pytest.approx(100.0) + assert act["ebitda"] == pytest.approx(100.0) + + +def test_recover_actuals_no_score_is_empty(): + snap = Snapshot(label="p", created="2026-01-01", assumptions={}, predicted={"revenue": 1.0}) + assert recover_actuals(snap) == {} + + +def test_best_snapshot_picks_lowest_fitness(tmp_path): + forecasts = tmp_path / ".fpa" / "forecasts" + save_snapshot(_snap("2026-01", 0.20, {"revenue": 1.0}, {"revenue": 0.0}), forecasts / "a.snapshot.yaml") + save_snapshot(_snap("2026-02", 0.05, {"revenue": 1.0}, {"revenue": 0.0}), forecasts / "b.snapshot.yaml") + best = best_snapshot(tmp_path) + assert best.label == "2026-02" + + +def test_best_snapshot_none_when_unscored_or_missing(tmp_path): + assert best_snapshot(tmp_path / "nope") is None + forecasts = tmp_path / ".fpa" / "forecasts" + unscored = Snapshot(label="p", created="2026-01-01", assumptions={}, predicted={"revenue": 1.0}) + save_snapshot(unscored, forecasts / "u.snapshot.yaml") + assert best_snapshot(tmp_path) is None +``` + +Note: `save_snapshot` takes a file path; it does not create parent dirs. Create them in the test if needed — `save_snapshot` writes to the given path, so ensure `forecasts` exists first (add `forecasts.mkdir(parents=True, exist_ok=True)` before the saves in each test that uses it). + +- [ ] **Step 2: Run, confirm fail** (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** `pyfpa/portfolio/recover.py`: + +```python +from __future__ import annotations + +from pathlib import Path + +from pyfpa.backtest.snapshot import Snapshot, load_snapshot + + +def recover_actuals(snapshot: Snapshot) -> dict[str, float]: + """Recover the realized actuals from a scored snapshot by inverting the stored + per-line error: actual = predicted / (1 + error). Only scored lines are + recoverable; an unscored snapshot yields {}.""" + if snapshot.score is None: + return {} + out: dict[str, float] = {} + for line, error in snapshot.score.per_line.items(): + if line in snapshot.predicted: + out[line] = snapshot.predicted[line] / (1.0 + error) + return out + + +def best_snapshot(client_path: str | Path) -> Snapshot | None: + """The lowest-fitness scored snapshot in /.fpa/forecasts/ — the + assumptions that worked best for that client. None if no scored snapshot.""" + forecasts = Path(client_path) / ".fpa" / "forecasts" + if not forecasts.exists(): + return None + scored = [ + snap for f in sorted(forecasts.glob("*.snapshot.yaml")) + if (snap := load_snapshot(f)).score is not None + ] + return min(scored, key=lambda s: s.score.fitness) if scored else None +``` + +- [ ] **Step 4: Run** `python3 -m pytest tests/test_portfolio_recover.py -v` → PASS. Then full suite. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/portfolio/recover.py tests/test_portfolio_recover.py +git commit -m "feat: recover_actuals (invert stored score) + best_snapshot" +``` + +--- + +## Group 4 — Mining (`pyfpa/portfolio/mine.py`) + +### Task 4: `mine_priors` + `find_recurring_skills` + +**Files:** +- Create: `pyfpa/portfolio/mine.py` +- Test: `tests/test_portfolio_mine.py` + +- [ ] **Step 1: Failing test** + +```python +# tests/test_portfolio_mine.py +from pathlib import Path + +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.snapshot import snapshot_forecast, save_snapshot +from pyfpa.backtest.score import score_forecast +from pyfpa.portfolio.manifest import Portfolio, ClientRef +from pyfpa.portfolio.mine import mine_priors, find_recurring_skills + + +def _base_cfg(dio): + return EntityConfig.model_validate({ + "name": "c", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": dio}, + "opening_balances": {"cash": 0.0}, + }) + + +def _make_client(tmp_path, name, dio, *, gen_skills=()): + root = tmp_path / name + cfg = _base_cfg(dio) + fc = cashflow_from_config(cfg) + snap = snapshot_forecast(cfg, fc, label="2026", created="2026-01-01") + snap = snap.model_copy(update={"score": score_forecast(snap.predicted, snap.predicted)}) + (root / ".fpa" / "forecasts").mkdir(parents=True, exist_ok=True) + save_snapshot(snap, root / ".fpa" / "forecasts" / "2026.snapshot.yaml") + for s in gen_skills: + d = root / "skills" / "generated" / s + d.mkdir(parents=True, exist_ok=True) + (d / "SKILL.md").write_text(f"---\nname: {s}\ndescription: x\n---\n") + return ClientRef(path=str(root), type="d2c") + + +def _portfolio(clients): + return Portfolio(library="lib", clients=clients) + + +def test_mine_priors_tight_cluster(tmp_path): + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 44.0), ("b", 45.0), ("c", 46.0)]] + cands = mine_priors(_portfolio(clients), "d2c", min_support=3, dispersion_max=0.15) + dio = [c for c in cands if c.driver == "working_capital.dio_days"] + assert len(dio) == 1 + assert dio[0].value == 45.0 # median + assert len(dio[0].support) == 3 + + +def test_mine_priors_scattered_yields_nothing(tmp_path): + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 20.0), ("b", 50.0), ("c", 95.0)]] + cands = mine_priors(_portfolio(clients), "d2c", min_support=3, dispersion_max=0.15) + assert [c for c in cands if c.driver == "working_capital.dio_days"] == [] + + +def test_mine_priors_below_min_support(tmp_path): + clients = [_make_client(tmp_path, n, 45.0) for n in ("a", "b")] + assert mine_priors(_portfolio(clients), "d2c", min_support=3) == [] + + +def test_find_recurring_skills(tmp_path): + clients = [_make_client(tmp_path, n, 45.0, gen_skills=["arr-waterfall"]) for n in ("a", "b", "c")] + clients.append(_make_client(tmp_path, "d", 45.0, gen_skills=["one-off"])) + skills = find_recurring_skills(_portfolio(clients), "d2c", min_support=3) + names = [s.name for s in skills] + assert "arr-waterfall" in names + assert "one-off" not in names +``` + +- [ ] **Step 2: Run, confirm fail** (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** `pyfpa/portfolio/mine.py`: + +```python +from __future__ import annotations + +import statistics +from pathlib import Path + +from pydantic import BaseModel + +from pyfpa.memory.corrections import load_corrections +from pyfpa.portfolio.manifest import ClientRef, Portfolio, clients_of_type +from pyfpa.portfolio.recover import best_snapshot + +MINEABLE_DRIVERS = [ + "working_capital.dso_days", "working_capital.dio_days", "working_capital.dpo_days", + "tax_rate", "da_monthly", "capex_monthly", +] + + +class PriorCandidate(BaseModel): + business_type: str + driver: str + value: float + support: list[str] + dispersion: float + + +class SkillCandidate(BaseModel): + business_type: str + name: str + support: list[str] + source: str + + +def _get_by_path(data: dict, path: str) -> float | None: + node = data + for segment in path.split("."): + if not isinstance(node, dict) or segment not in node: + return None + node = node[segment] + return float(node) if isinstance(node, (int, float)) else None + + +def client_driver_value(client: ClientRef, driver: str) -> float | None: + """The best-snapshot value for `driver`, overridden by an applied parametric + correction on that exact path if one exists.""" + snap = best_snapshot(client.path) + if snap is None: + return None + for c in load_corrections(Path(client.path) / ".fpa" / "corrections"): + if c.status == "applied" and c.type == "parametric" and c.override and c.override.path == driver: + return c.override.value + return _get_by_path(snap.assumptions, driver) + + +def mine_priors(portfolio: Portfolio, business_type: str, *, + min_support: int = 3, dispersion_max: float = 0.15) -> list[PriorCandidate]: + """Drivers that cluster tightly (CoV <= dispersion_max) across >= min_support + same-type clients become prior candidates at the median.""" + clients = clients_of_type(portfolio, business_type) + out: list[PriorCandidate] = [] + for driver in MINEABLE_DRIVERS: + present = [(c, v) for c in clients if (v := client_driver_value(c, driver)) is not None] + if len(present) < min_support: + continue + values = [v for _, v in present] + mean = statistics.fmean(values) + cov = (statistics.pstdev(values) / abs(mean)) if mean else float("inf") + if cov <= dispersion_max: + out.append(PriorCandidate( + business_type=business_type, driver=driver, + value=statistics.median(values), support=[c.path for c, _ in present], + dispersion=cov, + )) + return out + + +def find_recurring_skills(portfolio: Portfolio, business_type: str, *, + min_support: int = 3) -> list[SkillCandidate]: + """Generated-skill directory names that recur across >= min_support same-type + clients. (Structural corrections are an additional human signal the operator + weighs in the skill, not a mechanical trigger here.)""" + clients = clients_of_type(portfolio, business_type) + by_name: dict[str, list[tuple[str, str]]] = {} + for c in clients: + generated = Path(c.path) / "skills" / "generated" + if not generated.exists(): + continue + for d in sorted(generated.iterdir()): + if (d / "SKILL.md").exists(): + by_name.setdefault(d.name, []).append((c.path, str(d))) + out: list[SkillCandidate] = [] + for name, refs in by_name.items(): + if len(refs) >= min_support: + out.append(SkillCandidate( + business_type=business_type, name=name, + support=[p for p, _ in refs], source=refs[0][1], + )) + return out +``` + +- [ ] **Step 4: Run** `python3 -m pytest tests/test_portfolio_mine.py -v` → PASS (all four). Then full suite. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/portfolio/mine.py tests/test_portfolio_mine.py +git commit -m "feat: mine_priors (clustering) + find_recurring_skills" +``` + +--- + +## Group 5 — Leave-one-out validation (`pyfpa/portfolio/validate.py`) + +### Task 5: `validate_prior` + +**Files:** +- Create: `pyfpa/portfolio/validate.py` +- Test: `tests/test_portfolio_validate.py` + +- [ ] **Step 1: Failing test** (reuses the `_make_client` helper pattern from Task 4 — copy it into this file): + +```python +# tests/test_portfolio_validate.py +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.snapshot import snapshot_forecast, save_snapshot +from pyfpa.backtest.score import score_forecast +from pyfpa.portfolio.manifest import ClientRef +from pyfpa.portfolio.validate import validate_prior + + +def _base_cfg(dio): + return EntityConfig.model_validate({ + "name": "c", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": dio}, + "opening_balances": {"cash": 0.0}, + }) + + +def _make_client(tmp_path, name, dio): + root = tmp_path / name + cfg = _base_cfg(dio) + snap = snapshot_forecast(cfg, cashflow_from_config(cfg), label="2026", created="2026-01-01") + snap = snap.model_copy(update={"score": score_forecast(snap.predicted, snap.predicted)}) + (root / ".fpa" / "forecasts").mkdir(parents=True, exist_ok=True) + save_snapshot(snap, root / ".fpa" / "forecasts" / "2026.snapshot.yaml") + return ClientRef(path=str(root), type="d2c") + + +def test_validate_tight_cluster_is_validated(tmp_path): + # all three clients share dio ≈ 45 → a peer-derived value ≈ each client's own → mean_delta ≈ 0 + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 44.0), ("b", 45.0), ("c", 46.0)]] + res = validate_prior("working_capital.dio_days", clients, tolerance=0.01) + assert res.n_folds == 3 + assert res.mean_delta <= 0.01 + assert res.validated is True + + +def test_validate_scattered_not_validated(tmp_path): + # wildly different dio → forcing a peer-median onto each degrades fit → positive mean_delta + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 10.0), ("b", 60.0), ("c", 120.0)]] + res = validate_prior("working_capital.dio_days", clients, tolerance=0.0) + assert res.validated is False + + +def test_validate_too_few_clients(tmp_path): + clients = [_make_client(tmp_path, "a", 45.0)] + res = validate_prior("working_capital.dio_days", clients) + assert res.n_folds == 1 + assert res.validated is False +``` + +- [ ] **Step 2: Run, confirm fail** (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** `pyfpa/portfolio/validate.py`: + +```python +from __future__ import annotations + +import copy +import statistics + +from pydantic import BaseModel + +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.score import DEFAULT_SCORE_LINES, extract_lines, score_forecast +from pyfpa.memory.paths import apply_override +from pyfpa.portfolio.manifest import ClientRef +from pyfpa.portfolio.mine import client_driver_value +from pyfpa.portfolio.recover import best_snapshot, recover_actuals + + +class ValidationResult(BaseModel): + mean_delta: float + n_folds: int + validated: bool + + +def validate_prior(driver: str, type_clients: list[ClientRef], *, tolerance: float = 0.0) -> ValidationResult: + """Leave-one-out cross-client check. For each usable client, derive `driver`'s + value as the median across the OTHER clients, apply it to the held-out client's + best-snapshot config, re-forecast, and score against that client's recovered + actuals. A prior is `validated` if the mean fitness delta (new - original) is + <= tolerance with >= 2 folds — a peer-derived value does not degrade held-out fit.""" + usable = [] + for c in type_clients: + snap = best_snapshot(c.path) + value = client_driver_value(c, driver) + if snap is not None and snap.score is not None and value is not None: + usable.append((value, snap)) + n = len(usable) + if n < 2: + return ValidationResult(mean_delta=0.0, n_folds=n, validated=False) + + deltas = [] + for i, (_, snap) in enumerate(usable): + peer_values = [usable[j][0] for j in range(n) if j != i] + prior_value = statistics.median(peer_values) + data = copy.deepcopy(snap.assumptions) + apply_override(data, driver, prior_value) + forecast = cashflow_from_config(EntityConfig.model_validate(data)) + predicted = extract_lines(forecast, DEFAULT_SCORE_LINES) + new_fitness = score_forecast(predicted, recover_actuals(snap)).fitness + deltas.append(new_fitness - snap.score.fitness) + + mean_delta = statistics.fmean(deltas) + return ValidationResult(mean_delta=mean_delta, n_folds=n, validated=mean_delta <= tolerance) +``` + +- [ ] **Step 4: Run** `python3 -m pytest tests/test_portfolio_validate.py -v` → PASS. The tight-cluster test proves the metric stays ≈0 for a robust prior; the scattered test proves it rejects a non-generalizing one. Then full suite. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/portfolio/validate.py tests/test_portfolio_validate.py +git commit -m "feat: validate_prior — leave-one-out cross-client validation" +``` + +--- + +## Group 6 — Local library (`pyfpa/portfolio/library.py`) + +### Task 6: `load_library` + `promote_prior` + `promote_skill` + `seed_from_library` + +**Files:** +- Create: `pyfpa/portfolio/library.py` +- Test: `tests/test_portfolio_library.py` + +- [ ] **Step 1: Failing test** + +```python +# tests/test_portfolio_library.py +from pyfpa.config.schemas import EntityConfig +from pyfpa.portfolio.mine import PriorCandidate, SkillCandidate +from pyfpa.portfolio.validate import ValidationResult +from pyfpa.portfolio.library import ( + load_library, promote_prior, promote_skill, seed_from_library, +) + + +def _cfg(dio=30.0): + return EntityConfig.model_validate({ + "name": "c", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_000_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": dio}, + "opening_balances": {"cash": 0.0}, + }) + + +def test_promote_prior_and_seed(tmp_path): + lib = tmp_path / "library" + cand = PriorCandidate(business_type="d2c", driver="working_capital.dio_days", + value=45.0, support=["a", "b", "c"], dispersion=0.02) + promote_prior(lib, cand, ValidationResult(mean_delta=-0.01, n_folds=3, validated=True)) + # the log records it + assert "working_capital.dio_days" in (lib / "library-log.md").read_text() + # a new client's config gets the prior as its starting value + seeded = seed_from_library(lib, "d2c", _cfg(dio=30.0)) + assert seeded.working_capital.dio_days == 45.0 + assert _cfg(dio=30.0).working_capital.dio_days == 30.0 # base unmutated + + +def test_seed_unknown_type_is_noop(tmp_path): + lib = tmp_path / "library" + out = seed_from_library(lib, "saas", _cfg(dio=30.0)) + assert out.working_capital.dio_days == 30.0 + + +def test_load_library_round_trip(tmp_path): + lib = tmp_path / "library" + cand = PriorCandidate(business_type="d2c", driver="tax_rate", value=0.25, + support=["a", "b", "c"], dispersion=0.01) + promote_prior(lib, cand, ValidationResult(mean_delta=0.0, n_folds=3, validated=True)) + loaded = load_library(lib) + assert any(p["driver"] == "tax_rate" and p["value"] == 0.25 for p in loaded["priors"]["d2c"]) + + +def test_promote_skill_copies_and_logs(tmp_path): + src = tmp_path / "src" / "arr-waterfall" + src.mkdir(parents=True) + (src / "SKILL.md").write_text("---\nname: arr-waterfall\ndescription: x\n---\n") + lib = tmp_path / "library" + promote_skill(lib, SkillCandidate(business_type="saas", name="arr-waterfall", + support=["a", "b", "c"], source=str(src))) + assert (lib / "skills" / "arr-waterfall" / "SKILL.md").exists() + assert "arr-waterfall" in (lib / "library-log.md").read_text() +``` + +- [ ] **Step 2: Run, confirm fail** (`ModuleNotFoundError`). + +- [ ] **Step 3: Implement** `pyfpa/portfolio/library.py`: + +```python +from __future__ import annotations + +import shutil +from pathlib import Path + +import yaml + +from pyfpa.config.schemas import EntityConfig +from pyfpa.memory.paths import apply_override +from pyfpa.portfolio.mine import PriorCandidate, SkillCandidate +from pyfpa.portfolio.validate import ValidationResult + + +def _log(library: Path, line: str) -> None: + library.mkdir(parents=True, exist_ok=True) + log = library / "library-log.md" + header = "" if log.exists() else "# Library Log\n\n" + with log.open("a") as f: + f.write(header + line + "\n") + + +def load_library(library: str | Path) -> dict: + """Read the library: {'priors': {type: [prior dicts]}, 'skills': [names]}.""" + library = Path(library) + priors: dict[str, list[dict]] = {} + priors_dir = library / "priors" + if priors_dir.exists(): + for f in sorted(priors_dir.glob("*.yaml")): + doc = yaml.safe_load(f.read_text()) or {} + priors[doc.get("type", f.stem)] = doc.get("priors", []) + skills = sorted(p.name for p in (library / "skills").glob("*")) if (library / "skills").exists() else [] + return {"priors": priors, "skills": skills} + + +def promote_prior(library: str | Path, candidate: PriorCandidate, validation: ValidationResult) -> None: + """Append a ratified prior to priors/.yaml and log it.""" + library = Path(library) + priors_dir = library / "priors" + priors_dir.mkdir(parents=True, exist_ok=True) + path = priors_dir / f"{candidate.business_type}.yaml" + doc = yaml.safe_load(path.read_text()) if path.exists() else None + doc = doc or {"type": candidate.business_type, "priors": []} + doc["priors"].append({ + "driver": candidate.driver, "value": candidate.value, "support": candidate.support, + "cross_client_holdout_delta": validation.mean_delta, "n_folds": validation.n_folds, + }) + path.write_text(yaml.safe_dump(doc, sort_keys=False)) + _log(library, f"- prior `{candidate.driver}` = {candidate.value} for {candidate.business_type} " + f"(support {len(candidate.support)}, Δ {validation.mean_delta:+.4f})") + + +def promote_skill(library: str | Path, candidate: SkillCandidate) -> None: + """Copy a recurring generated skill into the library and log it.""" + library = Path(library) + dest = library / "skills" / candidate.name + dest.mkdir(parents=True, exist_ok=True) + shutil.copytree(candidate.source, dest, dirs_exist_ok=True) + _log(library, f"- skill `{candidate.name}` for {candidate.business_type} " + f"(support {len(candidate.support)})") + + +def seed_from_library(library: str | Path, business_type: str, cfg: EntityConfig) -> EntityConfig: + """Apply the library's promoted priors for `business_type` to a starting config. + Returns a NEW config (input unmutated). Priors are seeds; the per-client loop refines.""" + data = cfg.model_dump() + for prior in load_library(library)["priors"].get(business_type, []): + apply_override(data, prior["driver"], prior["value"]) + return EntityConfig.model_validate(data) +``` + +- [ ] **Step 4: Run** `python3 -m pytest tests/test_portfolio_library.py -v` → PASS. Then full suite. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/portfolio/library.py tests/test_portfolio_library.py +git commit -m "feat: local library — promote priors/skills, seed new clients" +``` + +--- + +## Group 7 — Package wiring + +### Task 7: Re-export + public API + +**Files:** +- Modify: `pyfpa/portfolio/__init__.py`, `pyfpa/__init__.py`, `tests/test_public_api.py` + +- [ ] **Step 1: Fill `pyfpa/portfolio/__init__.py`** + +```python +from pyfpa.portfolio.manifest import ClientRef, Portfolio, load_portfolio, clients_of_type +from pyfpa.portfolio.recover import recover_actuals, best_snapshot +from pyfpa.portfolio.mine import ( + MINEABLE_DRIVERS, PriorCandidate, SkillCandidate, mine_priors, find_recurring_skills, +) +from pyfpa.portfolio.validate import ValidationResult, validate_prior +from pyfpa.portfolio.library import ( + load_library, promote_prior, promote_skill, seed_from_library, +) + +__all__ = [ + "ClientRef", "Portfolio", "load_portfolio", "clients_of_type", + "recover_actuals", "best_snapshot", "MINEABLE_DRIVERS", "PriorCandidate", + "SkillCandidate", "mine_priors", "find_recurring_skills", "ValidationResult", + "validate_prior", "load_library", "promote_prior", "promote_skill", "seed_from_library", +] +``` + +- [ ] **Step 2: Export the headline names from `pyfpa/__init__.py`** — after the `from pyfpa.memory import (...)` block add: + +```python +from pyfpa.portfolio import ( + Portfolio, load_portfolio, mine_priors, find_recurring_skills, + validate_prior, promote_prior, promote_skill, seed_from_library, +) +``` + +And extend `pyfpa/__init__.py`'s `__all__` with those eight names. + +- [ ] **Step 3: Sync `tests/test_public_api.py`** — add the same eight names to the exact-set assertion. Read it first; keep exact-set-equality. + +- [ ] **Step 4: Verify + full suite** + +Run: `python3 -c "import pyfpa; print(pyfpa.mine_priors, pyfpa.validate_prior, pyfpa.seed_from_library)"` +Expected: prints the three. +Run: `python3 -m pytest -q` +Expected: all pass. + +- [ ] **Step 5: Commit** + +```bash +git add pyfpa/portfolio/__init__.py pyfpa/__init__.py tests/test_public_api.py +git commit -m "feat: export pyfpa.portfolio public API" +``` + +--- + +## Group 8 — Skill + integration + +### Task 8: `fpa-portfolio-learn` skill + `fpa-learn-business` seeding + +**Files:** +- Create: `skills/fpa-portfolio-learn/SKILL.md` +- Modify: `skills/fpa-learn-business/SKILL.md` + +- [ ] **Step 1: Write `skills/fpa-portfolio-learn/SKILL.md`**: + +```markdown +--- +name: fpa-portfolio-learn +description: Use when you run FP&A for several clients and want your practice to compound — mines patterns that generalize across your same-type clients, validates them by leave-one-out cross-client backtesting, and promotes ratified priors and skills into a local library that seeds every new client. All local; nothing leaves your machine. +--- + +# Portfolio Learn (Loop B) + +## Overview + +Loop A makes the model better at one client. This makes your *practice* compound: +client #10 starts smarter than client #1 because your library carries what generalized +across #1–9. Everything is local — your own book, on your own machine. + +**Core principle:** self-improving, never self-ratifying — propose, you accept. The +objective metric is cross-client: does a pattern learned on some clients fail to +degrade the *others*' backtest? + +## Setup + +A portfolio manifest `~/.fpa/portfolio.yaml` lists your clients + a business-type tag: +\```yaml +library: ~/.fpa/library +clients: + - { path: ~/clients/acme, type: d2c-inventory } + - { path: ~/clients/peak, type: d2c-inventory } + - { path: ~/clients/haul, type: trucking } +\``` + +## Workflow + +1. **Load** the manifest (`pyfpa.load_portfolio`). +2. For each business-type with ≥ 3 clients: + - **Priors:** `pyfpa.mine_priors(portfolio, type)` → drivers that cluster tightly. + Validate each with `pyfpa.validate_prior(driver, clients_of_type)` (leave-one-out). + Surface validated ones first (mean cross-client delta), then unvalidated/judgment. + - **Skills:** `pyfpa.find_recurring_skills(portfolio, type)` for recurring generated + skills. Also weigh recurring **structural corrections** across clients (read each + `.fpa/corrections/` for `type: structural`) — a human-authored pattern that repeats + is strong signal. +3. **Present** candidates ranked by evidence (support count + cross-client delta). +4. **Ratify.** On your acceptance, `pyfpa.promote_prior` / `pyfpa.promote_skill` writes the + `~/.fpa/library/` and `library-log.md`. Reversible. + +## Guardrails + +- Local-only; nothing phones home. +- ≥ 3 clients to propose; tight-cluster only; a prior must not degrade held-out clients. +- You ratify everything; priors are *seeds*, not mandates — each client's Loop A refines. + +## The payoff + +New clients inherit the library: **fpa-learn-business** seeds their starting model from +your promoted priors (`pyfpa.seed_from_library`) and offers the promoted skills. + +## Next + +Promoted → next new client onboarded via **fpa-learn-business** starts smarter. +``` + +(Note: in the file, the triple-backtick yaml block above should use normal triple backticks — the `\``` escaping here is only for this plan document.) + +- [ ] **Step 2: Wire seeding into `skills/fpa-learn-business/SKILL.md`** — in its workflow (when scaffolding a new client's model), add a step: + "**Seed from your library.** If a portfolio library exists, start the model from what generalized across your same-type clients: `cfg = pyfpa.seed_from_library(library_path, business_type, cfg)` (priors are seeds — this client's loop refines them). Offer any promoted skills for the type." + Place it consistent with the file's numbered-workflow style (read the file first). + +- [ ] **Step 3: Verify frontmatter** + +Run: `python3 -c "import pathlib,yaml; [yaml.safe_load(p.read_text().split('---')[1]) for p in pathlib.Path('skills').glob('*/SKILL.md')]; print('ok')"` +Expected: `ok`. + +- [ ] **Step 4: Commit** + +```bash +git add skills/fpa-portfolio-learn/ skills/fpa-learn-business/SKILL.md +git commit -m "feat: fpa-portfolio-learn skill + library seeding in learn-business" +``` + +--- + +## Group 9 — README pass + final + +### Task 9: README "how it compounds" pass + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Add `fpa-portfolio-learn` to the skillset list** — in the "The skillset is the point" numbered list **Operate** item, add `fpa-portfolio-learn` with a gloss: "and **`fpa-portfolio-learn`** — distills what generalizes across your whole book into a reusable library that seeds new clients (cross-client learning)." + +- [ ] **Step 2: Add a roadmap row** — after the corrections row: + `| Cross-client portfolio learning (\`pyfpa.portfolio\` + \`fpa-portfolio-learn\`) | ✅ Built |` + +- [ ] **Step 3: Add a short "How it compounds" subsection** right after the "Why not just point Claude at your books?" section. Use this content: + +```markdown +## How it compounds + +openfpa isn't static — it gets better the more you use it, on two loops, both with your data staying in your repo: + +- **Per client (Loop A).** Every close, it scores its last forecast against your actuals and proposes ratified tweaks. The model gets better at *this* business. Human corrections feed the same memory — the highest-signal fixes, captured durably. +- **Across your book (Loop B).** For a fractional CFO with many clients, it distills what *generalizes* — validated by leave-one-out cross-client backtesting — into a local library that seeds every new client. Client #10 starts smarter than client #1. + +It's [Karpathy's AutoResearch](https://github.com/karpathy/autoresearch) idea applied to FP&A: an objective fitness metric (reconciliation error on your books) drives measurable self-improvement — at the client level *and* the portfolio level. +``` + +- [ ] **Step 4: Commit** + +```bash +git add README.md +git commit -m "docs: 'how it compounds' — the two-loop self-improvement story" +``` + +### Task 10: Final verification + PR + +- [ ] **Step 1: Full suite** + +Run: `python3 -m pytest -q` +Expected: all green. + +- [ ] **Step 2: Open the PR** (stacked on `feat/memory-corrections`) + +```bash +git push -u origin feat/portfolio-learning +gh pr create --base feat/memory-corrections --head feat/portfolio-learning \ + --title "feat: cross-client portfolio learning (pyfpa.portfolio + fpa-portfolio-learn)" \ + --body "Loop B: a fractional CFO's practice compounds. Mines drivers that cluster across same-type clients (from Loop A snapshots + applied corrections), validates each with leave-one-out cross-client backtesting (reuses score_forecast — a prior must not degrade held-out clients), and promotes ratified priors/skills into a local ~/.fpa/library/ that seeds new clients via fpa-learn-business. All local, operator-ratified. Lean tested pyfpa.portfolio + fpa-portfolio-learn skill + the 'how it compounds' README. Spec: docs/superpowers/specs/2026-06-08-portfolio-learning-design.md. Stacked on the memory layer." +``` + +Expected: PR opened. Jeff reviews/merges. + +--- + +## Self-Review notes + +- **Spec coverage:** apply_override touch-up → Task 1. manifest → Task 2. recover_actuals/best_snapshot → Task 3. mine_priors/find_recurring_skills (+ MINEABLE_DRIVERS, dispersion gate, min_support) → Task 4. validate_prior leave-one-out → Task 5. library (load/promote_prior/promote_skill/seed_from_library) → Task 6. wiring → Task 7. fpa-portfolio-learn skill + learn-business seeding → Task 8. README "how it compounds" → Task 9. Guardrails (local-only, min_support, dispersion, held-out validation, ratify, seeds-not-mandates) → Tasks 4,5,6,8. Privacy (no network) → all tasks operate on caller paths. +- **Signature consistency:** `ClientRef(path,type)`, `Portfolio(library,clients)`, `clients_of_type`, `recover_actuals(snapshot)`, `best_snapshot(client_path)`, `client_driver_value(client,driver)`, `mine_priors(portfolio,type,*,min_support,dispersion_max)`, `PriorCandidate(business_type,driver,value,support,dispersion)`, `SkillCandidate(business_type,name,support,source)`, `find_recurring_skills(portfolio,type,*,min_support)`, `ValidationResult(mean_delta,n_folds,validated)`, `validate_prior(driver,type_clients,*,tolerance)`, `load_library`, `promote_prior(library,candidate,validation)`, `promote_skill(library,candidate)`, `seed_from_library(library,type,cfg)` — used identically across tasks. +- **Scope:** local-only (no central/community aggregation), entity-level mineable drivers (per-channel deferred), operator-ratified only, no new ingestion — all honored. diff --git a/docs/superpowers/specs/2026-06-08-backtest-learning-loop-design.md b/docs/superpowers/specs/2026-06-08-backtest-learning-loop-design.md new file mode 100644 index 0000000..4b2e21b --- /dev/null +++ b/docs/superpowers/specs/2026-06-08-backtest-learning-loop-design.md @@ -0,0 +1,128 @@ +# Self-Improving Backtest Loop — Design + +**Status:** Approved (design phase) +**Date:** 2026-06-08 +**Topic:** A per-client loop that scores past forecasts against the user's actuals +and proposes ratified improvements — the self-improvement engine behind openfpa's +mission ("it sharpens itself against your actuals every close"). + +## Why + +openfpa's differentiator is that the model **gets measurably better at your +business over time**. Karpathy's AutoResearch self-improves only because it has a +cheap, objective fitness function (validation loss). FP&A's equivalent is +**reconciliation error against the user's own actuals** — exactly the metric the +Fox Factory work proved out (`pyfpa.analysis.reconcile`). Today, forecasts are +ephemeral: nothing persists a forecast, scores it once the period closes, or +remembers which assumptions keep missing. This builds that loop. + +## Decisions (locked) + +| Decision | Choice | +| --- | --- | +| Autonomy | **Score + propose; human ratifies.** No autonomous apply. | +| Bootstrap | **Holdout backtest on existing history** (immediate value) + forward accumulation. | +| Learning scope | Both **parametric** (assumption tuning, auto-scored) and **structural** (methodology/skill proposals, human-judged) — one evidence trail. | +| Architecture | **Hybrid:** lean tested `pyfpa.backtest` (objective scoring) + `fpa-backtest-learn` skill (judgment). | +| Memory | **Transparent files in `.fpa/`** — the user owns and can audit/edit them. | +| Fitness metric | **Weighted MAPE** across decision-critical lines (ending cash, EBITDA, revenue, gross margin). | + +## Memory layout (`.fpa/`) + +``` +.fpa/ + business-profile.md # (existing) + forecasts/ + 2026-01.snapshot.yaml # assumptions + predicted lines, and (after close) the score + 2026-02.snapshot.yaml + scorecard.md # rendered running track record: fitness + per-line error, per period + learnings.md # accepted changes: what, evidence, backtest delta, date (reversible) +``` + +A **snapshot** is the full record of one forecast: the assumptions (config drivers) +it used, the lines it predicted, and — added once the period closes — the realized +`score`. `scorecard.md` is rendered from the scored snapshots (single source of +truth per period; no duplicate store). `learnings.md` is the audit log of every +ratified change. + +## Components + +### Engine — `pyfpa/backtest/` (lean, pure, tested) + +`pyfpa/backtest/snapshot.py` +- `class Snapshot(BaseModel)`: `label: str`, `created: str`, `assumptions: dict` + (the serialized `EntityConfig`), `predicted: dict[str, float]` (scored lines for + the forecast period), `score: ScoreResult | None = None`. +- `snapshot_forecast(cfg: EntityConfig, forecast_df: pd.DataFrame, *, label: str, created: str, score_lines: list[str]) -> Snapshot` — pure; captures assumptions + the annual/aggregate predicted values for `score_lines`. +- `save_snapshot(snapshot, path)` / `load_snapshot(path) -> Snapshot` — YAML round-trip. + (`created` is passed in, never generated — keeps the engine deterministic.) + +`pyfpa/backtest/score.py` +- `class ScoreResult(BaseModel)`: `fitness: float` (weighted MAPE; lower better), + `per_line: dict[str, float]` (signed error %, `predicted/actual - 1`), `weights: dict[str, float]`. +- `DEFAULT_WEIGHTS = {"ending_cash": 0.4, "ebitda": 0.3, "revenue": 0.2, "gross_margin": 0.1}`. +- `score_forecast(predicted: Mapping[str, float], actual: Mapping[str, float], *, weights=DEFAULT_WEIGHTS) -> ScoreResult` — built on `reconcile`; `fitness = Σ wᵢ·|predictedᵢ/actualᵢ − 1|` over lines present in both (weights renormalized over present lines). + +`pyfpa/backtest/holdout.py` +- `holdout_backtest(actuals_by_period: dict[str, dict[str, float]], build_cfg_fn: Callable[[dict[str, dict[str, float]]], EntityConfig], *, holdout: int, score_lines: list[str], weights=DEFAULT_WEIGHTS) -> ScoreResult` — split periods into fit (all but last `holdout`) and holdout (last `holdout`); `build_cfg_fn(fit_actuals)` → config; `cashflow_from_config` → predicted; score predicted vs the held-out actuals. The engine owns the split/score harness; the business-specific `build_cfg_fn` (actuals → assumptions) is supplied by the caller, never hardcoded. + +`pyfpa/backtest/__init__.py` re-exports the public names; `pyfpa/__init__.py` exposes them. + +### Skill — `skills/fpa-backtest-learn/SKILL.md` (judgment; hooks into `fpa-monthly-close`) + +Responsibilities (methodology, not code): +1. **Snapshot** every forecast it produces into `.fpa/forecasts/`. +2. **Score at close** — when a period closes (actuals via `fpa-configure-actuals`), load that period's snapshot, `score_forecast`, write the `score` back, re-render `scorecard.md`. +3. **Attribute** each material per-line miss to a driver (volume / price / cost ratio / WC timing), running the `fpa-cfo-judgment` one-time-item screen first. +4. **Propose**, tagged by type: + - *Parametric* — a concrete assumption change, each re-scored via `holdout_backtest` and shown with its **holdout** fitness delta; only surfaced if it improves out-of-sample. + - *Structural* — only when a per-line miss is persistent and same-signed across **K closes** and survives the one-time screen; proposed as a methodology/skill change (e.g. a revenue-recognition lag), handed to the existing `fpa-learn-business` skill-generation path on approval. +5. **Ratify + log** — the human accepts/rejects; accepted changes update the config and append to `learnings.md` with evidence, the backtest delta, and the date. + +## Data flow + +``` +forecast ──▶ snapshot_forecast() ──▶ .fpa/forecasts/.snapshot.yaml +period closes (actuals) ──▶ score_forecast() ──▶ snapshot.score ──▶ scorecard.md + │ + attribute miss (after one-time screen) + ▼ + parametric proposal ──(re-score via holdout_backtest)──▶ holdout delta + structural proposal ──(persistent K-close signed miss)──▶ surfaced + ▼ + human ratifies ──▶ config update + learnings.md ──▶ next forecast better +``` + +## Guardrails (anti-overfitting / trust) + +1. **Holdout separation** — `holdout_backtest` never scores on fit data. +2. **Persistence threshold `K`** (default 2) — no proposal off a single period; a miss must repeat same-signed across `K` closes. +3. **One-time-item screen** — `fpa-cfo-judgment` check before attributing a miss to the model. +4. **Improve-out-of-sample gate** — a parametric change must lower **holdout** fitness, not in-sample. +5. **Magnitude cap** — limit how far any single assumption moves per cycle (default ±25% relative) to prevent overcorrection. +6. **Human ratify + audit** — nothing is applied without acceptance; every change is logged in `learnings.md` and is reversible. + +## Success criteria + +1. `holdout_backtest` returns a fitness number from `≥ holdout + 1` periods of actuals, reusing `reconcile`. +2. Snapshots round-trip (save/load); scoring writes the `score` block back; `scorecard.md` renders from scored snapshots. +3. Parametric proposals appear **only** when they improve holdout fitness, ranked by delta. +4. Structural proposals appear **only** on a persistent same-signed miss that passes the one-time screen. +5. Every accepted change is logged in `learnings.md` with evidence + date and is reversible. +6. All `.fpa/` artifacts are plain, human-readable files — no hidden state. +7. `pyfpa.backtest` is unit-tested (repo's 80% norm); full suite green. + +## Testing + +- **snapshot** round-trip preserves assumptions + predictions. +- **score_forecast**: known predicted/actual → known weighted MAPE; absent lines renormalize weights. +- **holdout_backtest**: synthetic series with a *known* best assumption — the harness must score the correct config better than a wrong one (proves the metric discriminates). +- **guardrails**: single-period miss → no proposal; persistent same-signed miss → proposal; a change that improves in-sample but not holdout → rejected; assumption move beyond the cap → clamped. +- **scenario**: a synthetic series with a built-in revenue lag → backtest surfaces a persistent signed miss (the structural-proposal trigger). + +## Scope boundaries (YAGNI) + +- **No autonomous search** (the variant-search version of AutoResearch) — human-ratified only; the harness is built so search can be added later. +- **No cross-client pattern library** (promoting learnings into shared industry skills) — that's a separate future subsystem with its own review bar. +- **No new ingestion or output formats** — uses the existing `fpa-configure-actuals` path and `.fpa/` files. +- **Structural proposals are surfaced, not auto-written** — consistent with "self-extending, not self-executing": the loop proposes; skill generation happens on approval via the existing `fpa-learn-business` path. diff --git a/docs/superpowers/specs/2026-06-08-memory-corrections-design.md b/docs/superpowers/specs/2026-06-08-memory-corrections-design.md new file mode 100644 index 0000000..0c1555e --- /dev/null +++ b/docs/superpowers/specs/2026-06-08-memory-corrections-design.md @@ -0,0 +1,162 @@ +# Human Corrections & Vault-Friendly Memory — Design + +**Status:** Approved (design phase) +**Date:** 2026-06-08 +**Topic:** Capture unprompted human corrections as durable, structured memory that +grounds future forecasts — and make the `.fpa/` memory a coherent, Obsidian-friendly +vault. The highest-signal feedback channel, and the foundation Loop B mines. + +## Why + +The system learns from two signals today: **actuals** (the backtest — objective but +*lagging*) and **accept/reject** of its own proposals. It does *not* capture the +richest signal: a human reviewing a forecast who catches something off +("December always spikes," "you're double-counting deferred revenue"). That is +*leading* signal carrying domain knowledge the backtest can't see, and it's +currently lost. Memory is the core of this product; this makes human judgment a +first-class, durable, compounding part of it. + +## Decisions (locked) + +| Decision | Choice | +| --- | --- | +| Capture | **Both, skill-primary** — a conversational skill structures the correction; the file is plain markdown also hand-editable. | +| Correction types | **parametric** (driver override), **structural** (methodology/skill), **context** (one-time-item note). | +| Storage | **Vault-native markdown** in `.fpa/corrections/`, one file per correction (frontmatter + body). | +| Application | A tested `apply_corrections(cfg, …)` applies only `applied` + `parametric` overrides; structural/context are routed, not applied. | +| Authority | Human corrections apply on human authority (interpretation confirmed once at capture); the backtest **monitors** but never auto-reverts. | +| Module home | New `pyfpa/memory/` package (memory earns its own home). | +| Privacy | All local markdown the user owns; Obsidian is an optional UI, **never required**. | + +## Correction file format (vault-native) + +`.fpa/corrections/2026-06-08-december-seasonality.md`: +```markdown +--- +type: parametric +target: channels[*].seasonality[11] +status: applied +date: 2026-06-08 +override: { path: "channels[*].seasonality[11]", value: 2.0 } +--- +**Was off:** the model spread revenue evenly; our December runs ~2× a normal month. + +**Correction:** December seasonality weight ≈ 2.0. + +**Why:** [[business-profile#seasonality]] — Nov/Dec is ~35% of annual, every year. +``` +Frontmatter is the machine-readable contract (`type`, `target`, `status`, `date`, +`override`); the body is human narrative (`[[wikilinks]]` to the assumption/profile +it corrects). The engine reads frontmatter only — it never parses prose. + +## Components + +### Engine — `pyfpa/memory/` (lean, pure, tested) + +`pyfpa/memory/corrections.py` +- `class Override(BaseModel)`: `path: str`, `value: float`. +- `class Correction(BaseModel)`: `slug: str`, `type: Literal["parametric","structural","context"]`, + `target: str`, `status: Literal["open","applied","superseded"] = "open"`, `date: str`, + `override: Override | None = None`, `notes: str = ""` (the markdown body). +- `load_corrections(dir: str | Path) -> list[Correction]` — parse every `*.md` in the dir + (frontmatter → fields, body → `notes`); slug = filename stem. Missing dir → `[]`. +- `save_correction(correction, dir)` — write `.md` (frontmatter + body). +- `apply_corrections(cfg: EntityConfig, corrections) -> EntityConfig` — for each correction + with `status == "applied"` and `type == "parametric"` and an `override`, apply the override + to `cfg.model_dump()` and re-validate. Returns a **new** config; input never mutated. + Structural/context corrections are ignored here. + +`pyfpa/memory/paths.py` +- `_set_by_path(data: dict, path: str, value: float) -> None` (internal, used by apply): + dotted-path setter supporting `name`, `name[n]` (index), and `name[*]` (every list item). + An unresolvable path raises a descriptive `ValueError`. + +`pyfpa/memory/__init__.py` re-exports; `pyfpa/__init__.py` exposes the public names. + +### Skill — `skills/fpa-capture-correction/SKILL.md` + +Conversational capture: the human says "this is off because X"; the skill (1) classifies +the type, (2) identifies the `target`, (3) for parametric drafts the concrete `override`, +(4) writes the correction file, and (5) **echoes its interpretation back and waits for +confirmation** before setting `status: applied`. The file remains hand-editable. + +### Integrations (existing skills) + +- **fpa-learn-business** — when building/refreshing a client model, `apply_corrections` + the parametric ones; route `structural` corrections to its skill-generation path as + *pre-ratified* proposals (the human authored them — no waiting for K backtest misses). +- **fpa-cfo-judgment** — its one-time-item screen reads `context` corrections so the + backtest's attribution skips known one-offs. +- **fpa-monthly-close / fpa-backtest-learn** — *monitor* applied parametric corrections: + if a corrected line keeps missing, flag the correction as possibly stale (human decides; + `applied → superseded`). Never auto-revert. + +## Vault-friendliness + +- Corrections are fully vault-native (frontmatter + `[[wikilinks]]`). +- Existing memory (`business-profile.md`, `learnings.md`, `scorecard.md`, snapshots) gains + light frontmatter + cross-links on the high-value connections (correction ↔ assumption ↔ + profile) — not a blanket retrofit. +- A `.fpa/MEMORY.md` index documents the vault layout so a human, Obsidian, or Claude can + navigate it. The capture/learn skills keep it current. +- Result: `.fpa/` is one navigable knowledge base — Obsidian-openable, hand-editable, + Claude-readable. Obsidian is never required. + +## The loop-back + +Corrections apply on human authority, but Loop A's backtest monitors them: a parametric +correction whose target line keeps missing gets flagged (`"December set to 2×; last two +came in ~1.4× — revisit?"`). Human judgment leads; the backtest verifies; nothing reverts +silently. This completes the feedback story (leading + lagging signals reinforce). + +## Feeds Loop B + +Because corrections are typed and structured (frontmatter), Loop B mines them like +snapshots: a parametric correction recurring across the operator's clients of a type → +portfolio prior candidate; a recurring structural correction → portfolio skill candidate. +This spec only guarantees the **minable format**; Loop B consumes it. + +## Guardrails + +- Human authority, but interpretation **confirmed once** at capture before `applied`. +- **Reversible**: `status` + the file; superseding is explicit; nothing silently overwritten. +- **Transparent**: plain markdown the user owns; no hidden state; any editor works. +- `apply_corrections` is deterministic and acts **only** on `applied` + `parametric`. +- Backtest **flags, never reverts** a human correction. +- Obsidian optional — no hard dependency. + +## Success criteria + +1. A correction round-trips through `save_correction`/`load_corrections` preserving all + frontmatter fields and the body. +2. `apply_corrections` changes exactly the targeted driver(s) in a **new** config; input + unmutated; `*`-for-all-channels resolves; a bad path raises a clear `ValueError`. +3. Structural/context corrections are **not** applied to the config. +4. The three correction types are documented and routed (skill); `.fpa/MEMORY.md` exists + and describes the vault. +5. `pyfpa.memory` is unit-tested (repo's 80% norm); full suite green. +6. Everything is plain markdown the user owns; no Obsidian dependency. + +## Testing + +- **round-trip**: parametric correction with an override and a `[[wikilink]]` body → + save → load → identical fields + notes. +- **apply_corrections**: a `channels[*].seasonality[11] = 2.0` override sets month-12 + weight on every channel in a new cfg; `working_capital.dio_days = 45` sets that scalar; + input cfg unchanged. +- **path grammar**: `name`, `name[n]`, `name[*]` resolve; an unknown segment → `ValueError`. +- **routing**: a dir of mixed-type corrections → only `applied` + `parametric` change the + config; `open` parametric and `structural`/`context` are ignored. +- **load empty/missing dir** → `[]`. + +## Scope boundaries (YAGNI) + +- **No Obsidian plugin or integration** — we only make memory vault-*compatible* (markdown + + frontmatter + wikilinks). The user opens their own vault if they want. +- **No automatic correction generation** — humans author corrections; the skill structures + them. (The backtest *flags*, it does not author.) +- **No Loop B mining here** — Loop B consumes the format; that's its own spec. +- **No blanket retrofit** of every memory file — establish the vault convention and the + `MEMORY.md` index; deep-link only where it earns its keep. +- **Override grammar stays small** — dotted paths with `[n]`/`[*]` and a scalar value; no + arithmetic/expressions. diff --git a/docs/superpowers/specs/2026-06-08-portfolio-learning-design.md b/docs/superpowers/specs/2026-06-08-portfolio-learning-design.md new file mode 100644 index 0000000..e1a287a --- /dev/null +++ b/docs/superpowers/specs/2026-06-08-portfolio-learning-design.md @@ -0,0 +1,177 @@ +# Portfolio Learning (Loop B) — Design + +**Status:** Approved (design phase) +**Date:** 2026-06-08 +**Topic:** Cross-client learning for a fractional CFO's own book — distill patterns +that generalize across the operator's clients into a reusable local library that +seeds new clients. All local; the practice compounds. + +## Why + +Loop A makes the model better at *one* client over time. Loop B makes the +*operator's practice* compound: client #10 starts smarter than client #1 because the +operator's library carries what generalized across #1–9. The target user is often a +fractional CFO with many clients — so "cross-client" means *their own book, on their +own machine*. Nothing phones home; the Mission's privacy promise holds. + +The objective metric exists at the portfolio level: **does a pattern learned on some +clients improve (or at least not degrade) the backtest on the *other* clients?** — +a client-level train/test split that reuses Loop A's `holdout_backtest`/`score_forecast`. + +## Decisions (locked) + +| Decision | Choice | +| --- | --- | +| Aggregator | A fractional CFO's **own book, all local**. No central service, no phone-home. | +| Validation | **Graceful both**: recurrence surfaces candidates; cross-client holdout validates where enough clients exist; operator **always ratifies**. | +| Promote what | **Both** parametric priors (objective) and structural skills (recurrence + judgment). | +| Architecture | Snapshot-mining + cross-client holdout, local library, operator-ratified. Tested `pyfpa.portfolio` + `fpa-portfolio-learn` skill. | +| Mining sources | Loop A snapshots + applied parametric corrections (priors); generated skills + structural corrections (skills). | + +## Portfolio & Library (local, operator-owned) + +**Portfolio manifest** — `~/.fpa/portfolio.yaml`: +```yaml +library: ~/.fpa/library +clients: + - { path: ~/clients/acme-bikes, type: d2c-inventory } + - { path: ~/clients/haulco, type: trucking } + - { path: ~/clients/brightsaas, type: saas } +``` +Each client = a workspace root (containing `.fpa/` and `skills/`) + a business-type tag +(the clustering key). + +**Library** — `~/.fpa/library/`: +``` +library/ + priors/d2c-inventory.yaml # promoted parametric priors for this business-type + skills//SKILL.md # promoted structural skills + library-log.md # audit: what, evidence (recurrence + holdout delta), date — reversible +``` + +## Components — `pyfpa/portfolio/` (lean, pure, tested) + +`pyfpa/portfolio/manifest.py` +- `class ClientRef(BaseModel)`: `path: str`, `type: str`. +- `class Portfolio(BaseModel)`: `library: str`, `clients: list[ClientRef]`. +- `load_portfolio(path) -> Portfolio` (YAML). +- `clients_of_type(portfolio, business_type) -> list[ClientRef]`. + +`pyfpa/portfolio/recover.py` +- `recover_actuals(snapshot: Snapshot) -> dict[str, float]` — invert the stored score: + `actual = predicted / (1 + per_line_error)` for each scored line. Zero-touch to Loop A + (no new Snapshot field needed). Only scored lines are recoverable (the rest weren't scored). +- `best_snapshot(client_path) -> Snapshot | None` — the lowest-fitness scored snapshot in + `/.fpa/forecasts/` (the assumptions that worked best for that client). + +`pyfpa/portfolio/mine.py` +- `MINEABLE_DRIVERS = ["working_capital.dso_days", "working_capital.dio_days", + "working_capital.dpo_days", "tax_rate", "da_monthly", "capex_monthly"]` (entity-level scalars; + per-channel drivers are a documented extension). +- `class PriorCandidate(BaseModel)`: `business_type`, `driver`, `value`, `support: list[str]` + (client paths/slugs it came from), `dispersion: float`. +- `mine_priors(portfolio, business_type, *, min_support=3, dispersion_max=0.15) -> list[PriorCandidate]` — for each + client of the type, read `best_snapshot(...).assumptions` and applied parametric corrections; + for each mineable driver, collect the per-client values; a driver clustering **tightly** + (coefficient of variation ≤ `dispersion_max`, default 0.15) across ≥ `min_support` clients → + a `PriorCandidate` at the **median**. Scattered → no candidate (not everything generalizes). +- `class SkillCandidate(BaseModel)`: `business_type`, `name`, `support: list[str]`, `source: str`. +- `find_recurring_skills(portfolio, business_type, *, min_support=3) -> list[SkillCandidate]` — + scan each client's `skills/generated/` and structural corrections; a skill name recurring + across ≥ `min_support` clients → a `SkillCandidate` (source = one client's copy). + +`pyfpa/portfolio/validate.py` +- `class ValidationResult(BaseModel)`: `mean_delta: float`, `n_folds: int`, `validated: bool`. +- `validate_prior(driver: str, type_clients: list[ClientRef], *, tolerance=0.0) -> ValidationResult` + — **leave-one-out** across the type's clients. For each client C: derive the prior value as the + median of `driver` across the **other** type clients; take `best_snapshot(C)`, recover its actuals, + apply the override `driver = that median` (`pyfpa.memory.apply_override` on the assumptions), + re-run `cashflow_from_config` + `extract_lines`, `score_forecast` vs recovered actuals → + `delta = new_fitness − snapshot_fitness`. `mean_delta` = mean over the folds. + **A prior is `validated` if `mean_delta ≤ tolerance`** — i.e. a peer-derived value does not degrade + the held-out client's fit (a robust default). Fewer than 2 usable clients → + `validated=False, n_folds=` (recurrence-only, judgment). Always produces an out-of-sample + estimate when ≥2 clients have a scored snapshot — no dependence on the candidate's support set. + +`pyfpa/portfolio/library.py` +- `load_library(library_path) -> dict` (priors by type + skill names) / writers: +- `promote_prior(library_path, candidate, validation) -> None` — append to + `priors/.yaml` with `value`, `support`, `cross_client_holdout_delta`, `date`; log it. +- `promote_skill(library_path, candidate) -> None` — copy the skill into `skills//`; log it. +- `seed_from_library(library_path, business_type, cfg: EntityConfig) -> EntityConfig` — apply the + promoted priors for the type to a new client's starting config (a **new** cfg; the per-client + Loop A refines from there). Priors are seeds, not mandates. + +`pyfpa/portfolio/__init__.py` re-exports; `pyfpa/__init__.py` exposes the public names. + +**Small `pyfpa.memory` touch-up:** promote the existing private `_set_by_path` to a public +`apply_override(data: dict, path: str, value: float) -> None` (same logic) so the portfolio +package doesn't reach into a private helper. `apply_corrections` keeps using it internally. + +## Skill — `skills/fpa-portfolio-learn/SKILL.md` + +Workflow: load manifest → for each business-type with ≥ `min_support` clients, +`mine_priors` + `find_recurring_skills` → `validate_prior` on held-out clients → +present ranked candidates (priors with cross-client delta + support; skills with recurrence) +→ **operator ratifies** → `promote_prior`/`promote_skill` + `library-log.md`. Reversible. + +## The payoff — `fpa-learn-business` integration + +When onboarding a new client tagged a business-type, `fpa-learn-business` calls +`seed_from_library(library, type, base_cfg)` to start the model from the operator's promoted +priors and offers the promoted skills. Client #10 inherits what generalized across #1–9. + +## Guardrails + +- **Local-only** — operates on operator-provided paths; no network; library is transparent files. +- **Min support** (default 3) — never generalize from 1–2 clients. +- **Dispersion gate** — only tight-clustering drivers become candidates. +- **Held-out validation** — a prior must not degrade clients it wasn't derived from + (`mean_delta ≤ tolerance`), or it's flagged judgment-only. +- **Operator ratifies everything; priors are *seeds*** — the per-client Loop A always refines. + Reversible via `library-log.md`. + +## Success criteria + +1. `mine_priors` returns a candidate at the median for a tight-clustering driver across + ≥ `min_support` clients; a scattered driver yields no candidate. +2. `find_recurring_skills` returns a candidate for a skill name recurring across ≥ `min_support` + clients; below that, none. +3. `validate_prior` computes the leave-one-out mean cross-client fitness delta (peer-derived + value per fold) via `recover_actuals` + override + re-score; `validated` iff + `mean_delta ≤ tolerance` with `n_folds ≥ 2`; fewer than 2 usable clients → `validated=False`. +4. `seed_from_library` applies promoted priors to a new client's config (new cfg, input unmutated). +5. Promotions are logged in `library-log.md`, reversible. +6. `pyfpa.portfolio` unit-tested (repo's 80% norm); full suite green; all local (no network). + +## Testing + +- `recover_actuals`: a snapshot with predicted + per_line error → recovered actuals invert exactly. +- `best_snapshot`: picks the lowest-fitness scored snapshot; no scored snapshots → None. +- `mine_priors`: synthetic portfolio where DIO clusters tight across 3 clients → one candidate at + the median; a scattered driver → none; ScoreResult: + """Fit on all but the last `holdout` periods, predict the holdout, and score + predicted vs the held-out actuals. The business-specific `build_cfg_fn` + (fit actuals -> a config that forecasts the holdout window) is supplied by the + caller; this harness only owns the split and the scoring. Nothing is scored on + data it was fit on.""" + periods = list(actuals_by_period) + if len(periods) <= holdout: + raise ValueError(f"need more than {holdout} periods, got {len(periods)}") + fit_periods = periods[:-holdout] + holdout_periods = periods[-holdout:] + + fit_actuals = {p: dict(actuals_by_period[p]) for p in fit_periods} + cfg = build_cfg_fn(fit_actuals) + predicted = extract_lines(cashflow_from_config(cfg), score_lines) + actual = aggregate_periods([dict(actuals_by_period[p]) for p in holdout_periods], score_lines) + return score_forecast(predicted, actual, weights=weights) diff --git a/pyfpa/backtest/learn.py b/pyfpa/backtest/learn.py new file mode 100644 index 0000000..15dea12 --- /dev/null +++ b/pyfpa/backtest/learn.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +from collections.abc import Sequence + +from pyfpa.backtest.snapshot import Snapshot + + +def magnitude_cap(old: float, new: float, *, cap: float = 0.25) -> float: + """Clamp `new` to within ±`cap` (relative) of `old`, preventing overcorrection. + A zero base has no defined relative bound, so `new` passes through.""" + if old == 0: + return new + lo, hi = sorted((old * (1 - cap), old * (1 + cap))) + return max(lo, min(new, hi)) + + +def persistent_miss(errors: Sequence[float], *, k: int = 2, threshold: float = 0.0) -> bool: + """True when the last `k` per-line errors are all the same sign and all exceed + `threshold` in magnitude — a repeated, directional miss, not noise.""" + if len(errors) < k: + return False + last = errors[-k:] + return (all(e > threshold for e in last)) or (all(e < -threshold for e in last)) + + +def render_scorecard(snapshots: Sequence[Snapshot]) -> str: + """Render scored snapshots as a markdown track record (chronological).""" + lines = [ + "# Forecast Scorecard", + "", + "| Period | Fitness (lower=better) | Per-line error |", + "|---|--:|---|", + ] + for s in snapshots: + if s.score is None: + continue + errs = ", ".join(f"{k} {v * 100:+.1f}%" for k, v in s.score.per_line.items()) + lines.append(f"| {s.label} | {s.score.fitness:.4f} | {errs} |") + return "\n".join(lines) + "\n" diff --git a/pyfpa/backtest/score.py b/pyfpa/backtest/score.py new file mode 100644 index 0000000..9845924 --- /dev/null +++ b/pyfpa/backtest/score.py @@ -0,0 +1,77 @@ +from __future__ import annotations + +from collections.abc import Mapping, Sequence + +import pandas as pd +from pydantic import BaseModel + +from pyfpa.analysis.reconcile import reconcile + +DEFAULT_SCORE_LINES = ["ending_cash", "ebitda", "revenue", "gross_margin"] +DEFAULT_WEIGHTS = {"ending_cash": 0.4, "ebitda": 0.3, "revenue": 0.2, "gross_margin": 0.1} + + +def extract_lines(forecast_df: pd.DataFrame, score_lines: Sequence[str]) -> dict[str, float]: + """Reduce a monthly forecast to a {line: value} dict for scoring. + + `ending_cash` is a stock (take the last month); `gross_margin` is a ratio + (Σ gross_profit / Σ revenue); every other line is a flow (sum).""" + revenue = float(forecast_df["revenue"].sum()) + out: dict[str, float] = {} + for line in score_lines: + if line == "ending_cash": + out[line] = float(forecast_df["ending_cash"].iloc[-1]) + elif line == "gross_margin": + gp = float(forecast_df["gross_profit"].sum()) + out[line] = (gp / revenue) if revenue else 0.0 + else: + out[line] = float(forecast_df[line].sum()) + return out + + +def aggregate_periods( + period_dicts: Sequence[Mapping[str, float]], score_lines: Sequence[str] +) -> dict[str, float]: + """Aggregate a chronological list of per-period actual dicts the same way + `extract_lines` aggregates a forecast (flow=sum, stock=last, ratio=ΣGP/Σrev).""" + if not period_dicts: + return {line: 0.0 for line in score_lines} + revenue = sum(float(d.get("revenue", 0.0)) for d in period_dicts) + out: dict[str, float] = {} + for line in score_lines: + if line == "ending_cash": + out[line] = float(period_dicts[-1].get("ending_cash", 0.0)) + elif line == "gross_margin": + gp = sum(float(d.get("gross_profit", 0.0)) for d in period_dicts) + out[line] = (gp / revenue) if revenue else 0.0 + else: + out[line] = sum(float(d.get(line, 0.0)) for d in period_dicts) + return out + + +class ScoreResult(BaseModel): + fitness: float # weighted MAPE across scored lines; lower is better + per_line: dict[str, float] # signed error %, predicted/actual - 1 + weights: dict[str, float] # the (renormalized) weights actually used + + +def score_forecast( + predicted: Mapping[str, float], + actual: Mapping[str, float], + *, + weights: Mapping[str, float] | None = None, +) -> ScoreResult: + """Weighted MAPE of predicted vs actual over the scored lines present in both + (and with non-zero actual). Per-line error reuses `reconcile`'s variance_pct. + Weights are renormalized over the lines actually scored.""" + weights = dict(weights or DEFAULT_WEIGHTS) + lines = [line for line in weights if line in predicted and line in actual and actual[line] != 0] + if not lines: + return ScoreResult(fitness=0.0, per_line={}, weights={}) + rec = reconcile({line: predicted[line] for line in lines}, + {line: actual[line] for line in lines}) + per_line = {line: float(rec.loc[line, "variance_pct"]) for line in lines} + total_w = sum(weights[line] for line in lines) + used = {line: weights[line] / total_w for line in lines} + fitness = sum(used[line] * abs(per_line[line]) for line in lines) + return ScoreResult(fitness=fitness, per_line=per_line, weights=used) diff --git a/pyfpa/backtest/snapshot.py b/pyfpa/backtest/snapshot.py new file mode 100644 index 0000000..295c0dc --- /dev/null +++ b/pyfpa/backtest/snapshot.py @@ -0,0 +1,52 @@ +from __future__ import annotations + +from collections.abc import Sequence +from pathlib import Path + +import pandas as pd +import yaml +from pydantic import BaseModel + +from pyfpa.config.schemas import EntityConfig +from pyfpa.backtest.score import DEFAULT_SCORE_LINES, ScoreResult, extract_lines + + +class Snapshot(BaseModel): + """The full record of one forecast: the assumptions it used, the lines it + predicted, and (once the period closes) the realized score.""" + + label: str + created: str # caller-supplied date string; never generated here + assumptions: dict # serialized EntityConfig + predicted: dict[str, float] + score: ScoreResult | None = None + + +def snapshot_forecast( + cfg: EntityConfig, + forecast_df: pd.DataFrame, + *, + label: str, + created: str, + score_lines: Sequence[str] = DEFAULT_SCORE_LINES, +) -> Snapshot: + """Capture a forecast snapshot from an EntityConfig and its computed DataFrame.""" + return Snapshot( + label=label, + created=created, + assumptions=cfg.model_dump(), + predicted=extract_lines(forecast_df, score_lines), + ) + + +def save_snapshot(snapshot: Snapshot, path: str | Path) -> None: + """Serialize a Snapshot to YAML at the given path.""" + Path(path).write_text(yaml.safe_dump(snapshot.model_dump(), sort_keys=False)) + + +def load_snapshot(path: str | Path) -> Snapshot: + """Deserialize a Snapshot from a YAML file.""" + p = Path(path) + if not p.exists(): + raise FileNotFoundError(f"snapshot not found: {p}") + return Snapshot.model_validate(yaml.safe_load(p.read_text())) diff --git a/pyfpa/memory/__init__.py b/pyfpa/memory/__init__.py new file mode 100644 index 0000000..aaf9c4c --- /dev/null +++ b/pyfpa/memory/__init__.py @@ -0,0 +1,9 @@ +from pyfpa.memory.corrections import ( + Override, Correction, load_corrections, save_correction, apply_corrections, +) +from pyfpa.memory.paths import apply_override + +__all__ = [ + "Override", "Correction", "load_corrections", "save_correction", "apply_corrections", + "apply_override", +] diff --git a/pyfpa/memory/corrections.py b/pyfpa/memory/corrections.py new file mode 100644 index 0000000..de022dd --- /dev/null +++ b/pyfpa/memory/corrections.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Literal + +import yaml +from pydantic import BaseModel + +from pyfpa.config.schemas import EntityConfig +from pyfpa.memory.paths import _set_by_path + +CorrectionType = Literal["parametric", "structural", "context"] +CorrectionStatus = Literal["open", "applied", "superseded"] + + +class Override(BaseModel): + path: str + value: float + + +class Correction(BaseModel): + """One human correction. Frontmatter fields are the machine-readable contract; + `notes` is the human-readable markdown body.""" + slug: str + type: CorrectionType + target: str + status: CorrectionStatus = "open" + date: str + override: Override | None = None + notes: str = "" + + +def _split_frontmatter(text: str) -> tuple[dict, str]: + if text.startswith("---"): + _, frontmatter, body = text.split("---", 2) + return yaml.safe_load(frontmatter) or {}, body.strip() + return {}, text.strip() + + +def save_correction(correction: Correction, directory: str | Path) -> None: + """Write `.md` (YAML frontmatter + markdown body) into `directory`.""" + directory = Path(directory) + directory.mkdir(parents=True, exist_ok=True) + data = correction.model_dump(exclude_none=True) + body = data.pop("notes", "") + data.pop("slug") + text = "---\n" + yaml.safe_dump(data, sort_keys=False) + "---\n" + body + "\n" + (directory / f"{correction.slug}.md").write_text(text) + + +def load_corrections(directory: str | Path) -> list[Correction]: + """Load every `*.md` correction in `directory` (slug = filename stem). + A missing directory returns an empty list.""" + directory = Path(directory) + if not directory.exists(): + return [] + out: list[Correction] = [] + for path in sorted(directory.glob("*.md")): + frontmatter, body = _split_frontmatter(path.read_text()) + out.append(Correction.model_validate({**frontmatter, "slug": path.stem, "notes": body})) + return out + + +def apply_corrections(cfg: EntityConfig, corrections: list[Correction]) -> EntityConfig: + """Return a NEW config with every `applied` + `parametric` correction's override + written in. `open`, `structural`, and `context` corrections are ignored (the + latter two are routed by the skill, not applied to the model). Input unmutated.""" + data = cfg.model_dump() + for correction in corrections: + if correction.status == "applied" and correction.type == "parametric" and correction.override: + _set_by_path(data, correction.override.path, correction.override.value) + return EntityConfig.model_validate(data) diff --git a/pyfpa/memory/paths.py b/pyfpa/memory/paths.py new file mode 100644 index 0000000..17ac143 --- /dev/null +++ b/pyfpa/memory/paths.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +import re + +_SEGMENT = re.compile(r"^(\w+)(?:\[(\*|\d+)\])?$") + + +def _parse_path(path: str) -> list[tuple[str, str | int | None]]: + parsed: list[tuple[str, str | int | None]] = [] + for segment in path.split("."): + match = _SEGMENT.match(segment) + if not match: + raise ValueError(f"malformed override path segment: {segment!r}") + key, index = match.group(1), match.group(2) + if index is None: + parsed.append((key, None)) + elif index == "*": + parsed.append((key, "*")) + else: + parsed.append((key, int(index))) + return parsed + + +def _set_segments(node, segments, value) -> None: + (key, index), rest = segments[0], segments[1:] + if not isinstance(node, dict) or key not in node: + raise ValueError(f"override path key not found: {key!r}") + if index is None: + if rest: + _set_segments(node[key], rest, value) + else: + node[key] = value + return + target = node[key] + if not isinstance(target, list): + raise ValueError(f"override path expects a list at {key!r}, got {type(target).__name__}") + items = range(len(target)) if index == "*" else [int(index)] + for i in items: + if rest: + _set_segments(target[i], rest, value) + else: + target[i] = value + + +def _set_by_path(data: dict, path: str, value: float) -> None: + """Set ``value`` at ``path`` in ``data`` (in place). + + Supports: + - Dotted key navigation: ``"working_capital.dio_days"`` + - Numeric list index: ``"channels[0].seasonality[11]"`` + - Wildcard list spread: ``"channels[*].cogs_pct"`` + + Raises ``ValueError`` on any malformed or unresolvable path segment. + """ + try: + _set_segments(data, _parse_path(path), value) + except (KeyError, IndexError, TypeError) as exc: + raise ValueError(f"cannot apply override path {path!r}: {exc}") from exc + + +def apply_override(data: dict, path: str, value: float) -> None: + """Set ``value`` at dotted ``path`` (supports ``name``, ``name[n]``, ``name[*]``) + in ``data``, in place. Public wrapper over the internal path setter.""" + _set_by_path(data, path, value) diff --git a/pyfpa/portfolio/__init__.py b/pyfpa/portfolio/__init__.py new file mode 100644 index 0000000..d568aff --- /dev/null +++ b/pyfpa/portfolio/__init__.py @@ -0,0 +1,16 @@ +from pyfpa.portfolio.manifest import ClientRef, Portfolio, load_portfolio, clients_of_type +from pyfpa.portfolio.recover import recover_actuals, best_snapshot +from pyfpa.portfolio.mine import ( + MINEABLE_DRIVERS, PriorCandidate, SkillCandidate, mine_priors, find_recurring_skills, +) +from pyfpa.portfolio.validate import ValidationResult, validate_prior +from pyfpa.portfolio.library import ( + load_library, promote_prior, promote_skill, seed_from_library, +) + +__all__ = [ + "ClientRef", "Portfolio", "load_portfolio", "clients_of_type", + "recover_actuals", "best_snapshot", "MINEABLE_DRIVERS", "PriorCandidate", + "SkillCandidate", "mine_priors", "find_recurring_skills", "ValidationResult", + "validate_prior", "load_library", "promote_prior", "promote_skill", "seed_from_library", +] diff --git a/pyfpa/portfolio/library.py b/pyfpa/portfolio/library.py new file mode 100644 index 0000000..2476fe3 --- /dev/null +++ b/pyfpa/portfolio/library.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +import shutil +from pathlib import Path + +import yaml + +from pyfpa.config.schemas import EntityConfig +from pyfpa.memory.paths import apply_override +from pyfpa.portfolio.mine import PriorCandidate, SkillCandidate +from pyfpa.portfolio.validate import ValidationResult + + +def _log(library: Path, line: str) -> None: + library.mkdir(parents=True, exist_ok=True) + log = library / "library-log.md" + header = "" if log.exists() else "# Library Log\n\n" + with log.open("a") as f: + f.write(header + line + "\n") + + +def load_library(library: str | Path) -> dict: + """Read the library: {'priors': {type: [prior dicts]}, 'skills': [names]}.""" + library = Path(library) + priors: dict[str, list[dict]] = {} + priors_dir = library / "priors" + if priors_dir.exists(): + for f in sorted(priors_dir.glob("*.yaml")): + doc = yaml.safe_load(f.read_text()) or {} + priors[doc.get("type", f.stem)] = doc.get("priors", []) + skills = sorted(p.name for p in (library / "skills").glob("*")) if (library / "skills").exists() else [] + return {"priors": priors, "skills": skills} + + +def promote_prior(library: str | Path, candidate: PriorCandidate, validation: ValidationResult) -> None: + """Append a ratified prior to priors/.yaml and log it.""" + library = Path(library) + priors_dir = library / "priors" + priors_dir.mkdir(parents=True, exist_ok=True) + path = priors_dir / f"{candidate.business_type}.yaml" + doc = yaml.safe_load(path.read_text()) if path.exists() else None + doc = doc or {"type": candidate.business_type, "priors": []} + doc["priors"].append({ + "driver": candidate.driver, "value": candidate.value, "support": candidate.support, + "cross_client_holdout_delta": validation.mean_delta, "n_folds": validation.n_folds, + }) + path.write_text(yaml.safe_dump(doc, sort_keys=False)) + _log(library, f"- prior `{candidate.driver}` = {candidate.value} for {candidate.business_type} " + f"(support {len(candidate.support)}, delta {validation.mean_delta:+.4f})") + + +def promote_skill(library: str | Path, candidate: SkillCandidate) -> None: + """Copy a recurring generated skill into the library and log it.""" + library = Path(library) + dest = library / "skills" / candidate.name + dest.mkdir(parents=True, exist_ok=True) + shutil.copytree(candidate.source, dest, dirs_exist_ok=True) + _log(library, f"- skill `{candidate.name}` for {candidate.business_type} " + f"(support {len(candidate.support)})") + + +def seed_from_library(library: str | Path, business_type: str, cfg: EntityConfig) -> EntityConfig: + """Apply the library's promoted priors for `business_type` to a starting config. + Returns a NEW config (input unmutated). Priors are seeds; the per-client loop refines.""" + data = cfg.model_dump() + for prior in load_library(library)["priors"].get(business_type, []): + apply_override(data, prior["driver"], prior["value"]) + return EntityConfig.model_validate(data) diff --git a/pyfpa/portfolio/manifest.py b/pyfpa/portfolio/manifest.py new file mode 100644 index 0000000..83242b9 --- /dev/null +++ b/pyfpa/portfolio/manifest.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +from pathlib import Path + +import yaml +from pydantic import BaseModel + + +class ClientRef(BaseModel): + path: str # client workspace root (contains .fpa/ and skills/) + type: str # business-type tag (the clustering key) + + +class Portfolio(BaseModel): + library: str + clients: list[ClientRef] + + +def load_portfolio(path: str | Path) -> Portfolio: + p = Path(path) + if not p.exists(): + raise FileNotFoundError(f"portfolio manifest not found: {p}") + return Portfolio.model_validate(yaml.safe_load(p.read_text())) + + +def clients_of_type(portfolio: Portfolio, business_type: str) -> list[ClientRef]: + return [c for c in portfolio.clients if c.type == business_type] diff --git a/pyfpa/portfolio/mine.py b/pyfpa/portfolio/mine.py new file mode 100644 index 0000000..231ea6d --- /dev/null +++ b/pyfpa/portfolio/mine.py @@ -0,0 +1,100 @@ +from __future__ import annotations + +import statistics +from pathlib import Path + +from pydantic import BaseModel + +from pyfpa.memory.corrections import load_corrections +from pyfpa.portfolio.manifest import ClientRef, Portfolio, clients_of_type +from pyfpa.portfolio.recover import best_snapshot + +MINEABLE_DRIVERS = [ + "working_capital.dso_days", "working_capital.dio_days", "working_capital.dpo_days", + "tax_rate", "da_monthly", "capex_monthly", +] + + +class PriorCandidate(BaseModel): + business_type: str + driver: str + value: float + support: list[str] + dispersion: float + + +class SkillCandidate(BaseModel): + business_type: str + name: str + support: list[str] + source: str + + +def _get_by_path(data: dict, path: str) -> float | None: + node = data + for segment in path.split("."): + if not isinstance(node, dict) or segment not in node: + return None + node = node[segment] + return float(node) if isinstance(node, (int, float)) else None + + +def client_driver_value(client: ClientRef, driver: str) -> float | None: + """The best-snapshot value for `driver`, overridden by an applied parametric + correction on that exact path if one exists.""" + snap = best_snapshot(client.path) + if snap is None: + return None + for c in load_corrections(Path(client.path) / ".fpa" / "corrections"): + if c.status == "applied" and c.type == "parametric" and c.override and c.override.path == driver: + return c.override.value + return _get_by_path(snap.assumptions, driver) + + +def mine_priors(portfolio: Portfolio, business_type: str, *, + min_support: int = 3, dispersion_max: float = 0.15) -> list[PriorCandidate]: + """Drivers that cluster tightly (CoV <= dispersion_max) across >= min_support + same-type clients become prior candidates at the median.""" + clients = clients_of_type(portfolio, business_type) + out: list[PriorCandidate] = [] + for driver in MINEABLE_DRIVERS: + present = [(c, v) for c in clients if (v := client_driver_value(c, driver)) is not None] + if len(present) < min_support: + continue + values = [v for _, v in present] + spread = statistics.pstdev(values) + mean = statistics.fmean(values) + # Coefficient of variation; a perfectly tight cluster (incl. unanimous zero, + # where mean is 0) is the strongest possible signal → dispersion 0. + cov = 0.0 if spread == 0 else (spread / abs(mean) if mean else float("inf")) + if cov <= dispersion_max: + out.append(PriorCandidate( + business_type=business_type, driver=driver, + value=statistics.median(values), support=[c.path for c, _ in present], + dispersion=cov, + )) + return out + + +def find_recurring_skills(portfolio: Portfolio, business_type: str, *, + min_support: int = 3) -> list[SkillCandidate]: + """Generated-skill directory names that recur across >= min_support same-type + clients. (Structural corrections are an additional human signal the operator + weighs in the skill, not a mechanical trigger here.)""" + clients = clients_of_type(portfolio, business_type) + by_name: dict[str, list[tuple[str, str]]] = {} + for c in clients: + generated = Path(c.path) / "skills" / "generated" + if not generated.exists(): + continue + for d in sorted(generated.iterdir()): + if (d / "SKILL.md").exists(): + by_name.setdefault(d.name, []).append((c.path, str(d))) + out: list[SkillCandidate] = [] + for name, refs in by_name.items(): + if len(refs) >= min_support: + out.append(SkillCandidate( + business_type=business_type, name=name, + support=[p for p, _ in refs], source=refs[0][1], + )) + return out diff --git a/pyfpa/portfolio/recover.py b/pyfpa/portfolio/recover.py new file mode 100644 index 0000000..301db61 --- /dev/null +++ b/pyfpa/portfolio/recover.py @@ -0,0 +1,30 @@ +from __future__ import annotations + +from pathlib import Path + +from pyfpa.backtest.snapshot import Snapshot, load_snapshot + + +def recover_actuals(snapshot: Snapshot) -> dict[str, float]: + """Recover the realized actuals from a scored snapshot by inverting the stored + per-line error: actual = predicted / (1 + error). Only scored lines are + recoverable; an unscored snapshot yields {}. A line that predicted exactly 0 + (error == -1) is unrecoverable from the stored error, so it is skipped.""" + if snapshot.score is None: + return {} + out: dict[str, float] = {} + for line, error in snapshot.score.per_line.items(): + if line in snapshot.predicted and error != -1.0: + out[line] = snapshot.predicted[line] / (1.0 + error) + return out + + +def best_snapshot(client_path: str | Path) -> Snapshot | None: + """The lowest-fitness scored snapshot in /.fpa/forecasts/ — the + assumptions that worked best for that client. None if no scored snapshot.""" + forecasts = Path(client_path) / ".fpa" / "forecasts" + if not forecasts.exists(): + return None + snaps = [load_snapshot(f) for f in sorted(forecasts.glob("*.snapshot.yaml"))] + scored = [s for s in snaps if s.score is not None] + return min(scored, key=lambda s: s.score.fitness) if scored else None diff --git a/pyfpa/portfolio/validate.py b/pyfpa/portfolio/validate.py new file mode 100644 index 0000000..7ffcc5b --- /dev/null +++ b/pyfpa/portfolio/validate.py @@ -0,0 +1,56 @@ +from __future__ import annotations + +import copy +import statistics + +from pydantic import BaseModel + +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.score import DEFAULT_SCORE_LINES, extract_lines, score_forecast +from pyfpa.memory.paths import apply_override +from pyfpa.portfolio.manifest import ClientRef +from pyfpa.portfolio.mine import client_driver_value +from pyfpa.portfolio.recover import best_snapshot, recover_actuals + + +class ValidationResult(BaseModel): + mean_delta: float + n_folds: int + validated: bool + + +def validate_prior(driver: str, type_clients: list[ClientRef], *, tolerance: float = 0.0) -> ValidationResult: + """Leave-one-out cross-client check. For each usable client, derive `driver`'s + value as the median across the OTHER clients, apply it to the held-out client's + best-snapshot config, re-forecast, and score against that client's recovered + actuals. A prior is `validated` if the mean fitness delta (new - original) is + <= tolerance with >= 2 folds — a peer-derived value does not degrade held-out fit.""" + usable = [] + for c in type_clients: + snap = best_snapshot(c.path) + value = client_driver_value(c, driver) + if snap is not None and snap.score is not None and value is not None: + usable.append((value, snap)) + n = len(usable) + if n < 2: + return ValidationResult(mean_delta=0.0, n_folds=n, validated=False) + + deltas = [] + for i, (_, snap) in enumerate(usable): + peer_values = [usable[j][0] for j in range(n) if j != i] + prior_value = statistics.median(peer_values) + data = copy.deepcopy(snap.assumptions) + apply_override(data, driver, prior_value) + forecast = cashflow_from_config(EntityConfig.model_validate(data)) + # Score over the SAME lines/weights Loop A used for this snapshot, so the new + # fitness and snap.score.fitness are apples-to-apples (not assumed defaults). + scored_lines = list(snap.score.per_line) or DEFAULT_SCORE_LINES + predicted = extract_lines(forecast, scored_lines) + new_fitness = score_forecast( + predicted, recover_actuals(snap), weights=snap.score.weights or None + ).fitness + deltas.append(new_fitness - snap.score.fitness) + + mean_delta = statistics.fmean(deltas) + return ValidationResult(mean_delta=mean_delta, n_folds=n, validated=mean_delta <= tolerance) diff --git a/skills/fpa-backtest-learn/SKILL.md b/skills/fpa-backtest-learn/SKILL.md new file mode 100644 index 0000000..950dd7c --- /dev/null +++ b/skills/fpa-backtest-learn/SKILL.md @@ -0,0 +1,64 @@ +--- +name: fpa-backtest-learn +description: Use when you want the model to learn from how its past forecasts actually turned out — scoring forecasts against the company's real actuals, backtesting assumptions on history, and proposing ratified improvements. Runs at/after monthly close. +--- + +# Backtest & Learn (Operate) + +## Overview + +The model should get measurably better at this business over time. This skill +scores past forecasts against the company's actuals, surfaces what keeps missing, +and proposes improvements a human ratifies. The objective metric is reconciliation +error against the user's own books (`pyfpa.score_forecast`) — the FP&A analog of a +validation loss. + +**Core principle:** self-improving, but never self-ratifying. The loop proposes; +a human accepts. Everything it learns lives as plain files in `.fpa/` the user owns. + +## Memory (`.fpa/`) + +- `forecasts/.snapshot.yaml` — each forecast's assumptions + predictions, and (after close) its score. +- `scorecard.md` — the running track record (rendered, never hand-edited). +- `learnings.md` — every accepted change: what, the evidence, the backtest delta, the date. + +## Workflow + +1. **Snapshot every forecast.** When you produce a forecast, persist it: + `snapshot_forecast(cfg, forecast_df, label=, created=)` → + `save_snapshot(..., ".fpa/forecasts/.snapshot.yaml")`. +2. **Score at close.** When a period closes (actuals via **fpa-configure-actuals**), + load that period's snapshot, `score_forecast(snap.predicted, actuals)`, write the + score back into the snapshot, and re-render `scorecard.md` with `render_scorecard`. +3. **Attribute** each material per-line miss to a driver (volume / price / cost ratio + / working-capital timing). **Run the fpa-cfo-judgment one-time-item screen first** — + never blame the model for a one-off. + - **Monitor applied corrections:** if a `type: parametric` correction's target line keeps missing, flag it as possibly stale (`applied → superseded`) for the human — never auto-revert. +4. **Propose**, tagged by type: + - **Parametric** (an assumption change): re-score it with `holdout_backtest` on the + company's history. Surface it **only if it lowers holdout fitness** (not in-sample), + ranked by the delta. Clamp the proposed move with `magnitude_cap` (±25%/cycle). + - **Structural** (a methodology/skill change, e.g. a revenue-recognition lag): surface + **only** when `persistent_miss` is true for the line (same-signed across K≥2 closes) + and it survived the one-time screen. Hand it to **fpa-learn-business** to generate the + skill *on approval* — propose, don't auto-write. +5. **Ratify + log.** Present proposals; the human accepts/rejects. On accept, update the + config and append to `learnings.md` (what, evidence, holdout delta, date). Reversible. + +## Bootstrap (day one) + +If the company already has ~12+ months of actuals, you don't have to wait: run +`holdout_backtest` immediately (fit on the earlier months, hold out the recent ones) +to report current accuracy and the first round of parametric proposals. + +## Guardrails (always) + +- Never score on data the model was fit on (`holdout_backtest` enforces this). +- No proposal off a single period — require a persistent, same-signed miss. +- A parametric change must improve **holdout** fitness, not in-sample fit. +- Cap how far any assumption moves per cycle (`magnitude_cap`). +- Human ratifies everything; every change is logged and reversible. + +## Next + +Scored + learned → **fpa-board-briefing** (report the forecast and how it's tracking). diff --git a/skills/fpa-capture-correction/SKILL.md b/skills/fpa-capture-correction/SKILL.md new file mode 100644 index 0000000..f89a487 --- /dev/null +++ b/skills/fpa-capture-correction/SKILL.md @@ -0,0 +1,69 @@ +--- +name: fpa-capture-correction +description: Use when a human reviewing a forecast catches something off ("December always spikes", "you're double-counting deferred revenue", "that Q3 number was a one-time contract") — captures it as a durable, typed correction in the company's memory so future forecasts are grounded by it. +--- + +# Capture a Correction (Operate) + +## Overview + +A human reviewing a forecast is the highest-signal feedback there is — they catch +structural errors and domain knowledge the backtest can't see, and catch them *now*. +This skill turns that into durable memory: a typed correction in `.fpa/corrections/` +that grounds every future forecast. + +**Core principle:** the human is the authority; capture, confirm interpretation once, +then it persists. Everything is plain markdown the user owns. + +## The three correction types + +- **parametric** — a concrete driver fix ("December runs ~2× a normal month"). Becomes + an `override` (a config path + value) applied to every future forecast via + `pyfpa.apply_corrections`. +- **structural** — a methodology fix ("you're double-counting deferred revenue"). A + *pre-ratified* structural proposal (the human authored it) — route it to + **fpa-learn-business** to generate the skill/model change; do NOT wait for backtest misses. +- **context** — a one-time-item note ("that Q3 spike was a one-off contract"). Annotates so + **fpa-cfo-judgment**'s one-time screen keeps the backtest from "learning" a one-off. + +## Workflow + +1. **Classify** the correction (parametric / structural / context). +2. **Identify the target** — the driver path (e.g. `channels[*].seasonality[11]`, + `working_capital.dio_days`), line, or profile area. For parametric, draft the concrete + `override: {path, value}`. +3. **Write** the correction with `pyfpa.save_correction`. Set `slug` to a + `-` string (e.g. `2026-06-08-december-seasonality`) — `save_correction` + uses the whole slug as the filename (`.fpa/corrections/.md`), so keep the date in + it. Include frontmatter (`type`, `target`, `status`, `date`, `override`) and a markdown + body (`**Was off:** … **Correction:** … **Why:** [[…]]`), linking to the assumption/profile + it corrects with `[[wikilinks]]`. +4. **Confirm interpretation.** Echo back the concrete change ("I'll set December + seasonality to 2.0 on all channels — right?"). Only on confirmation set + `status: applied`. +5. **Keep `.fpa/MEMORY.md` current** — the vault index (see below). + +## Applying corrections + +When building any forecast, `pyfpa.apply_corrections(cfg, load_corrections(".fpa/corrections"))` +folds the applied parametric corrections into the config. The per-client loop refines from +there — corrections are *seeds*, not mandates. + +## The `.fpa/` vault (`MEMORY.md` index) + +Keep a `.fpa/MEMORY.md` that orients a human, Obsidian, or Claude: +- `business-profile.md` — what we know about the business. +- `corrections/` — human corrections (this skill). +- `forecasts/*.snapshot.yaml`, `scorecard.md` — forecast snapshots + backtest track record. +- `learnings.md` — accepted model changes. +All plain markdown — open it in Obsidian if you like, but never required. + +## Guardrails + +- Confirm interpretation before `applied`. Reversible via `status` (`open`/`applied`/`superseded`). +- The backtest *monitors* applied corrections and may flag a stale one — it never reverts; + the human decides. + +## Next + +Correction captured → **fpa-monthly-close** / **fpa-board-briefing** (re-run grounded by it). diff --git a/skills/fpa-cfo-judgment/SKILL.md b/skills/fpa-cfo-judgment/SKILL.md index 21a7ed2..f5c3201 100644 --- a/skills/fpa-cfo-judgment/SKILL.md +++ b/skills/fpa-cfo-judgment/SKILL.md @@ -24,6 +24,7 @@ A forecast that's arithmetically correct can still be wrong about reality. This | **D&A is a real expense** | The engine models D&A: `ebitda` is true EBITDA, EBIT = EBITDA − D&A (set `da_monthly`), and D&A is added back in operating cash flow — never let an add-back inflate cash without first expensing it in the P&L. | | **Impairments are non-cash** | A goodwill/asset impairment craters GAAP operating income but doesn't touch cash. Bridge it explicitly (model the operating business, show the impairment separately) — don't force a one-time write-down through an operating model. | | **Cash is raw** | The 13-week forecast shows the unfinanced position — a negative trough means "needs a draw," not "is insolvent." Say which. | +| **Known one-offs are flagged** | Read `.fpa/corrections/` for `type: context` notes (e.g. "Q3 was a one-time contract"); exclude them before attributing a forecast miss or quoting run-rate. | ## How to apply diff --git a/skills/fpa-configure-actuals/SKILL.md b/skills/fpa-configure-actuals/SKILL.md index c0da61a..c300a53 100644 --- a/skills/fpa-configure-actuals/SKILL.md +++ b/skills/fpa-configure-actuals/SKILL.md @@ -1,46 +1,34 @@ --- name: fpa-configure-actuals -description: Use when wiring a company's real numbers into an openfpa model — mapping a spreadsheet/CSV export into actuals, or connecting a live data source (NetSuite, QuickBooks, Shopify) — after a model config exists. +description: Use when wiring a company's real numbers into an openfpa model — from local spreadsheets (P&L, balance sheet, AR/AP aging, inventory), a live system via MCP or API (QuickBooks, NetSuite), public filings (10-K/10-Q), or anything else. Not married to one source — build the ingestion for whatever the company has; produces one normalized account-amount shape the rest of the toolkit reads. --- # Configure Actuals & Data Sources (Phase 2) ## Overview -Connect the model to real data. Two paths: an **offline** spreadsheet/CSV mapping, or a **live** source adapter. Either way you produce a normalized `{account: amount}` and reconcile it against the model's expected lines. +Connect the model to real data — from wherever it lives. openfpa is **not married to a connector**: everything normalizes to one shape (`{account: amount}`), and where no built-in path exists, **you build the ingestion for this source**. That's the job, not a workaround — see `examples/foxfactory/pull_edgar.py`, a SEC-EDGAR adapter the agent wrote from scratch because no built-in one existed. -**Core principle:** One normalized shape (`{account: amount}`) regardless of source, so the rest of the toolkit doesn't care where numbers came from. +**Core principle:** one normalized shape regardless of source, so the engine never cares where the numbers came from — and the agent meets the data where it is. -## When to use +## The data can come from anywhere -- Replacing scaffolded estimates with the company's actual numbers -- "Connect my QuickBooks / NetSuite / Shopify" requests -- Standing up the monthly refresh +- **Local spreadsheets** (always works, no credentials). A P&L, balance sheet, AR/AP aging, or inventory export. `pyfpa.read_pl_csv(path)` reads any two-column `Account, Amount` CSV (handles `$`, commas, `(parens)` negatives) → `{account: amount}` — it is generic, not P&L-only. For **richer tables** (aged AR/AP buckets, item-level inventory) there is no rigid reader by design: parse the file to what the model needs — derive **DSO** from AR aging, **DIO** from inventory, **DPO** from AP aging. +- **A live accounting system via MCP** — the cleanest live path. If a **QuickBooks** or **NetSuite** MCP server is connected, pull the trial balance / P&L / balance sheet through it and map the result to `{account: amount}`. openfpa never handles credentials — the MCP server owns auth. +- **A live system via API** — `pyfpa.io.adapters.from_quickbooks()` / `from_netsuite()` / `from_shopify()` are starting points (fixture-backed; each docstring documents the live SuiteQL/OAuth or QuickBooks Online / Shopify Admin call). Flesh out the live call when you go that route. +- **Public filings** — a 10-K / 10-Q from SEC EDGAR (curl + a compliant User-Agent), as in the Fox Factory example. +- **Anything else** — if the source isn't covered, write a small ingestion that returns `{account: amount}` (or parses the richer statement to the drivers the model needs). That is the toolkit working exactly as intended. -## Paths - -**Offline (always works, no credentials):** -```python -from pyfpa import read_pl_csv -actuals = read_pl_csv("path/to/export.csv") # {account: amount} -``` -Handles `$`, thousands commas, and `(parens)` negatives. Two columns: `Account`, `Amount`. +## Workflow -**Live source adapters:** -```python -from pyfpa.io import adapters -gl = adapters.from_netsuite() # SuiteQL / OAuth 1.0a -gl = adapters.from_quickbooks() # QuickBooks Online API -ops = adapters.from_shopify() # Shopify Admin API (D2C ops, not a full GL) -``` -Each returns `{account: amount}`. **Without credentials they read bundled synthetic fixtures** — set real credentials in the host environment to go live (see each adapter's docstring). Credentials are never committed. +1. **Pull** actuals via whatever path fits the source. +2. **Map** source accounts onto the model's channels / opex / working-capital lines (reuse the mapping from **fpa-scaffold-model**). +3. **Reconcile** — do source totals match what the model expects? Flag unmapped accounts; never silently drop them. +4. **Judgment** — apply **fpa-cfo-judgment** (pre-close months, flash-vs-GL cash, one-time items). -## Workflow +## Credentials -1. Pull actuals via the chosen path. -2. Map source accounts onto the model's channels/opex lines (reuse the mapping from **fpa-scaffold-model**). -3. **Reconcile**: do the source totals match what the model expects? Flag unmapped accounts rather than silently dropping them. -4. Apply the judgment checks from **fpa-cfo-judgment** — especially pre-close months and flash-vs-GL cash. +Never commit secrets. Live API credentials come from the host environment; MCP servers own their own auth. openfpa ships only synthetic fixtures. ## Next diff --git a/skills/fpa-learn-business/SKILL.md b/skills/fpa-learn-business/SKILL.md index a5c45fd..0dd3fce 100644 --- a/skills/fpa-learn-business/SKILL.md +++ b/skills/fpa-learn-business/SKILL.md @@ -28,12 +28,16 @@ Before scaffolding any model, learn the business. This produces two artifacts: a 2. **Write the business profile** to `.fpa/business-profile.md` (committed). Capture: entity structure, revenue model + channels, cost drivers, seasonality, working-capital rhythm, financing, and the quirks. This is the contract the rest of the toolkit reads. -3. **Identify gaps the 7 standard skills don't cover**, and propose bespoke skills/agents: +3. **Seed from your library.** If a portfolio library exists, start the model from what generalized across your same-type clients: `cfg = pyfpa.seed_from_library(library_path, business_type, cfg)` (priors are seeds — this client's loop refines them). Offer any promoted skills for the type. + +4. **Identify gaps the 7 standard skills don't cover**, and propose bespoke skills/agents: - Product company with SKUs → a `sku-profitability` skill - SaaS → an `arr-waterfall` / cohort-retention skill - Logistics/fleet → a `driver-cost-scorecard` skill -4. **Generate them** following `superpowers:writing-skills` discipline, into `skills/generated/` (and `agents/generated/`). Each generated skill MUST cite the profile facts that justify it. +5. **Generate them** following `superpowers:writing-skills` discipline, into `skills/generated/` (and `agents/generated/`). Each generated skill MUST cite the profile facts that justify it. + +6. **Apply existing corrections.** Before forecasting, fold in human corrections: `pyfpa.apply_corrections(cfg, pyfpa.load_corrections('.fpa/corrections'))`. Route any `type: structural` corrections through this skill's skill-generation path as *pre-ratified* proposals (the human already authored them — don't wait for backtest misses). ## Guardrails (self-extending, NOT self-executing) diff --git a/skills/fpa-monthly-close/SKILL.md b/skills/fpa-monthly-close/SKILL.md index 75abe3f..c803ef0 100644 --- a/skills/fpa-monthly-close/SKILL.md +++ b/skills/fpa-monthly-close/SKILL.md @@ -36,4 +36,4 @@ Fold the latest actuals into the model, refresh the forward forecast from the la ## Next -Close done → **fpa-cash-runway** (near-term liquidity) and **fpa-board-briefing** (the writeup). +Close done → **fpa-cash-runway** (near-term liquidity) and **fpa-board-briefing** (the writeup) and **fpa-backtest-learn** (score this close against the prior forecast and learn from the miss). diff --git a/skills/fpa-portfolio-learn/SKILL.md b/skills/fpa-portfolio-learn/SKILL.md new file mode 100644 index 0000000..985a88d --- /dev/null +++ b/skills/fpa-portfolio-learn/SKILL.md @@ -0,0 +1,59 @@ +--- +name: fpa-portfolio-learn +description: Use when you run FP&A for several clients and want your practice to compound — mines patterns that generalize across your same-type clients, validates them by leave-one-out cross-client backtesting, and promotes ratified priors and skills into a local library that seeds every new client. All local; nothing leaves your machine. +--- + +# Portfolio Learn (Loop B) + +## Overview + +Loop A makes the model better at one client. This makes your *practice* compound: +client #10 starts smarter than client #1 because your library carries what generalized +across #1–9. Everything is local — your own book, on your own machine. + +**Core principle:** self-improving, never self-ratifying — propose, you accept. The +objective metric is cross-client: does a pattern learned on some clients fail to +degrade the *others*' backtest? + +## Setup + +A portfolio manifest `~/.fpa/portfolio.yaml` lists your clients + a business-type tag: + +```yaml +library: ~/.fpa/library +clients: + - { path: ~/clients/acme, type: d2c-inventory } + - { path: ~/clients/peak, type: d2c-inventory } + - { path: ~/clients/haul, type: trucking } +``` + +## Workflow + +1. **Load** the manifest (`pyfpa.load_portfolio`). +2. For each business-type with at least 3 clients: + - **Priors:** let `type_clients = pyfpa.portfolio.clients_of_type(portfolio, type)`. + `pyfpa.mine_priors(portfolio, type)` finds drivers that cluster tightly; validate each + with `pyfpa.validate_prior(driver, type_clients)` (leave-one-out). Surface validated + ones first (by cross-client delta), then unvalidated/judgment. + - **Skills:** `pyfpa.find_recurring_skills(portfolio, type)` for recurring generated + skills. Also weigh recurring **structural corrections** across clients (read each + `.fpa/corrections/` for `type: structural`) — a human-authored pattern that repeats + is strong signal. +3. **Present** candidates ranked by evidence (support count + cross-client delta). +4. **Ratify.** On your acceptance, `pyfpa.promote_prior` / `pyfpa.promote_skill` writes the + `~/.fpa/library/` and `library-log.md`. Reversible. + +## Guardrails + +- Local-only; nothing phones home. +- At least 3 clients to propose; tight-cluster only; a prior must not degrade held-out clients. +- You ratify everything; priors are *seeds*, not mandates — each client's Loop A refines. + +## The payoff + +New clients inherit the library: **fpa-learn-business** seeds their starting model from +your promoted priors (`pyfpa.seed_from_library`) and offers the promoted skills. + +## Next + +Promoted → the next new client onboarded via **fpa-learn-business** starts smarter. diff --git a/tests/test_backtest_holdout.py b/tests/test_backtest_holdout.py new file mode 100644 index 0000000..f652f4f --- /dev/null +++ b/tests/test_backtest_holdout.py @@ -0,0 +1,49 @@ +import pytest +from pyfpa.config.schemas import EntityConfig +from pyfpa.backtest.holdout import holdout_backtest + + +def _actuals(): + out = {} + cash = 0.0 + for i in range(6): + cash += 20.0 + out[f"2026-{i+1:02d}"] = {"revenue": 100.0, "gross_profit": 40.0, + "ebitda": 30.0, "ending_cash": cash} + return out + + +def _build_cfg(growth): + def _fn(fit_actuals): + n = len(fit_actuals) + last_rev = list(fit_actuals.values())[-1]["revenue"] + annual = last_rev * 12 * (1 + growth) + return EntityConfig.model_validate({ + "name": "h", "start_month": f"2026-{n+1:02d}", "horizon_months": 3, + "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": annual, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.6}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 0, "dpo_days": 0, "dio_days": 0}, + "opening_balances": {"cash": 80.0}, + }) + return _fn + + +def test_holdout_backtest_scores_holdout(): + res = holdout_backtest(_actuals(), _build_cfg(0.0), holdout=3, + score_lines=["revenue", "ebitda"]) + assert "revenue" in res.per_line + assert res.fitness >= 0.0 + + +def test_holdout_discriminates_better_assumption(): + good = holdout_backtest(_actuals(), _build_cfg(0.0), holdout=3, score_lines=["revenue"]) + bad = holdout_backtest(_actuals(), _build_cfg(0.5), holdout=3, score_lines=["revenue"]) + assert good.fitness < bad.fitness + + +def test_holdout_rejects_too_few_periods(): + with pytest.raises(ValueError): + holdout_backtest({"2026-01": {"revenue": 1.0}}, _build_cfg(0.0), holdout=3, + score_lines=["revenue"]) diff --git a/tests/test_backtest_learn.py b/tests/test_backtest_learn.py new file mode 100644 index 0000000..24a7f1e --- /dev/null +++ b/tests/test_backtest_learn.py @@ -0,0 +1,32 @@ +import pytest +from pyfpa.backtest.learn import magnitude_cap, persistent_miss, render_scorecard +from pyfpa.backtest.snapshot import Snapshot +from pyfpa.backtest.score import ScoreResult + + +def test_magnitude_cap_clamps_relative(): + assert magnitude_cap(100.0, 200.0, cap=0.25) == pytest.approx(125.0) + assert magnitude_cap(100.0, 10.0, cap=0.25) == pytest.approx(75.0) + assert magnitude_cap(100.0, 110.0, cap=0.25) == pytest.approx(110.0) + assert magnitude_cap(0.0, 5.0, cap=0.25) == 5.0 + + +def test_persistent_miss_requires_k_same_signed(): + assert persistent_miss([0.1, 0.2], k=2) is True + assert persistent_miss([-0.1, -0.2], k=2) is True + assert persistent_miss([0.1, -0.2], k=2) is False + assert persistent_miss([0.2], k=2) is False + assert persistent_miss([0.001, 0.001], k=2, threshold=0.01) is False + + +def test_render_scorecard_table(): + snaps = [ + Snapshot(label="2026-01", created="2026-02-01", assumptions={}, predicted={}, + score=ScoreResult(fitness=0.05, per_line={"revenue": 0.02}, weights={})), + Snapshot(label="2026-02", created="2026-03-01", assumptions={}, predicted={}, + score=ScoreResult(fitness=0.03, per_line={"revenue": -0.01}, weights={})), + ] + md = render_scorecard(snaps) + assert "| 2026-01 |" in md + assert "| 2026-02 |" in md + assert "fitness" in md.lower() diff --git a/tests/test_backtest_scenario.py b/tests/test_backtest_scenario.py new file mode 100644 index 0000000..01cd925 --- /dev/null +++ b/tests/test_backtest_scenario.py @@ -0,0 +1,15 @@ +from pyfpa.backtest.score import score_forecast +from pyfpa.backtest.learn import persistent_miss + + +def test_persistent_cash_overstatement_is_flagged(): + # Across three closes the model predicts ending cash above actual every time + # (a real collections lag the model doesn't capture) -> structural trigger. + closes = [ + ({"ending_cash": 100.0}, {"ending_cash": 90.0}), + ({"ending_cash": 120.0}, {"ending_cash": 108.0}), + ({"ending_cash": 140.0}, {"ending_cash": 126.0}), + ] + cash_errors = [score_forecast(p, a).per_line["ending_cash"] for p, a in closes] + assert all(e > 0 for e in cash_errors) # always overstated + assert persistent_miss(cash_errors, k=3, threshold=0.05) is True diff --git a/tests/test_backtest_score.py b/tests/test_backtest_score.py new file mode 100644 index 0000000..1635251 --- /dev/null +++ b/tests/test_backtest_score.py @@ -0,0 +1,60 @@ +import pandas as pd +import pytest +from pyfpa.backtest.score import extract_lines, aggregate_periods, DEFAULT_SCORE_LINES +from pyfpa.backtest.score import ScoreResult, score_forecast, DEFAULT_WEIGHTS + + +def _forecast(): + idx = pd.period_range("2026-01", periods=3, freq="M") + return pd.DataFrame({ + "revenue": [100.0, 100.0, 100.0], + "gross_profit": [40.0, 40.0, 40.0], + "ebitda": [30.0, 30.0, 30.0], + "ending_cash": [50.0, 70.0, 90.0], + }, index=idx) + + +def test_extract_lines_flow_stock_ratio(): + out = extract_lines(_forecast(), DEFAULT_SCORE_LINES) + assert out["revenue"] == 300.0 + assert out["ebitda"] == 90.0 + assert out["ending_cash"] == 90.0 + assert out["gross_margin"] == pytest.approx(120.0 / 300.0) + + +def test_aggregate_periods_matches_extract(): + periods = [ + {"revenue": 100.0, "gross_profit": 40.0, "ebitda": 30.0, "ending_cash": 50.0}, + {"revenue": 100.0, "gross_profit": 40.0, "ebitda": 30.0, "ending_cash": 70.0}, + {"revenue": 100.0, "gross_profit": 40.0, "ebitda": 30.0, "ending_cash": 90.0}, + ] + out = aggregate_periods(periods, DEFAULT_SCORE_LINES) + assert out["revenue"] == 300.0 + assert out["ebitda"] == 90.0 + assert out["ending_cash"] == 90.0 + assert out["gross_margin"] == pytest.approx(0.4) + + +def test_aggregate_periods_empty_is_zero_filled(): + out = aggregate_periods([], DEFAULT_SCORE_LINES) + assert out == {line: 0.0 for line in DEFAULT_SCORE_LINES} + + +def test_score_forecast_weighted_mape(): + predicted = {"ending_cash": 110.0, "ebitda": 90.0, "revenue": 300.0, "gross_margin": 0.40} + actual = {"ending_cash": 100.0, "ebitda": 100.0, "revenue": 300.0, "gross_margin": 0.40} + res = score_forecast(predicted, actual) + assert res.per_line["ending_cash"] == pytest.approx(0.10) + assert res.per_line["ebitda"] == pytest.approx(-0.10) + assert res.per_line["revenue"] == pytest.approx(0.0) + assert res.fitness == pytest.approx(0.07) + + +def test_score_forecast_skips_absent_and_zero_actual_lines(): + res = score_forecast( + {"ending_cash": 90.0, "ebitda": 30.0}, + {"ending_cash": 100.0, "ebitda": 0.0}, + ) + assert set(res.per_line) == {"ending_cash"} + assert res.weights == {"ending_cash": pytest.approx(1.0)} # renormalized to the one line + assert res.fitness == pytest.approx(0.10) diff --git a/tests/test_backtest_snapshot.py b/tests/test_backtest_snapshot.py new file mode 100644 index 0000000..8b42154 --- /dev/null +++ b/tests/test_backtest_snapshot.py @@ -0,0 +1,41 @@ +import pandas as pd +import pytest +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.snapshot import Snapshot, snapshot_forecast, save_snapshot, load_snapshot +from pyfpa.backtest.score import score_forecast + + +def _cfg(): + return EntityConfig.model_validate({ + "name": "T", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 0, "dpo_days": 0, "dio_days": 0}, + "opening_balances": {"cash": 0.0}, + }) + + +def test_snapshot_forecast_captures_assumptions_and_predictions(): + cfg = _cfg() + snap = snapshot_forecast(cfg, cashflow_from_config(cfg), + label="2026", created="2026-02-01") + assert snap.label == "2026" + assert snap.assumptions["channels"][0]["annual_revenue"] == 1_200_000.0 + assert snap.predicted["revenue"] == pytest.approx(1_200_000.0) + assert snap.score is None + + +def test_snapshot_round_trip(tmp_path): + cfg = _cfg() + snap = snapshot_forecast(cfg, cashflow_from_config(cfg), + label="2026", created="2026-02-01") + snap = snap.model_copy(update={"score": score_forecast(snap.predicted, snap.predicted)}) + p = tmp_path / "2026.snapshot.yaml" + save_snapshot(snap, p) + back = load_snapshot(p) + assert back.label == "2026" + assert back.predicted == snap.predicted + assert back.score.fitness == pytest.approx(0.0) + assert back.assumptions == snap.assumptions diff --git a/tests/test_memory_corrections.py b/tests/test_memory_corrections.py new file mode 100644 index 0000000..7a08931 --- /dev/null +++ b/tests/test_memory_corrections.py @@ -0,0 +1,100 @@ +import pytest +from pyfpa.memory.corrections import ( + Override, Correction, save_correction, load_corrections, +) +from pyfpa.config.schemas import EntityConfig +from pyfpa.memory.corrections import apply_corrections + + +def _cfg(): + return EntityConfig.model_validate({ + "name": "T", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": 30.0}, + "opening_balances": {"cash": 0.0}, + }) + + +def _correction(): + return Correction( + slug="2026-06-08-december-seasonality", + type="parametric", + target="channels[*].seasonality[11]", + status="applied", + date="2026-06-08", + override=Override(path="channels[*].seasonality[11]", value=2.0), + notes="**Was off:** even spread.\n\n**Why:** [[business-profile#seasonality]]", + ) + + +def test_correction_round_trip(tmp_path): + save_correction(_correction(), tmp_path) + loaded = load_corrections(tmp_path) + assert len(loaded) == 1 + c = loaded[0] + assert c.slug == "2026-06-08-december-seasonality" + assert c.type == "parametric" + assert c.status == "applied" + assert c.override == Override(path="channels[*].seasonality[11]", value=2.0) + assert "Was off" in c.notes + assert "[[business-profile#seasonality]]" in c.notes + + +def test_load_missing_dir_is_empty(tmp_path): + assert load_corrections(tmp_path / "nope") == [] + + +def test_structural_correction_has_no_override(tmp_path): + c = Correction(slug="defrev", type="structural", target="revenue", + status="open", date="2026-06-08", + notes="**Was off:** double-counting deferred revenue.") + save_correction(c, tmp_path) + back = load_corrections(tmp_path)[0] + assert back.type == "structural" + assert back.override is None + + +def test_apply_parametric_override_returns_new_cfg(): + cfg = _cfg() + corr = Correction(slug="dio", type="parametric", target="working_capital.dio_days", + status="applied", date="2026-06-08", + override=Override(path="working_capital.dio_days", value=45.0)) + out = apply_corrections(cfg, [corr]) + assert out.working_capital.dio_days == 45.0 + assert cfg.working_capital.dio_days == 30.0 # input unmutated + + +def test_apply_star_seasonality(): + cfg = _cfg() + corr = Correction(slug="dec", type="parametric", target="channels[*].seasonality[11]", + status="applied", date="2026-06-08", + override=Override(path="channels[*].seasonality[11]", value=2.0)) + out = apply_corrections(cfg, [corr]) + assert out.channels[0].seasonality[11] == 2.0 + + +def test_apply_ignores_open_structural_context(): + cfg = _cfg() + corrections = [ + Correction(slug="a", type="parametric", target="working_capital.dio_days", + status="open", date="2026-06-08", + override=Override(path="working_capital.dio_days", value=99.0)), + Correction(slug="b", type="structural", target="revenue", status="applied", + date="2026-06-08"), + Correction(slug="c", type="context", target="revenue", status="applied", + date="2026-06-08"), + ] + out = apply_corrections(cfg, corrections) + assert out.working_capital.dio_days == 30.0 # nothing applied + + +def test_apply_bad_path_raises_loudly(): + # a human-authored correction with a typo'd path must fail loud, not silently no-op + cfg = _cfg() + bad = Correction(slug="x", type="parametric", target="nonexistent.field", + status="applied", date="2026-06-08", + override=Override(path="nonexistent.field", value=1.0)) + with pytest.raises(ValueError): + apply_corrections(cfg, [bad]) diff --git a/tests/test_memory_paths.py b/tests/test_memory_paths.py new file mode 100644 index 0000000..650f389 --- /dev/null +++ b/tests/test_memory_paths.py @@ -0,0 +1,42 @@ +import pytest +from pyfpa.memory.paths import _set_by_path + + +def test_set_nested_scalar(): + data = {"working_capital": {"dio_days": 30.0}} + _set_by_path(data, "working_capital.dio_days", 45.0) + assert data["working_capital"]["dio_days"] == 45.0 + + +def test_set_list_index(): + data = {"channels": [{"seasonality": [1.0] * 12}]} + _set_by_path(data, "channels[0].seasonality[11]", 2.0) + assert data["channels"][0]["seasonality"][11] == 2.0 + + +def test_set_star_applies_to_all_list_items(): + data = {"channels": [{"cogs_pct": 0.5}, {"cogs_pct": 0.4}]} + _set_by_path(data, "channels[*].cogs_pct", 0.6) + assert [c["cogs_pct"] for c in data["channels"]] == [0.6, 0.6] + + +def test_bad_path_raises_valueerror(): + with pytest.raises(ValueError): + _set_by_path({"a": {}}, "a.missing.deep", 1.0) + with pytest.raises(ValueError): + _set_by_path({"a": 1}, "a[", 1.0) # malformed segment + + +def test_index_into_non_list_raises_clear_valueerror(): + # a scalar can't be indexed with [*]/[n] — error names the offending key + type + with pytest.raises(ValueError, match="expects a list"): + _set_by_path({"tax_rate": 0.21}, "tax_rate[*]", 0.25) + + +from pyfpa.memory.paths import apply_override + + +def test_apply_override_is_public_set_by_path(): + data = {"working_capital": {"dio_days": 30.0}} + apply_override(data, "working_capital.dio_days", 45.0) + assert data["working_capital"]["dio_days"] == 45.0 diff --git a/tests/test_portfolio_library.py b/tests/test_portfolio_library.py new file mode 100644 index 0000000..88ad49d --- /dev/null +++ b/tests/test_portfolio_library.py @@ -0,0 +1,66 @@ +from pyfpa.config.schemas import EntityConfig +from pyfpa.portfolio.mine import PriorCandidate, SkillCandidate +from pyfpa.portfolio.validate import ValidationResult +from pyfpa.portfolio.library import ( + load_library, promote_prior, promote_skill, seed_from_library, +) + + +def _cfg(dio=30.0): + return EntityConfig.model_validate({ + "name": "c", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_000_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": dio}, + "opening_balances": {"cash": 0.0}, + }) + + +def test_promote_prior_and_seed(tmp_path): + lib = tmp_path / "library" + cand = PriorCandidate(business_type="d2c", driver="working_capital.dio_days", + value=45.0, support=["a", "b", "c"], dispersion=0.02) + promote_prior(lib, cand, ValidationResult(mean_delta=-0.01, n_folds=3, validated=True)) + assert "working_capital.dio_days" in (lib / "library-log.md").read_text() + seeded = seed_from_library(lib, "d2c", _cfg(dio=30.0)) + assert seeded.working_capital.dio_days == 45.0 + assert _cfg(dio=30.0).working_capital.dio_days == 30.0 # base unmutated + + +def test_seed_unknown_type_is_noop(tmp_path): + lib = tmp_path / "library" + out = seed_from_library(lib, "saas", _cfg(dio=30.0)) + assert out.working_capital.dio_days == 30.0 + + +def test_promote_two_priors_same_type_appends(tmp_path): + # a second prior for the same business-type must EXTEND the file, not overwrite + lib = tmp_path / "library" + val = ValidationResult(mean_delta=0.0, n_folds=3, validated=True) + promote_prior(lib, PriorCandidate(business_type="d2c", driver="working_capital.dio_days", + value=45.0, support=["a", "b", "c"], dispersion=0.02), val) + promote_prior(lib, PriorCandidate(business_type="d2c", driver="tax_rate", + value=0.25, support=["a", "b", "c"], dispersion=0.01), val) + drivers = {p["driver"] for p in load_library(lib)["priors"]["d2c"]} + assert drivers == {"working_capital.dio_days", "tax_rate"} + + +def test_load_library_round_trip(tmp_path): + lib = tmp_path / "library" + cand = PriorCandidate(business_type="d2c", driver="tax_rate", value=0.25, + support=["a", "b", "c"], dispersion=0.01) + promote_prior(lib, cand, ValidationResult(mean_delta=0.0, n_folds=3, validated=True)) + loaded = load_library(lib) + assert any(p["driver"] == "tax_rate" and p["value"] == 0.25 for p in loaded["priors"]["d2c"]) + + +def test_promote_skill_copies_and_logs(tmp_path): + src = tmp_path / "src" / "arr-waterfall" + src.mkdir(parents=True) + (src / "SKILL.md").write_text("---\nname: arr-waterfall\ndescription: x\n---\n") + lib = tmp_path / "library" + promote_skill(lib, SkillCandidate(business_type="saas", name="arr-waterfall", + support=["a", "b", "c"], source=str(src))) + assert (lib / "skills" / "arr-waterfall" / "SKILL.md").exists() + assert "arr-waterfall" in (lib / "library-log.md").read_text() diff --git a/tests/test_portfolio_manifest.py b/tests/test_portfolio_manifest.py new file mode 100644 index 0000000..62e4bb8 --- /dev/null +++ b/tests/test_portfolio_manifest.py @@ -0,0 +1,23 @@ +import pytest +from pyfpa.portfolio.manifest import ClientRef, Portfolio, load_portfolio, clients_of_type + + +def test_load_portfolio_missing_raises(): + with pytest.raises(FileNotFoundError): + load_portfolio("/nonexistent/portfolio.yaml") + + +def test_load_portfolio_and_filter(tmp_path): + (tmp_path / "p.yaml").write_text( + "library: ~/.fpa/library\n" + "clients:\n" + " - { path: ~/clients/acme, type: d2c-inventory }\n" + " - { path: ~/clients/haul, type: trucking }\n" + " - { path: ~/clients/peak, type: d2c-inventory }\n" + ) + pf = load_portfolio(tmp_path / "p.yaml") + assert pf.library == "~/.fpa/library" + assert len(pf.clients) == 3 + d2c = clients_of_type(pf, "d2c-inventory") + assert [c.path for c in d2c] == ["~/clients/acme", "~/clients/peak"] + assert all(isinstance(c, ClientRef) for c in d2c) diff --git a/tests/test_portfolio_mine.py b/tests/test_portfolio_mine.py new file mode 100644 index 0000000..99ced3a --- /dev/null +++ b/tests/test_portfolio_mine.py @@ -0,0 +1,78 @@ +from pathlib import Path + +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.snapshot import snapshot_forecast, save_snapshot +from pyfpa.backtest.score import score_forecast +from pyfpa.portfolio.manifest import Portfolio, ClientRef +from pyfpa.portfolio.mine import mine_priors, find_recurring_skills + + +def _base_cfg(dio): + return EntityConfig.model_validate({ + "name": "c", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": dio}, + "opening_balances": {"cash": 0.0}, + }) + + +def _make_client(tmp_path, name, dio, *, gen_skills=()): + root = tmp_path / name + cfg = _base_cfg(dio) + fc = cashflow_from_config(cfg) + snap = snapshot_forecast(cfg, fc, label="2026", created="2026-01-01") + snap = snap.model_copy(update={"score": score_forecast(snap.predicted, snap.predicted)}) + (root / ".fpa" / "forecasts").mkdir(parents=True, exist_ok=True) + save_snapshot(snap, root / ".fpa" / "forecasts" / "2026.snapshot.yaml") + for s in gen_skills: + d = root / "skills" / "generated" / s + d.mkdir(parents=True, exist_ok=True) + (d / "SKILL.md").write_text(f"---\nname: {s}\ndescription: x\n---\n") + return ClientRef(path=str(root), type="d2c") + + +def _portfolio(clients): + return Portfolio(library="lib", clients=clients) + + +def test_mine_priors_tight_cluster(tmp_path): + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 44.0), ("b", 45.0), ("c", 46.0)]] + cands = mine_priors(_portfolio(clients), "d2c", min_support=3, dispersion_max=0.15) + dio = [c for c in cands if c.driver == "working_capital.dio_days"] + assert len(dio) == 1 + assert dio[0].value == 45.0 + assert len(dio[0].support) == 3 + + +def test_mine_priors_scattered_yields_nothing(tmp_path): + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 20.0), ("b", 50.0), ("c", 95.0)]] + cands = mine_priors(_portfolio(clients), "d2c", min_support=3, dispersion_max=0.15) + assert [c for c in cands if c.driver == "working_capital.dio_days"] == [] + + +def test_mine_priors_below_min_support(tmp_path): + clients = [_make_client(tmp_path, n, 45.0) for n in ("a", "b")] + assert mine_priors(_portfolio(clients), "d2c", min_support=3) == [] + + +def test_mine_priors_unanimous_value_is_strongest_cluster(tmp_path): + # all three clients run da_monthly=0 — a perfectly tight (unanimous) cluster + # should still become a prior, not be dropped by a divide-by-zero mean. + clients = [_make_client(tmp_path, n, 45.0) for n in ("a", "b", "c")] + cands = mine_priors(_portfolio(clients), "d2c", min_support=3) + da = [c for c in cands if c.driver == "da_monthly"] + assert len(da) == 1 + assert da[0].value == 0.0 + assert da[0].dispersion == 0.0 + + +def test_find_recurring_skills(tmp_path): + clients = [_make_client(tmp_path, n, 45.0, gen_skills=["arr-waterfall"]) for n in ("a", "b", "c")] + clients.append(_make_client(tmp_path, "d", 45.0, gen_skills=["one-off"])) + skills = find_recurring_skills(_portfolio(clients), "d2c", min_support=3) + names = [s.name for s in skills] + assert "arr-waterfall" in names + assert "one-off" not in names diff --git a/tests/test_portfolio_recover.py b/tests/test_portfolio_recover.py new file mode 100644 index 0000000..7a28cce --- /dev/null +++ b/tests/test_portfolio_recover.py @@ -0,0 +1,47 @@ +import pytest +from pyfpa.backtest.snapshot import Snapshot, save_snapshot +from pyfpa.backtest.score import ScoreResult +from pyfpa.portfolio.recover import recover_actuals, best_snapshot + + +def _snap(label, fitness, predicted, per_line): + return Snapshot(label=label, created="2026-01-01", assumptions={}, predicted=predicted, + score=ScoreResult(fitness=fitness, per_line=per_line, weights={})) + + +def test_recover_actuals_inverts_error(): + snap = _snap("p", 0.1, {"revenue": 110.0, "ebitda": 90.0}, {"revenue": 0.10, "ebitda": -0.10}) + act = recover_actuals(snap) + assert act["revenue"] == pytest.approx(100.0) + assert act["ebitda"] == pytest.approx(100.0) + + +def test_recover_actuals_skips_zero_predicted(): + # predicted 0 with non-zero actual stores error -1.0 → unrecoverable, skip (no crash) + snap = _snap("p", 0.5, {"revenue": 0.0, "ebitda": 100.0}, {"revenue": -1.0, "ebitda": 0.0}) + act = recover_actuals(snap) + assert "revenue" not in act + assert act["ebitda"] == pytest.approx(100.0) + + +def test_recover_actuals_no_score_is_empty(): + snap = Snapshot(label="p", created="2026-01-01", assumptions={}, predicted={"revenue": 1.0}) + assert recover_actuals(snap) == {} + + +def test_best_snapshot_picks_lowest_fitness(tmp_path): + forecasts = tmp_path / ".fpa" / "forecasts" + forecasts.mkdir(parents=True, exist_ok=True) + save_snapshot(_snap("2026-01", 0.20, {"revenue": 1.0}, {"revenue": 0.0}), forecasts / "a.snapshot.yaml") + save_snapshot(_snap("2026-02", 0.05, {"revenue": 1.0}, {"revenue": 0.0}), forecasts / "b.snapshot.yaml") + best = best_snapshot(tmp_path) + assert best.label == "2026-02" + + +def test_best_snapshot_none_when_unscored_or_missing(tmp_path): + assert best_snapshot(tmp_path / "nope") is None + forecasts = tmp_path / ".fpa" / "forecasts" + forecasts.mkdir(parents=True, exist_ok=True) + unscored = Snapshot(label="p", created="2026-01-01", assumptions={}, predicted={"revenue": 1.0}) + save_snapshot(unscored, forecasts / "u.snapshot.yaml") + assert best_snapshot(tmp_path) is None diff --git a/tests/test_portfolio_validate.py b/tests/test_portfolio_validate.py new file mode 100644 index 0000000..81c71a4 --- /dev/null +++ b/tests/test_portfolio_validate.py @@ -0,0 +1,48 @@ +from pyfpa.config.schemas import EntityConfig +from pyfpa.models.cashflow import cashflow_from_config +from pyfpa.backtest.snapshot import snapshot_forecast, save_snapshot +from pyfpa.backtest.score import score_forecast +from pyfpa.portfolio.manifest import ClientRef +from pyfpa.portfolio.validate import validate_prior + + +def _base_cfg(dio): + return EntityConfig.model_validate({ + "name": "c", "start_month": "2026-01", "horizon_months": 12, "tax_rate": 0.0, + "channels": [{"name": "C", "annual_revenue": 1_200_000.0, "growth_rate": 0.0, + "seasonality": [1.0] * 12, "cogs_pct": 0.5}], + "opex": [], "debt": [], + "working_capital": {"dso_days": 30.0, "dpo_days": 30.0, "dio_days": dio}, + "opening_balances": {"cash": 0.0}, + }) + + +def _make_client(tmp_path, name, dio): + root = tmp_path / name + cfg = _base_cfg(dio) + snap = snapshot_forecast(cfg, cashflow_from_config(cfg), label="2026", created="2026-01-01") + snap = snap.model_copy(update={"score": score_forecast(snap.predicted, snap.predicted)}) + (root / ".fpa" / "forecasts").mkdir(parents=True, exist_ok=True) + save_snapshot(snap, root / ".fpa" / "forecasts" / "2026.snapshot.yaml") + return ClientRef(path=str(root), type="d2c") + + +def test_validate_tight_cluster_is_validated(tmp_path): + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 44.0), ("b", 45.0), ("c", 46.0)]] + res = validate_prior("working_capital.dio_days", clients, tolerance=0.01) + assert res.n_folds == 3 + assert res.mean_delta <= 0.01 + assert res.validated is True + + +def test_validate_scattered_not_validated(tmp_path): + clients = [_make_client(tmp_path, n, dio) for n, dio in [("a", 10.0), ("b", 60.0), ("c", 120.0)]] + res = validate_prior("working_capital.dio_days", clients, tolerance=0.0) + assert res.validated is False + + +def test_validate_too_few_clients(tmp_path): + clients = [_make_client(tmp_path, "a", 45.0)] + res = validate_prior("working_capital.dio_days", clients) + assert res.n_folds == 1 + assert res.validated is False diff --git a/tests/test_public_api.py b/tests/test_public_api.py index 5bfe772..7bae9fd 100644 --- a/tests/test_public_api.py +++ b/tests/test_public_api.py @@ -22,6 +22,13 @@ def test_all_matches_public_api(): "Sku", "sku_profitability", "pareto_breakpoint", "load_skus", "Segment", "segment_pnl", "roll_up_segments", "segments_to_channels", "Carveout", "divest", "net_debt_to_ebitda", "reconcile", + "ScoreResult", "score_forecast", "Snapshot", "snapshot_forecast", + "save_snapshot", "load_snapshot", "holdout_backtest", + "magnitude_cap", "persistent_miss", "render_scorecard", + "Override", "Correction", "load_corrections", "save_correction", "apply_corrections", + "apply_override", + "Portfolio", "load_portfolio", "mine_priors", "find_recurring_skills", + "validate_prior", "promote_prior", "promote_skill", "seed_from_library", }, "pyfpa.__all__ must exactly match expected public API"