Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
130 lines (102 loc) · 8.32 KB

ARCHITECTURE.md

File metadata and controls

130 lines (102 loc) · 8.32 KB
Raw

Architecture

forgequant is a thin orchestrator. It owns no quantization math and no benchmark data — it turns a JSON recipe into the exact ds4 command lines that do the work, runs them, and records what it did. Everything here is stdlib-only Python (numpy is used opportunistically if importable, never required).

Read this if you want to change forgequant. For using it, see the README and EXAMPLES.md.

The idea in one paragraph

A model's 2-bit budget can be spent asymmetrically, at three depths: family (keep attention/shared/output near-lossless, push routed experts to 2 bits), expert (an imatrix re-allocates bits inside every tensor toward the experts a workload activates), and layer (upcast the routed experts of the hottest layers). forgequant makes that allocation a file you can version, diff, and re-run.

Data flow

flowchart TD
    R["recipe.json<br/>(families · imatrix · boost)"] --> L[load_recipe<br/>resolve paths · validate]

    subgraph cal["Calibration — build an imatrix (.dat)"]
      B["bench block"] --> EC[ensure_corpus] --> C["corpus.txt"]
      P["raw prompts (.txt/.jsonl)"] --> RND[render] --> C
      C --> IMX["ds4 --imatrix-dataset"]
      LIVE["live traffic"] --> CAP["ds4-server --imatrix-out"]
      IMX --> DAT[("imatrix .dat")]
      CAP --> DAT
    end

    L --> cal
    DAT --> RB["resolve_boost_layers<br/>auto:N → hot/contrast/sensitivity"]

    subgraph prod["Produce a GGUF"]
      RB --> Q["deepseek4-quantize<br/>(--routed-w* · --tensor-type · --reuse)"]
      DON["donor.gguf"] --> SP["splice_mixed_expert_layers_gguf.py"]
      Q --> G[("model.gguf")]
      SP --> G
    end

    L --> prod
    G --> M["&lt;out&gt;.manifest.json<br/>recipe · command · ds4 rev · SHA-256"]
Loading

The paths / suggest analysis tools branch off the .dat: they parse it directly and never run ds4.

Modules

Module Role CLI?
forgequant.py The CLI and orchestrator. Loads/validates recipes, resolves path templates, builds every ds4 command, runs subprocesses, writes manifests. yes — the entry point
forge_imatrix.py Reads & analyzes ds4's .dat imatrix format. Energy/sensitivity/contrast aggregation, hot-layer ranking, boost suggestion, terminal heatmaps, boost size-delta estimates, JSON export for the UI. yes (standalone analyzer)
forge_corpus.py Renders raw prompts / chat histories / benchmark rows into a DeepSeek-V4-templated calibration corpus (mirrors ds4_server.c's prompt rendering). think/nothink modes, gold-answer turns, domain mixing. yes (standalone renderer)
forge_bench.py The single bridge to benchy (a separate project, located via $BENCHY_DIR / ./benchy). Lists the registry, resolves selections (key/bundle/domain), fetches rows, renders corpora, and writes calibration provenance to bench/runs/. yes (list/bundles/fetch/corpus)
forge_ui.py Single-file stdlib web dashboard (port 8060): recipe builder, build/quantize/imatrix/capture/splice actions with live per-tensor/per-layer progress, an interactive 43×256 brain-map, and a runs browser. yes (the server)
expert_profile_drive.py Research driver: measures coding-expert concentration via ds4 --expert-profile (LRU cache-hit curves over the 256-expert pool). yes (drive/parse)
test_forge.py Stdlib unittest suite. Covers the .dat parser, corpus renderer, recipe loader/validator, and UI path-traversal guards. No ds4, models, or network required. —

forgequant.py is the only module the others don't import; it imports forge_imatrix, forge_corpus, and forge_bench. forge_ui.py reuses forge_imatrix for analysis and shells out to forgequant.py for actions.

CLI surface (forgequant.py)

Dispatch is an explicit CMDS table (no argparse); main() matches sys.argv[1].

Command What it does
list List recipes in recipes/ with [boost] / [splice] tags
show <recipe> Print the resolved recipe and the exact commands — runs nothing
verify <recipe> Preflight: paths exist, disk space, boost resolves, dry-run cost
bench … Proxy to forge_bench (list/bundles/fetch/corpus) — no recipe needed
imatrix <recipe> Build the .dat via ds4 --imatrix-dataset (auto-builds the corpus from a bench block first)
capture <recipe> [--port N] Serve live (ds4-server --imatrix-out) and record activation paths from real traffic; Ctrl-C flushes a final snapshot
render <in> -o <out> [--mode …] Render raw prompts into a corpus
paths <recipe|.dat> [--json] [--contrast B] [--diff B] Per-(layer, expert) heatmap; contrast/diff two imatrices
suggest <recipe> [--top N] [--type T] Propose a boost block + estimated size delta
quantize <recipe> Run deepseek4-quantize; write GGUF + manifest
splice <recipe> Copy hot expert layers from a donor GGUF (no requantize); write GGUF + manifest
build <recipe> Full pipeline: imatrix (if missing) → quantize

External dependencies

forgequant calls three ds4 artifacts (resolved under $DS4_DIR, default ~/ds4):

  • gguf-tools/deepseek4-quantize — the quantizer (families, --tensor-type boost, --reuse).
  • ds4 --imatrix-dataset / ds4-server --imatrix-out — offline and live imatrix collection.
  • gguf-tools/mixed/splice_mixed_expert_layers_gguf.py — the layer splicer.

It also uses benchy (a separate project, located via $BENCHY_DIR, else a ./benchy checkout, else ~/benchy) for benchmark calibration. forgequant talks to benchy only through its stable api.py contract (API_VERSION), never its internals, so a benchy refactor can't break forgequant; api.py absent → forge_bench falls back to the legacy fetcher automatically.

$MODELS_DIR (default ~/ds4-models) is the base for the {models} template var and the default location for sources, imatrices, corpora, and outputs.

Key invariants

  • Deterministic. The same recipe + the same imatrix produce the same GGUF. Manifests add provenance (ds4 git rev, timestamps), not nondeterminism.
  • Manifested. Every quantize/splice writes <out>.manifest.json: resolved recipe, exact command, ds4 revision, duration, and SHA-256 + size of both the imatrix and the output.
  • Template fall-through. Any tensor family omitted from quant is copied verbatim from template. Omission is meaningful — run show to see what's copied.
  • Producible types only. deepseek4-quantize can generate iq2_xxs, q2_k, q4_k, q8_0 (plus f16/bf16/f32 passthrough). Recipes naming other types (q3_k, iq2_s, q5_k, …) are rejected at load time.
  • 43 layers. DeepSeek-V4-Flash routed layers are 0..42; layer specs are validated against that range.
  • Privacy of capture. ds4-server --imatrix-out records only aggregate per-expert statistics — no prompt text is stored (see ds4's ONEDGE_IMATRIX.md).
  • No data redistribution. Benchmark rows are fetched from HuggingFace on demand; rendered corpora are gitignored. Only provenance (bench/runs/*.json) is committed.

The .dat imatrix format (what forge_imatrix parses)

A packed binary file: int32 entry count, then per entry `name_len + name + ncall(=1)

  • nval + nval × float32, then an optional trailer (int32 chunks + dataset name). Values are **count-normalized** mean-squared activations: *how hard an expert works when routed*, not how often it is routed. Never-routed experts appear as all-zero ("cold"). Family is derived from the tensor name (ffn_gate_exps→w1, ffn_down_exps→w2, ffn_up_exps→w3). Parsed stats are cached next to the .datas.fqstats.json`, keyed on size+mtime so stale caches regenerate.

What's tracked in git

Small, reproducible, traceable things only — see .gitignore. Weights, imatrices (.dat), GGUFs, manifests, and rendered corpora are local/large and never committed; calibration provenance (bench/runs/*.json) — which records the exact benchmark snapshot, including the upstream dataset commit — is. Pre-registration notes (PREREG_*.md) are committed and frozen — they prove a hypothesis was locked before measurement.