Skip to content

EXAMPLES.md

Latest commit

 

History

History
166 lines (114 loc) · 5.42 KB

EXAMPLES.md

File metadata and controls

166 lines (114 loc) · 5.42 KB
Raw

Examples

Worked, end-to-end recipes. They assume a configured environment:

export DS4_DIR=~/ds4
export MODELS_DIR=~/ds4-models
# the benchmark-calibration examples additionally need a benchy checkout —
# clone it beside forgequant (./benchy) or set $BENCHY_DIR:
git clone https://github.com/andreaborio/benchy.git

Every command that builds anything has a dry equivalent — run show first to see the exact ds4 invocation without executing it:

python3 forgequant.py show <recipe>

1. "Keep my coding expert sharp" — build a code-calibrated quant from scratch

The flagship flow: calibrate an imatrix on real, non-saturated code benchmarks, then upcast the routed experts of the layers that code lights up hardest.

python3 forgequant.py verify coder-q4boost     # preflight: paths, disk, boost resolution
python3 forgequant.py build  coder-q4boost     # corpus → imatrix → quantize → GGUF + manifest

build does the whole pipeline because the recipe carries a bench block:

"bench": { "keys": ["humaneval","mbpp","mmlu_cs"], "answers": true, "mix": "reasoning", "cap": 400 }

It fetches HumanEval + MBPP + MMLU-CS from benchy, renders a corpus (with gold answers, mixed with a reasoning set so it doesn't over-specialize), collects coder.dat, then quantizes IQ2 everywhere with Q4_K on the 6 code-hottest layers (boost: auto:6). The provenance of the corpus lands in bench/runs/.

2. See where a workload lives — and what makes it different

paths parses the .dat directly (no ds4) and prints a per-(layer, expert) heatmap.

python3 forgequant.py paths coder-q4boost                  # hot layers + per-expert energy
python3 forgequant.py paths coder.dat --diff medical.dat   # where do CODE and MEDICAL diverge?
python3 forgequant.py paths coder.dat --contrast general-baseline.dat   # what CODE adds over a general baseline
python3 forgequant.py paths coder.dat --json               # machine-readable, for scripting/the UI

--diff is symmetric (two domains); --contrast is directional (domain vs a general baseline) and is the signal behind boost.mode: contrast.

3. Let the imatrix propose the boost

python3 forgequant.py suggest coder-q4boost --top 6 --type q4_k

Prints a ready-to-paste boost block and the estimated GGUF size delta. Tune --top to trade size for fidelity, then paste it into your recipe (or leave auto:6 to resolve at build time).

4. Fastest A/B loop — splice instead of requantize

A boost only changes a few layers. Instead of regenerating the whole model, copy the hot layers from a higher-precision donor GGUF — minutes, not hours:

python3 forgequant.py splice splice-fast
"splice": { "donor": "{models}/<q4-variant>.gguf", "layers": "auto:6" }

Donor and base must come from the same template (same shapes/layer count). Use this to A/B a layer set quickly, then bake the winner with a real quantize.

…or incremental requantize with reuse

If you do requantize but only changed a few layers, point reuse at a prior build with the same imatrix and the quantizer copies the byte-identical unchanged tensors (~85% less work):

"reuse": "{models}/DeepSeek-V4-Flash-coder-iq2.gguf"

Safe by construction: it's gated on a hash of the safetensors index + imatrix, so a mismatched prior silently falls back to a full quantize.

5. Calibrate from live traffic instead of a benchmark

Serve the model, use it for real, Ctrl-C when done. The imatrix records only aggregate per-expert statistics — no prompt text is ever stored.

python3 forgequant.py capture coder-q4boost --port 8000
# ... drive real requests at http://localhost:8000 ...
# Ctrl-C → forgequant waits for ds4-server to flush the final snapshot

6. Calibrate from your own prompts (no benchmark, no ds4 tooling)

Render a raw .txt (blank-line-separated) or .jsonl prompt list into a corpus, then collect the imatrix from it:

python3 forgequant.py render my_prompts.txt -o coder_corpus.txt --mode both
python3 forgequant.py imatrix coder-q4boost          # recipe points `corpus` at coder_corpus.txt

--mode both emits a nothink and a think variant per prompt, so the imatrix sees the activation paths of both reasoning styles.

7. Explore from the browser

python3 forge_ui.py        # → http://localhost:8060

Template gallery, recipe builder (families + boost + imatrix), all five actions with live per-tensor/per-layer progress, the interactive 43×256 brain-map (single, diff, and contrast views), one-click boost suggestion, and a past-runs browser.

8. Measure expert concentration (research driver)

How many of the 256 experts does coding actually use? Drive a representative coding mix through a profiling server and read the LRU cache-hit curve:

./run_expert_profile.sh 180                          # serve + drive + flush profile.json
python3 expert_profile_drive.py parse profile.json   # unique experts/layer, 16/24/32-expert thresholds

A/B the result with benchy

Every output is a drop-in GGUF. Serve it and measure:

ds4-server -m $MODELS_DIR/DeepSeek-V4-Flash-coder-q4boost.gguf --ssd-streaming --port 8000 &
python3 benchy/eval_mcq.py benchy/data/humaneval.jsonl 60 think coder-q4boost

See bench/README.md for the calibration side and ARCHITECTURE.md for how the pieces fit together.