Dv/cpu image bringup report by DenisValeev · Pull Request #20 · PrismML-Eng/Bonsai-Image-Demo

DenisValeev · 2026-06-03T02:42:13Z

Summary

Adds a focused CPU image generation bring-up report for bonsai-image.

The report centers the strongest demonstrated result: the unpacked transformer CPU path can produce coherent 128x128 outputs end to end, including:

plain ostrich
a coherent ostrich silhouette
a large centered red circle
a clean 4-quadrant multi-color layout

Why this is useful

This makes the current CPU status easier to understand and reproduce.

It documents that:

the unpacked CPU path can converge on globally coherent images
both geometric and object-level prompts can work on CPU
128x128 is a practical validation target for CPU image generation

Guidance captured in the report

use 128x128+ for structure/composition validation
judge the active path primarily by final outputs
use the unpacked transformer CPU path as a reference-capable CPU configuration

Notes

this is a bring-up/status report, not a claim that every CPU failure mode is solved
no host-specific, secret, or private-environment details are included

khosravipasha · 2026-06-06T01:18:17Z

oh nice, how fast was it?

Copilot

Pull request overview

Adds an experimental CPU image-generation script plus accompanying bring-up/report docs to document and reproduce coherent 128x128 CPU outputs (notably on the unpacked transformer path) for bonsai-image.

Changes:

Added a standalone scripts/generate_cpu_experimental.py script to run Flux2 CPU diffusion with logging and optional step image dumps.
Added two markdown reports capturing validated CPU bring-up results and suggested practical guidance.
Documented an example command shape intended to reproduce the 128x128, 4-step CPU results.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
scripts/generate_cpu_experimental.py	New experimental CPU generation entrypoint (prompt encode → diffusion → VAE decode) with model loading helpers and detailed logging.
docs/upstream_issue_cpu_bringup.md	Focused bring-up/status note describing the strongest validated CPU results and reproduction shape.
docs/upstream_cpu_image_report_draft.md	Longer-form draft report consolidating the same CPU bring-up evidence and guidance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+REPO_ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str((REPO_ROOT / "vendor" / "image-studio").resolve()))
+
+from backend_gpu.diffusion_klein import _mflux_empirical_mu  # noqa: E402


+def mem_gib() -> float:
+    with open("/proc/self/status") as fh:
+        for line in fh:
+            if line.startswith("VmRSS:"):
+                return int(line.split()[1]) / 1024 / 1024
+    return 0.0


+    parser.add_argument("--output", required=True)
+    parser.add_argument("--height", type=int, default=256)
+    parser.add_argument("--width", type=int, default=256)
+    parser.add_argument("--steps", type=int, default=1)


+    if args.height % 32 != 0 or args.width % 32 != 0:
+        raise SystemExit("height and width must be multiples of 32")
+


+python scripts/generate_cpu_experimental.py \
+  --prompt 'ostrich' \
+  --height 128 \
+  --width 128 \
+  --steps 4 \
+  --seed 7 \
+  --transformer-dir models/bonsai-image-4B-ternary-unpacked/transformer


+python scripts/generate_cpu_experimental.py \
+  --prompt 'ostrich' \
+  --height 128 \
+  --width 128 \
+  --steps 4 \
+  --seed 7 \
+  --transformer-dir models/bonsai-image-4B-ternary-unpacked/transformer


DenisValeev · 2026-06-06T01:32:06Z

oh nice, how fast was it?

Per codex: fastest passing config here was 96x96, 2-step, fp32, 4 threads: 11.4s warm render, about 49.6s total including 38.2s setup. 128x128 at 2 steps was 13.2s warm / 54.7s total, and 128x128 at 4 steps was 33.0s warm / 80.2s total. Old 128x128 4-step baseline here was about 20m42s, so the search materially improved it.

This is with 4 neoverse arm vcpus and 24 gigs of ram on a free tier oracle cloud server. Will push latest changes to this branch.

DenisValeev · 2026-06-06T01:39:21Z

DenisValeev · 2026-06-06T15:16:25Z

khosravipasha · 2026-06-07T18:06:23Z

@@ -0,0 +1 @@
+scripts: add warm CPU server benchmark helpers


can delete this file probably

khosravipasha · 2026-06-07T18:06:47Z

@@ -0,0 +1,38 @@
+## Summary


do we need to commit this file?
should the in the PR itself

khosravipasha · 2026-06-07T18:13:08Z

Nice that's faster than I imagined for CPU only, is this running in fp16 or using the 1-bit or ternary packing? Kinda saw mix of both when skimming through the code.
Happy to merge it after some clean up and making it blend better with rest of demo, e.g. have a script that starts a server, etc.

DenisValeev · 2026-06-07T18:16:32Z

Codex is on it, will post a reply shortly Sincerely, Denis Valeev

…

On Sun, Jun 7, 2026 at 2:13 PM Pasha Khosravi ***@***.***> wrote: *khosravipasha* left a comment (PrismML-Eng/Bonsai-Image-Demo#20) <#20 (comment)> Nice that's faster than I imagined for CPU only, is this running in fp16 or using the 1-bit or ternary packing? Kinda saw mix of both when skimming through the code. Happy to merge it after some clean up and making it blend better with rest of demo, e.g. have a script that starts a server, etc. — Reply to this email directly, view it on GitHub <#20?email_source=notifications&email_token=AAGX7TWKWSRAATI5PIDAFRD46WWEXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRUGM2TIMZXHE32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4643543797>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGX7TUILCX6Q3HBNLSWAAL46WWEXAVCNFSM6AAAAACZYBFU56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DMNBTGU2DGNZZG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

DenisValeev · 2026-06-07T18:21:19Z

Pushed a cleanup pass to this branch.

What changed:

deleted docs/pr_title.txt
deleted docs/pr_body.md
fixed the Copilot nits in scripts/generate_cpu_experimental.py
- only prepend vendor/image-studio if it exists
- make RSS logging fail-safe when /proc/self/status is unavailable
- default --steps to 4 to match the validated/report path
- reject non-positive --steps early
fixed the example commands in both docs to include required --output
added scripts/start_cpu_image_server.sh so the warm CPU image path has a demo-shaped server entrypoint instead of only a raw Python module

On the runtime question: the path here is not fp16 on this ARM CPU box.

model family: bonsai-image-4B-ternary
transformer path for the stronger CPU runs: unpacked transformer sibling, not GemLite dense reconstruction
main inference dtype on this host: float32
text encoder: quantized on disk, then dequantized for prompt encoding on CPU

Heads-up on the warm timings: the faster numbers from the later server work are for cached prompt requests on a resident server. A brand-new uncached prompt still pays a large cold prompt-encode cost first on this CPU path, so warm cached latency and first-hit latency are very different. The warm-server helper now makes that split explicit.

If you want, I can do one more follow-up pass after this and fold the CPU server start path more tightly into the existing demo flow.

DenisValeev · 2026-06-07T20:53:06Z

For anything substantial it's pretty slow.

But as a fire and forget with a follow-up to telegram it may be of interest.

image 1024x1024 4-step total 1928.8s

seed: 3995096562
prewarm: 669.0s
restart_wait: 195.0s
render: 1031.8s
max_seq: 128

So about 2000 seconds end to end.

DenisValeev · 2026-06-07T20:58:16Z

For the above example run this:

python scripts/generate_cpu_experimental.py --prompt "Macro close-up of an iridescent peacock feather eye, emerald green and teal strands radiating outward, deep blue-black oval center, shimmering metallic texture, shallow depth of field, soft dark blurred background, glossy natural fibers, cinematic macro photography, high detail." --output outputs/cpu-peacock-feather.png --height 1024 --width 1024 --steps 4 --max-seq 128 --transformer-dir models/bonsai-image-4B-ternary-unpacked/transformer

DenisValeev · 2026-06-07T21:34:42Z

A lone samurai warrior in ornate black lacquer armor standing in mist, katana held low at his side, crimson silk cords and weathered metal plates, rain droplets glistening on the armor, stern shadowed face under a kabuto helmet, dramatic rim lighting, foggy bamboo forest background, shallow depth of field, cinematic composition, high detail, realistic historical texture, moody atmosphere, ultra-detailed photography style.

size: 1024x1024
steps: 4
seed: 2265673322
prewarm: 255.0s
restart_wait: 93.0s
render: 1031.6s
total: 1411.6s

khosravipasha · 2026-06-10T17:30:57Z

    "huggingface-hub",
+    # Experimental CPU bring-up dequantizes the HQQ text encoder on both macOS
+    # and Linux, so keep hqq available outside the Linux CUDA path too.
+    "hqq",


hqq runs ok on cpu? is it mostly just used as loading the checpoint?

khosravipasha · 2026-06-10T17:35:25Z


 [[package]]
 name = "accelerate"
 version = "1.13.0"


for the uv lock what's the main required changes?
I might just generate the uv lock myself after I merge this, too many changes to review.
Is the pyproject.toml enough to generate this? if yes could you remove this file from the diff. I can generate myself and push after.

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 3 comments.

+    in_channels_latents = transformer.config.in_channels // 4
+    h_lat = width // 8
+    w_lat = height // 8
+    noise_shape = (1, in_channels_latents * 4, h_lat // 2, w_lat // 2)


+    output_path = Path(req.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+


+class GenerateRequest(BaseModel):
+    prompt: str
+    output: str
+    width: int
+    height: int
+    steps: int
+    seed: int
+    guidance: float = 1.0
+    max_seq: int = 64
+


DenisValeev · 2026-06-10T17:39:12Z

I tried to repro my own work here in this branch on a mac with m1; failed miserably; tasked Codex to fix it all. It purportedly did in the latest commits. Per codex: Yep. HQQ here is mostly a loading/dequantization path for the text encoder and for getting the CPU path into a usable dense form; the validated CPU inference itself is not doing packed HQQ/GemLite kernel execution on CPU. On this host the working path ended up being float32 CPU inference with unpacked dense transformer weights. For `uv.lock`, the main required change is just reflecting the `pyproject.toml` dependency updates, especially keeping `hqq` available outside the Linux CUDA-only path because the experimental CPU bring-up dequantizes the HQQ text encoder on both macOS and Linux. If you prefer, `pyproject.toml` is enough for this PR and I can drop `uv.lock` from the diff so you can regenerate it after merge. Sincerely, Denis Valeev

…

On Wed, Jun 10, 2026 at 1:35 PM Pasha Khosravi ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In uv.lock <#20 (comment)> : > @@ -19,13 +19,13 @@ name = "accelerate" version = "1.13.0" for the uv lock what's the main required changes? I might just generate the uv lock myself after I merge this, too many changes to review. Is the pyproject.toml enough to generate this? if yes could you remove this file from the diff. I can generate myself and push after. — Reply to this email directly, view it on GitHub <#20?email_source=notifications&email_token=AAGX7TWBZZEFPZHFYQPKP2T47GL7JA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINBXGAYTMNZQGAY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4470167001>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGX7TXB3UOSVB2EEIMUW6347GL7JAVCNFSNUABGKJSXA33TNF2G64TZHMYTEMZQGI4DCMRRGQ5US43TOVSTWNBVG43DINBUG43DJILWAI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

DenisValeev · 2026-06-10T17:57:31Z

Codex: merged the overlapping CPU bring-up docs into docs/cpu-experimental.md, deleted the two redundant upstream draft docs, and pushed b15cea2 to dv/cpu-image-bringup-report Sincerely, Denis Valeev

…

On Wed, Jun 10, 2026 at 1:37 PM Pasha Khosravi ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In docs/cpu-experimental.md <#20 (comment)> : > @@ -0,0 +1,95 @@ +# Experimental CPU Bring-Up Thanks for the details doc :D now there is 3 similar ones, can we merge all in one and summarize them a bit. — Reply to this email directly, view it on GitHub <#20?email_source=notifications&email_token=AAGX7TQKRSUOBXAFMUOXPML47GMHJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINBXGAYTQMZTGQY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4470183341>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGX7TTGDDPYPIJHOJ6MSGT47GMHJAVCNFSNUABGKJSXA33TNF2G64TZHMYTEMZQGI4DCMRRGQ5US43TOVSTWNBVG43DINBUG43DJILWAI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

DenisValeev · 2026-06-10T23:47:03Z

./scripts/generate_cpu_low_memory.sh --prompt 'ostrich' --output outputs/cpu-ostrich-seed7.png --height 1024 --width 1024 --steps 4 --seed 7

DenisValeev · 2026-06-11T03:23:38Z

CPU bring-up update from this Mac:

The tiled ostrich output was not a VAE/display issue. The CPU unpack/export path decoded GemLite int2 weights with the wrong layout for ambiguous packed shapes.
GemLite stores uint8-packed weights over K with transpose, effectively (in_features / 4, out_features). The CPU decoder checked the direct layout first, so square projection weights like 3072 x 3072 and some packed to_out cases were written with the wrong orientation.
The transformer loaded and ran, but denoising stayed as structured latent noise, which decoded as the tiled texture.
The branch also had a prompt-cache regression: prompt embeds were trimmed to actual token count. Klein/Qwen3 expects the padded max-sequence embedding shape, so I restored padded prompt embeds and bumped the cache key.
On this 8 GB Apple Silicon Mac, dense GemLite reconstruction is not practical in memory. I rewrote the existing unpacked transformer artifact in place from the GemLite source state with the corrected layout.
CPU utilization was low because PyTorch was defaulting to fewer threads. The wrapper now exports CPU/BLAS thread env vars from the logical CPU count, and the Python runner defaults --threads to the host logical CPU count with --interop-threads 1.

Validation on this Mac:

./scripts/generate_cpu_low_memory.sh \
  --prompt 'ostrich' \
  --output outputs/cpu-ostrich-128.png \
  --height 128 \
  --width 128 \
  --steps 4 \
  --seed 7

Result: coherent 128x128 ostrich image, and scripts/check_image.py outputs/cpu-ostrich-128.png passes.

Pushed fix commit: 3c63a93 fix cpu gemlite unpack layout.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

+    payload = {
+        "prompt": prompt,
+        "output": str(output_path),
+        "width": width,
+        "height": height,
+        "steps": steps,
+        "seed": seed,
+        "guidance": guidance,
+        "max_seq": max_seq,
+    }


+if [ -s "$UNPACKED_ROOT/transformer/diffusion_pytorch_model.safetensors" ] || \
+   [ -s "$UNPACKED_ROOT/transformer/diffusion_pytorch_model.safetensors.index.json" ] || \
+   find "$UNPACKED_ROOT/transformer" -maxdepth 1 -type f -name 'diffusion_pytorch_model-*.safetensors' -size +0c | grep -q .; then


+    parser.add_argument("--dtype", default="float32")
+    parser.add_argument("--text-encoder-dtype", default="auto")
+    parser.add_argument("--threads", type=int, default=4)
+    parser.add_argument("--interop-threads", type=int, default=4)


DenisValeev added 4 commits June 3, 2026 02:25

docs: add CPU image generation bring-up report

7c36d10

docs: add upstream CPU bring-up issue draft

d8bd9b3

scripts: add experimental CPU image generation tools

39c7141

scripts: drop block permutation debug helper

47734f4

khosravipasha requested a review from Copilot June 6, 2026 01:18

Copilot started reviewing on behalf of khosravipasha June 6, 2026 01:18 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

scripts: add CPU prompt-cache and autoresearch tuning

058a03e

scripts: add warm CPU server benchmark helpers

950e2de

khosravipasha reviewed Jun 7, 2026

View reviewed changes

cpu image: clean up bringup PR

bee8015

DenisValeev added 3 commits June 10, 2026 00:27

cpu repro: document gemlite workflow

9375ac2

cpu: restore experimental path compatibility defaults

e5a4184

cpu: prefer validated unpacked transformer weights

a562d1f

khosravipasha reviewed Jun 10, 2026

View reviewed changes

khosravipasha requested a review from Copilot June 10, 2026 17:33

Copilot started reviewing on behalf of khosravipasha June 10, 2026 17:33 View session

khosravipasha reviewed Jun 10, 2026

View reviewed changes

Copilot AI reviewed Jun 10, 2026

View reviewed changes

khosravipasha reviewed Jun 10, 2026

View reviewed changes

Comment thread docs/cpu-experimental.md

DenisValeev added 3 commits June 10, 2026 17:41

build: drop uv lock from CPU bringup PR

5dcc000

cpu server: address review follow-ups

ec2ad28

docs: consolidate CPU bringup notes

b15cea2

DenisValeev added 2 commits June 10, 2026 18:34

docs: capture clean-room CPU bringup flow

c0b25ec

cpu: restore working GemLite decode path

70f3b76

DenisValeev and others added 3 commits June 11, 2026 00:40

cpu: install and verify mac export deps

b27b1bd

cpu: trim prompt encodes to actual token length

3ad2301

fix cpu gemlite unpack layout

3c63a93

Denis Valeev added 2 commits June 10, 2026 23:38

make cpu bringup reproducible

9cf8571

fit cpu prompt cache on 8gb mac

b5d6846

khosravipasha requested a review from Copilot June 12, 2026 01:36

Copilot started reviewing on behalf of khosravipasha June 12, 2026 01:37 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

cpu: address review follow-ups

74ecb6c

		if args.height % 32 != 0 or args.width % 32 != 0:
		raise SystemExit("height and width must be multiples of 32")

		output_path = Path(req.output)
		output_path.parent.mkdir(parents=True, exist_ok=True)

Uh oh!

Conversation

DenisValeev commented Jun 3, 2026

Summary

Why this is useful

Guidance captured in the report

Notes

Uh oh!

khosravipasha commented Jun 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

DenisValeev commented Jun 6, 2026

Uh oh!

DenisValeev commented Jun 6, 2026

Uh oh!

DenisValeev commented Jun 6, 2026

Uh oh!

khosravipasha Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

khosravipasha Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

khosravipasha commented Jun 7, 2026

Uh oh!

DenisValeev commented Jun 7, 2026 via email

Uh oh!

DenisValeev commented Jun 7, 2026

Uh oh!

DenisValeev commented Jun 7, 2026

Uh oh!

DenisValeev commented Jun 7, 2026

Uh oh!

DenisValeev commented Jun 7, 2026

Uh oh!

khosravipasha Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

khosravipasha Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

DenisValeev commented Jun 10, 2026 via email

Uh oh!

DenisValeev commented Jun 10, 2026 via email

Uh oh!

DenisValeev commented Jun 10, 2026

Uh oh!

DenisValeev commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DenisValeev commented Jun 11, 2026 •

edited

Loading