[Bug] High VRAM usage (23.5GB) at startup with acestep-5Hz-lm-4B despite offload_to_cpu = true

[devsynck]

Describe the bug
When launching the ACE-Step-1.5 backend with the 4B model and CPU offloading enabled, the application immediately consumes ~23.5GB of VRAM (on an RTX 4090) upon startup. This occurs effectively at an idle state, before any generation tasks have been initiated. The offload_to_cpu setting appears to be ignored or ineffective, as the model seems to be fully loaded into GPU memory, nearly maxing out the 24GB capacity.

To Reproduce
Steps to reproduce the behavior:

1. Configure the backend (acestep/gpu_config.py or via launch arguments) with:
   • LM Backend: PT
   • LM Model: acestep-5Hz-lm-4B
   • offload_to_cpu: true
2. Start the application (e.g., uv run acestep or start_gradio_ui.bat).
3. Open Windows Task Manager and navigate to the Performance tab for the GPU.
4. Observe that Dedicated GPU memory usage stays at ~23.5 GB even while the system is idle.

Expected behavior
With offload_to_cpu = true, VRAM usage at idle should be significantly lower.

Screenshots

Desktop (please complete the following information):

• OS: Windows 11
• GPU: NVIDIA GeForce RTX 4090 (24GB)
• Python Version: 3.11
• PyTorch Version: 2.7.1 (CUDA 12.8)

Additional context

• The issue occurs immediately upon model initialization at startup.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• OOM with API server at model init #166

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devsynck]

Basically, it should not be using that much VRAM in idle.

[1larity]

Even with CPU offloading enabled, you can still see VRAM used during init because:

Model weights may be instantiated/loaded on GPU briefly before being moved/offloaded.
Some tensors (for example persistent working tensors/caches) can remain on device by design.
CUDA allocator may keep reserved memory after transfers, so “used/reserved” looks non-zero.
So “CPU offload enabled” does not necessarily mean “0 VRAM at init.”

[devsynck]

I don't expect it to be 0. It's using 100% at init currently. Is that normal?

[1larity]

To help determine whether this is expected behaviour or a bug, please share:

config_path / model variant used (turbo, base, etc.)
Device (cuda, xpu, mps, cpu)
Offload settings:
offload_to_cpu
offload_dit_to_cpu
Quantization setting (on/off, mode)
VRAM readings:
before init
right after init
during first generation
Full init log output
If possible, please also test this matrix and report results:

A: offload_to_cpu=true, offload_dit_to_cpu=true
B: offload_to_cpu=true, offload_dit_to_cpu=false
C: offload_to_cpu=false (baseline)

That comparison will confirm whether offload behaviour matches design and identify any unintended VRAM retention. Please be aware as this is not a negatively effecting behaviour for Ace-step, it still performs it's core functions, a resolution if it is a bug, may not be swift.

[robustini]

Same issue here.
Until the day before yesterday I was generating songs with about 24 GB of VRAM and that was it, but now it swaps to RAM.