Skip to content

Fix Falcon-Mamba long prompt resource exhaustion - #1656

Merged
michalk8 merged 2 commits into
ml-explore:mainfrom
mrjoema:joema-1st-pr
Aug 5, 2026
Merged

Fix Falcon-Mamba long prompt resource exhaustion#1656
michalk8 merged 2 commits into
ml-explore:mainfrom
mrjoema:joema-1st-pr

Conversation

@mrjoema

@mrjoema mrjoema commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Reproduction

  • Apple M4 Pro (48 GB), macOS 27.0, Python 3.13.5
  • mlx==0.32.0, mlx-lm==0.31.3
  • mlx-community/Falcon3-Mamba-7B-Instruct-8bits
  • A 1,298-token effective prompt fails with [metal::malloc] Resource limit (499000) exceeded.

Root cause and fix

Falcon-Mamba normalized delta, B, and C twice, and each call allocated an mx.ones scale. The lazy graph therefore accumulated six temporary scales per token per layer and exceeded Metal's resource-count limit.

Normalize each value once with weight=None. The new code also replaces the nested map/lambda expression with an explicit split and three normalization calls for readability.

Verification

  • Full model tests: 78 passed, 1 skipped
  • Real-model prompts of 1,909 and 4,105 effective tokens complete successfully
  • A regular 50-token greedy generation exactly matches the pre-fix output

Fixes #1637

@mrjoema
mrjoema marked this pull request as ready for review August 4, 2026 11:01
@michalk8
michalk8 self-requested a review August 5, 2026 13:22

@michalk8 michalk8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this locally with a prompt The quick brown fox jumps over the lazy dog. " * 2000, this PR fixes it.
LGTM, thanks a lot!

@michalk8
michalk8 merged commit 254d153 into ml-explore:main Aug 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mamba2 models (mlx_lm.generate and mlx_lm.server) crash/hang silently on prompts beyond ~1-2k tokens

2 participants