Fix Falcon-Mamba long prompt resource exhaustion - #1656
Merged
Conversation
mrjoema
marked this pull request as ready for review
August 4, 2026 11:01
michalk8
self-requested a review
August 5, 2026 13:22
michalk8
approved these changes
Aug 5, 2026
michalk8
left a comment
Collaborator
There was a problem hiding this comment.
I tested this locally with a prompt The quick brown fox jumps over the lazy dog. " * 2000, this PR fixes it.
LGTM, thanks a lot!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reproduction
mlx==0.32.0,mlx-lm==0.31.3mlx-community/Falcon3-Mamba-7B-Instruct-8bits[metal::malloc] Resource limit (499000) exceeded.Root cause and fix
Falcon-Mamba normalized
delta,B, andCtwice, and each call allocated anmx.onesscale. The lazy graph therefore accumulated six temporary scales per token per layer and exceeded Metal's resource-count limit.Normalize each value once with
weight=None. The new code also replaces the nestedmap/lambdaexpression with an explicit split and three normalization calls for readability.Verification
Fixes #1637