Skip to content

Fix bug in prefill_chunk_size that ignores disable_compile flag #38067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

xmarva
Copy link

@xmarva xmarva commented May 11, 2025

This PR fixes a bug in the prefill_chunking function where the compilation check is not performed before calling get_compiled_call().

When using prefill_chunk_size > 0 in a GenerationConfig, the model's forward function is always compiled, even if disable_compile=True is specified. This happens because the prefill_chunking function directly calls get_compiled_call() without checking if compilation should occur:

model_forward = self.get_compiled_call(generation_config.compile_config)

Modified to:

compile_forward = self._valid_auto_compile_criteria(model_kwargs, generation_config)
if compile_forward:
    model_forward = self.get_compiled_call(generation_config.compile_config)

This ensures that models aren't compiled when disable_compile=True is set.

Also fixed a typo in the error message ("chunkink" -> "chunking"), lol

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
    Yes, but this is still my first PR here, hope it's ok
  • Was this discussed/approved via a Github issue or the forum?
  • Did you make sure to update the documentation with your changes?
    As far as I understand it, there's no need to change the documentation
  • Did you write any new necessary tests?
    No, tested with a simple generation script using prefill_chunk_size=8 and disable_compile=True. Before the fix, torch.compile was being called despite disable_compile=True. After the fix, no compilation occurs as expected.

Who can review?

@gante

Copy link

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@github-actions github-actions bot marked this pull request as draft May 11, 2025 02:36
@xmarva xmarva marked this pull request as ready for review May 11, 2025 02:38
@github-actions github-actions bot requested a review from gante May 11, 2025 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant