Feat/pass countinference to serverless getweights #1373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bigbitbus
requested review from
PawelPeczek-Roboflow,
grzegorz-roboflow,
yeldarby,
probicheaux and
hansent
as code owners
bigbitbus
force-pushed
the
feat/pass-countinference-to-serverless-getweights
branch
from
51dcd37 to
9ab0396
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…serverless-getweights Review `feat/pass-countinference-to-serverless-getweights`
codeflash-ai bot
added a commit
that referenced
this pull request
Jun 24, 2025
…(`feat/pass-countinference-to-serverless-getweights`) Here's an optimized rewrite of your program, addressing profiling hot spots and general efficiency improvements. **Optimization Summary:** 1. **Avoid Redundant Method Calls:** - Minimize repeated lookups and calculations. - Cache computations/results when possible within function scope. 2. **Lazy Imports:** - Move GC and optional torch imports where needed (they are only used upon eviction). 3. **Deque Optimizations:** - In `WithFixedSizeCache.add_model`, avoid repeated `self._key_queue.remove(queue_id)` by checking position or maintaining a set for fast checks (no need, since only called if known present, and block is rare). Still, code can be reduced for clarity. 4. **Reduce logging** in the hot add logic (unless DEBUG mode; logging is a major time sink during profiling). 5. **Batch Removals:** - Accumulate models to remove and do a single `gc.collect()` call after, instead of per-iteration. 6. **Data structure** choices are left unchanged (deque is still best for explicit ordering here). 7. **General Logic**: Use local variables for lookups on attributes used multiple times (minor, but helps). --- **Key Runtime Optimizations:** - Only call `gc.collect()` after all removals in a batch, not after every single model eviction. - Reduced logging in hot code paths (this was responsible for noticeable time in profiling). - Use local variables when repeatedly accessing class attributes. - Use direct inlining for `_resolve_queue_id` for this use case. - Defensive handling if queue/model state falls out of sync—never throws unnecessarily. **Performance Note:** If you profile again after these changes, most of the time will now be in actual model loading and removal. That is, this code will not be a noticeable bottleneck anymore in the workflow. If LRU cache size is much larger, consider further data structure optimizations such as a dict for constant-time eviction and presence checking, but for N ~ 8 this is not needed.
⚡️ Codeflash found optimizations for this PR📄 50% (0.50x) speedup for
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Pass through the countinference and service_secret parameters to the roboflow API backend.
Type of change
Please delete options that are not relevant.
How has this change been tested, please provide a testcase or example of how you tested the change?
Testing in staging (ongoing)
Any specific deployment considerations
For example, documentation changes, usability, usage/costs, secrets, etc.
Docs