[Runtime] solve the problem that llama frequently calls autotuner #332
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
StrongSpoon
changed the title
tongxin
reviewed
Nov 27, 2024
src/flag_gems/utils/libentry.py
Outdated
Comment on lines
57
to
96
| connect = sqlite3.connect(self.cache_dir) | ||
| c = connect.cursor() | ||
| c.execute(f"CREATE TABLE IF NOT EXISTS {self.__name__} (key TEXT, config TEXT)") | ||
| cursor = c.execute(f"SELECT key, config from {self.__name__}") | ||
|
|
||
| for row in cursor: | ||
| key_str, config_str = row | ||
| key = (self.getvalue(k) for k in key_str[1:-1].split(", ")) | ||
|
|
||
| cfg_ls = [item.split(": ") for item in config_str.split(", ")] | ||
| config = triton.Config({}) | ||
| for k, v in cfg_ls[:-4]: | ||
| config.kwargs[k] = eval(v) | ||
| config.num_warps = int(cfg_ls[-4][1]) | ||
| config.num_ctas = int(cfg_ls[-3][1]) | ||
| config.num_stages = int(cfg_ls[-2][1]) | ||
| config.enable_fp_fusion = eval(cfg_ls[-1][1]) | ||
|
|
||
| self.cache[key] = config | ||
|
|
||
| connect.close() |
There was a problem hiding this comment.
Probably we could leave the connection open and shared because we need to fetch stored params for all gems ops. A try catch block is possibly necessary for external IO operations.
There was a problem hiding this comment.
But I think its fine at this stage. Let's improve it later.
There was a problem hiding this comment.
there is not always a explicit exit of flag_gems. I'm not sure what time to close the connection if keep it open.
iclementine
reviewed
Nov 27, 2024
|
|
||
|
|
||
| def config_cache_dir() -> Path: | ||
| _config_cache_dir = cache_dir() / "config_cache" |
There was a problem hiding this comment.
Suggestion for name: "tunning_cache"
There was a problem hiding this comment.
what's the difference?
iclementine
reviewed
Nov 27, 2024
kiddyjinjin
reviewed
Dec 12, 2024
* new feature, muti_backend * update auto_tune_module * update auto_tune_module * update auto_tune_module * update __init__ * rebase * fix bug * modifiy auto_tune_config * fix bug * fix bug * update * update * update scatter&gather * fix auto_tune * add gen_torch_device_fn * fix codestyle * fix codestyle * Modify code based on comments * Modify gen_impl with loops instead of recursion * Update code structure * Polish code * update * Polish code * Modify code based on comments * modify based on comment * Modify code based on comments * update * final fix
kiddyjinjin
approved these changes
Dec 19, 2024
Gxiandy
pushed a commit
to Gxiandy/FlagGems
that referenced
this pull request
Jan 12, 2025
…agOpen#332) * [operator] turn autotuner to heuristics * [operator] heuristics for gather & index_select * [runtime] libtuner for matmul * [runtime] store config data in one db * [bugfix] parse key as list instead of tuple * [no ci] update var name * [Muti_backend] muti_backend-part_1-framework-and-tune_config (FlagOpen#294) * new feature, muti_backend * update auto_tune_module * update auto_tune_module * update auto_tune_module * update __init__ * rebase * fix bug * modifiy auto_tune_config * fix bug * fix bug * update * update * update scatter&gather * fix auto_tune * add gen_torch_device_fn * fix codestyle * fix codestyle * Modify code based on comments * Modify gen_impl with loops instead of recursion * Update code structure * Polish code * update * Polish code * Modify code based on comments * modify based on comment * Modify code based on comments * update * final fix * [bugfix] update libtuner to be compatible with triton2 * [no ci]reformat * [operator] update log_softmax * [pretune] move pretune to ./examples for models * [format] delete useless print * [format] delete unused import * [format] [no ci] remove useless print --------- Co-authored-by: Galaxy1458 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Category
Runtime
Type of Change
New Feature
Description
implement LibTuner to take place of Autotuner.
it stores tuned configs in database and preloads them before calling.
Issue
Progress
Performance