[bugfix] Add kv_transfer_params to prefill and generation requests #63
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JohnLiu97Huawei
force-pushed
the
bugfix-p2p
branch
2 times, most recently
from
7b627fd to
081e7e0
Compare
JohnLiu97Huawei
force-pushed
the
bugfix-p2p
branch
from
081e7e0 to
e4621ef
Compare
JohnLiu97Huawei
force-pushed
the
bugfix-p2p
branch
2 times, most recently
from
9b89178 to
5b26413
Compare
Contributor
Author
Self Test ResultUsing the P2P mooncake kv connector, the inference is successfully executed under the deployment of 1E 1P 1D pattern. looking up the node log we can see "I20251219 15:17:36.256863 281473336216192 ascend_direct_transport.cpp:358] AscendDirectTransport register mem addr:0x12c607200000, length:290258944, location:*, mem type:0" as evidence of the successful kv transfer. |
Introduces kv_transfer_params to GenerationResponse and ensures it is set for P_INSTANCE server types. Updates request handling and response processing to propagate kv_transfer_params, enabling remote decode and prefill control. besides, the bug of duplicated D instance init process is fixed. Signed-off-by: John Liu BUAA <[email protected]>
JohnLiu97Huawei
force-pushed
the
bugfix-p2p
branch
from
5b26413 to
b549ab9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request updates the
disagg_worker.pylogic to support key-value (KV) transfer parameters for distributed inference requests. Specifically, it ensures that when the worker is configured for KV transfer, both prefill and generation requests include the necessary parameters to enable remote decoding.Enhancements for distributed inference:
_handle_requestmethod ofdisagg_worker.py, added logic to injectkv_transfer_paramsintosampling_paramsfor both prefill and generation requests whenkv_transfer_configis set, enabling remote decode functionality.