Skip to content

[Bugfix]fix the thread lock when user input same id#585

Open
GGBond8488 wants to merge 1 commit intotriton-inference-server:mainfrom
GGBond8488:Bugfix]fix_the_thread_lock_when_receive_same_id
Open

[Bugfix]fix the thread lock when user input same id#585
GGBond8488 wants to merge 1 commit intotriton-inference-server:mainfrom
GGBond8488:Bugfix]fix_the_thread_lock_when_receive_same_id

Conversation

@GGBond8488
Copy link
Copy Markdown

@GGBond8488 GGBond8488 commented Aug 27, 2024

deploy with the IFB, when user input the paylod as follow:

    {
        "text_input": str(question),
        "max_tokens": 512,
        "bad_words": "",
        "stop_words": stop_words,
        "pad_id": pad_id,
        "end_id": end_id,
        "top_p": 1,
        "id": "ggbond_test",
        "temperature": 0.0000001
    }

if every pyload‘s id is same, will cause the error:

Exception in thread Thread-1 (awaiter_loop):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
self._target(*self._args, **self._kwargs)
File "/code/model_pipeline_name/tensorrt_llm/1/model.py", line 920, in awaiter_loop
del self.triton_user_id_to_req_ids[
KeyError: 'ggbond_test'

@GGBond8488 GGBond8488 changed the title fix the thread lock when user input same id [Bugfix]fix the thread lock when user input same id Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant