then timed out in Llamafile class method chat model still works - how stop it?

[Direct-Dev-Ru]

Hello, everyones!

this is my fast api code to aceess api running with llamafile

import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from llama_index.llms.llamafile import Llamafile
from llama_index.core.llms import ChatMessage
from openai import OpenAI
import requests

app = FastAPI()
# Default to localhost if
base_url = os.getenv("BASE_URL", "http://localhost:9999")
api_key = os.getenv("OPENAI_API_KEY", "sk-no-key-required")


@app.post("/llm-chat")
async def chat(request: ChatRequest):
    try:
        llm = Llamafile(temperature=request.temperature, seed=request.seed,
                        base_url=base_url, request_timeout=request.timeout)
        messages = [
            ChatMessage(
                role="system",
                content=request.system_content,
            ),
            ChatMessage(
                role="user",
                content=request.user_content,
            ),
        ]        
        resp = llm.chat(messages)                
        return {"response": resp}

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

then i run this fastapi server on localhost:8000

the i do curl
curl -X POST "http://localhost:8000/llm-chat/" -H "Content-Type: application/json" -d '{"system_conte
nt": "Pretend you are an ai assistant with a big knowledge base.", "user_content": "What is the best first programming language to study?", "temperature": 0.
0, "seed": 1556, "timeout": 60}' | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   224    0    22  100   202      0      3  0:01:07  0:01:00  0:00:07     4
{
  "detail": "timed out"
}

ok i got timeout on client but my model (llamafile) stiil work and send response to black hole

Oct 24 13:50:57 stf20-pipve--01 bash[3638406]: {"function":"launch_slot_with_data","level":"INFO","line":886,"msg":"slot is processing task","slot_id":0,"task_id":1226,"tid":"11681088","timestamp":1729756257}
Oct 24 13:50:57 stf20-pipve--01 bash[3638406]: {"function":"update_slots","level":"INFO","line":1912,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":1226,"tid":"11681088","timestamp":1729756257}
Oct 24 13:53:59 stf20-pipve--01 bash[3638406]: {"function":"print_timings","level":"INFO","line":313,"msg":"prompt eval time     =    4570.54 ms /    33 tokens (  138.50 ms per token,     7.22 tokens per second)","n_tokens_second":7.22015973618849,"num_prompt_tokens_processed":33,"slot_id":0,"t_prompt_processing":4570.536,"t_token":138.50109090909092,"task_id":1226,"tid":"11681088","timestamp":1729756439}
Oct 24 13:53:59 stf20-pipve--01 bash[3638406]: {"function":"print_timings","level":"INFO","line":327,"msg":"generation eval time =  177696.00 ms /   314 runs   (  565.91 ms per token,     1.77 tokens per second)","n_decoded":314,"n_tokens_second":1.767062878743408,"slot_id":0,"t_token":565.9108184713376,"t_token_generation":177695.997,"task_id":1226,"tid":"11681088","timestamp":1729756439}
Oct 24 13:53:59 stf20-pipve--01 bash[3638406]: {"function":"print_timings","level":"INFO","line":337,"msg":"          total time =  182266.53 ms","slot_id":0,"t_prompt_processing":4570.536,"t_token_generation":177695.997,"t_total":182266.533,"task_id":1226,"tid":"11681088","timestamp":1729756439}
Oct 24 13:53:59 stf20-pipve--01 bash[3638406]: {"function":"update_slots","level":"INFO","line":1723,"msg":"slot released","n_cache_tokens":347,"n_ctx":8192,"n_past":346,"n_system_tokens":0,"slot_id":0,"task_id":1226,"tid":"11681088","timestamp":1729756439,"truncated":false}
Oct 24 13:53:59 stf20-pipve--01 bash[3638406]: {"function":"log_server_request","level":"INFO","line":2766,"method":"POST","msg":"request","params":{},"path":"/v1/chat/completions","remote_addr":"192.168.xx.xx","remote_port":60746,"status":200,"tid":"140321560006880","timestamp":1729756439}

how i can stop execution and release slot on timeout?