Extend timeout in Lemonade Copilot extension

[stormenergy91]

I am trying models 70B+ with lemonade copilot extension on Strix Halo and Copilot go in timeout while lemonade is still generating the response. There is any way to extend the timeout?

[jeremyfowers]

Hi @stormenergy91 sorry you are running into this problem.

The timeout for generation is infinite:

lemonade/src/cpp/include/lemon/wrapped_server.h

Line 54 in 608c74b

static constexpr long INFERENCE_TIMEOUT_SECONDS = 0;

Can you provide some logs showing that Lemonade is timing out?

Or is the copilot extension itself timing out?

[stormenergy91]

Thanks @jeremyfowers, I think is something is vs side (or maybe in the extension?). Lemonade is working, in the logs i see something

Nov 31 21:53:59 lemonade[1110649]: slot update_slots: id  3 | task 0 | n_tokens = 22528, memory_seq_rm [22528, end)
Nov 31 21:53:59 lemonade[1110649]: slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 24576, batch.n_tokens = 2048, progress = 0.552270

But in the Copilot chat i get an error message. I'll try to get it again and then add here.

That's why I was wondering if there's a way to extend the timeout. For example, in OpenHands you can set the LLM_TIMEOUT environment variable, and maybe there's something similar for the Lemonade copilot extension.

[jeremyfowers]

If the client was timing out I would expect Lemonade to say something about the connection being terminated.

[stormenergy91]

I make a test right now:

In the Copilot Chat:

The request failed. Please try again.
Copilot request ID: 16b6aac4-7ba7-4e92-b7db-8538668e5517
Reason: terminated: TypeError: terminated at Fetch.onAborted (node:internal/deps/undici/undici:11132:53) at Fetch.emit (node:events:519:28) at Fetch.terminate (node:internal/deps/undici/undici:10290:14) at Object.onError (node:internal/deps/undici/undici:11253:38) at Request.onError (/usr/share/code/resources/app/node_modules/undici/lib/core/request.js:301:27) at Object.errorRequest (/usr/share/code/resources/app/node_modules/undici/lib/core/util.js:836:13) at Socket.onHttpSocketClose (/usr/share/code/resources/app/node_modules/undici/lib/dispatcher/client-h1.js:939:10) at Socket.emit (node:events:531:35) at TCP.<anonymous> (node:net:346:12) at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

In the lemoande logs

Dec 02 07:29:19 lemonade[1746178]: slot launch_slot_: id  2 | task 68889 | processing task
Dec 02 07:29:19 lemonade[1746178]: slot update_slots: id  2 | task 68889 | new prompt, n_ctx_slot = 128000, n_keep = 16, task.n_tokens = 14240
Dec 02 07:29:19 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 5, memory_seq_rm [5, end)
Dec 02 07:29:19 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 2053, batch.n_tokens = 2048, progress = 0.144171
Dec 02 07:30:17 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 2053, memory_seq_rm [2053, end)
Dec 02 07:30:17 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 4101, batch.n_tokens = 2048, progress = 0.287992
Dec 02 07:31:24 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 4101, memory_seq_rm [4101, end)
Dec 02 07:31:24 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 6149, batch.n_tokens = 2048, progress = 0.431812
Dec 02 07:32:36 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 6149, memory_seq_rm [6149, end)
Dec 02 07:32:36 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 8197, batch.n_tokens = 2048, progress = 0.575632
Dec 02 07:33:53 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 8197, memory_seq_rm [8197, end)
Dec 02 07:33:53 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 10245, batch.n_tokens = 2048, progress = 0.719452
Dec 02 07:35:14 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 10245, memory_seq_rm [10245, end)
Dec 02 07:35:14 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 12293, batch.n_tokens = 2048, progress = 0.863272
Dec 02 07:36:40 lemonade[1746178]: slot update_slots: id  2 | task 68889 | n_tokens = 12293, memory_seq_rm [12293, end)
Dec 02 07:36:40 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt processing progress, n_tokens = 14240, batch.n_tokens = 1947, progress = 1.000000
Dec 02 07:36:40 lemonade[1746178]: slot update_slots: id  2 | task 68889 | prompt done, n_tokens = 14240, batch.n_tokens = 1947
Dec 02 07:38:12 lemonade[1746178]: [HttpClient] Stream write stopped (client disconnected or write error)
Dec 02 07:38:12 lemonade[1746178]: [StreamingProxy] WARNING: Backend did not send [DONE] marker, adding it
Dec 02 07:38:12 lemonade[1746178]: [Server] Streaming completed - 200 OK
Dec 02 07:38:12 lemonade[1746178]: [Server] POST /api/v1/chat/completions - 200
Dec 02 07:38:13 lemonade[1746178]: srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
Dec 02 07:38:13 lemonade[1746178]: srv          stop: cancel task, id_task = 68889
Dec 02 07:38:13 lemonade[1746178]: slot      release: id  2 | task 68889 | stop processing: n_tokens = 14242, truncated = 0
Dec 02 07:38:13 lemonade[1746178]: srv  update_slots: all slots are idle

[jeremyfowers]

@danielholanda can you please take a look?

User is driving GitHub Copilot with some large models (>70B) and prompts (>10k). It appears copilot is timing out because Lemonade itself is saying “[HttpClient] Stream write stopped (client disconnected or write error)”

[danielholanda]

Thanks for reporting this! I will analyze our VS code extension to see if there is something we can do there to avoid this timeout.

[stormenergy91]

Hi @danielholanda , I recorded a video so you can see exactly what happens. The timeout occurs when Copilot tries to read long files, but the lemonade server is still running (task 266). The video should help you understand the timing.

If I can help, feel free to ask. However, I noticed there isn’t a public repo for the extension.

lemonade_copilot.mp4

[danielholanda]

Aha! This vide actually really helps me to understand what is going on here.
I can confirm this does not look like a timeout issue, since we are often prompting the LLM and each response only takes a few seconds.

I've seen a very similar message in the past when either the model or Lemonade itself was running out of context size. The fact that you are often seeing this when working with larger files also makes this theory sound more plausible.

Which context size are you using? Can you try 32 or 64k? You can adjust it by either selecting it on your Tray app (Windows) or using the --ctx-size param when running lemonade-server serve

Let me know if that works for you!

[stormenergy91]

This is my actual configuration:

• Strix Halo with 128GB di RAM and this Grub settings GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=33554432"
• Ubuntu 24.04
• Lemonade installed with the .deb package
• LEMONADE_LLAMACPP_ARGS="--no-mmap --no-warmup"
• LEMONADE_CTX_SIZE=128000

I also tested with gpt-oss-120B but the same error.

[jeremyfowers]

I watched through the video and make these observations:

1. llama-server clearly says context size is set to 128k.
2. Then it spends 13 minutes (!) on prompt processing. Which I guess makes sense for a 235B model with a 32k token input.
3. "Client disconnected" about 40 seconds after prompt processing completes. llama-server says it decided to cancel the task.

My next theory is that the system is running out of RAM for the KV-Cache, so llama-server calls it quits.

@stormenergy91 any chance you could keep an eye on RAM usage over time as this runs?

[stormenergy91]

I make two new test (same settings and same prompt).
In the first I got the error but in the second no (i stopped manually the process).

The difference is that in the first there was a single long task (task 219) but in the second it never generate a long task and it go on.

In the two test i update lemonade to the 9.0.6 and in the video you can see radeontop and the lemonade logs.

I hope this can help.

lemonade-qwen3-test1-fast-lw.mp4

lemonade-qwen3-test2-fast-lw.mp4