How to quickly obtain interface services in openai and claude formats

[Yelart]

How to quickly obtain interface services in openai and claude formats.
I will use that in my production

[davide221]

Hello! Thanks for trying out Lucebox! We have a fastAPI server, both formats dflash/scripts/server.py

• OpenAI: POST /v1/chat/completions
• Anthropic: POST /v1/messages

Terminal instructions:
hf download Lucebox/Qwen3.6-27B-DFlash-GGUF
dflash-draft-3.6-q8_0.gguf --local-dir models/draft/

hf download unsloth/Qwen3.6-27B-GGUF
Qwen3.6-27B-Q4_K_M.gguf --local-dir models/

python3 dflash/scripts/server.py
--target models/Qwen3.6-27B-Q4_K_M.gguf
--draft models/draft/dflash-draft-3.6-q8_0.gguf
--port 8000

Point Open WebUI / Hermes / OpenClaw etc at http://host:8000/v1. Tool calling → server_tools.py (same routes)

[slavap]

@davide221
Can I use Qwen3.6-27B-Q8_0.gguf or Qwen3.6-27B-Q6_K.gguf instead of Qwen3.6-27B-Q4_K_M.gguf ?

Also, it is not clear to me why it's complaining about PyTorch? Looks like it answers in WebUI with decent speed, but only 512 tokens, then cut off.
And what pflash = off means? How to define model temperature and thinking mode?

I was following steps from https://www.lucebox.com/blog/amd
The only difference that PR 119 is already in main branch.

LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH DFLASH_BIN=$PWD/build/test_dflash DFLASH27B_DRAFT_SWA=2048 DFLASH27B_PREFILL_UBATCH=512   python3 scripts/server.py --target models/Qwen3.6-27B-Q4_K_M.gguf --draft models/draft/dflash-draft-3.6-q8_0.gguf --port 8082
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[pc] chat markers: family=qwen sys_seq=[248045, 8678]… end_seqs=[[248046]] next_seqs=[[248045]]
/home/sl/temp/lucebox-hub/dflash/scripts/server.py:717: DeprecationWarning: 
        on_event is deprecated, use lifespan event handlers instead.

        Read more about it in the
        [FastAPI docs for Lifespan Events](https://fastapi.tiangolo.com/advanced/events/).
        
  @app.on_event("startup")
Luce DFlash OpenAI server on http://0.0.0.0:8082
  arch      = unknown
  target    = models/Qwen3.6-27B-Q4_K_M.gguf
  draft     = models/draft/dflash-draft-3.6-q8_0.gguf
  bin       = /home/sl/temp/lucebox-hub/dflash/build/test_dflash
  budget    = 22
  max_ctx   = 16384
  tokenizer = Qwen/Qwen3.5-27B
  pflash    = off
INFO:     Started server process [22223]
INFO:     Waiting for application startup.
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 126976 MiB):
  Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 126976 MiB
  [daemon] [cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=22 temp=1.00 chain_seed=1 fa_window=2048 draft_swa=2048 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0
  [daemon] [loader] eos_id=248046 eos_chat_id=-1
  [daemon] [target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
  [daemon] [draft]  loaded
  [daemon] [draft]  SWA layers: 4/5 (window=2048)
  [daemon] [daemon] ready
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8082 (Press CTRL+C to quit)

[jkyamog]

Also, it is not clear to me why it's complaining about PyTorch? Looks like it answers in WebUI with decent speed, but only 512 tokens, then cut off. And what pflash = off means? How to define model temperature and thinking mode?

Please test my PR #165 it maybe related to your issue of only getting 512 tokens and then a cut off.

[slavap]

@jkyamog
nope, still the same, cut off at 512 tokens. Should I change something in server cmd:

LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH DFLASH_BIN=$PWD/build/test_dflash DFLASH27B_DRAFT_SWA=2048 DFLASH27B_PREFILL_UBATCH=512 \
venv/bin/python3 scripts/server.py --target models/Qwen3.6-27B-Q4_K_M.gguf --draft models/draft/dflash-draft-3.6-q8_0.gguf --port 8082

[davide221]

@slavap can you try my latest PR #178 to see if it solve the problem? It seems to work for me on OpenWebUI

[slavap]

@davide221 yes, PR 178 helps. Not clear why I have to set env variable though, I had to add FLASH_DEFAULT_MAX_TOKENS=16384

Still the following questions remain:

• Can I use Qwen3.6-27B-Q8_0.gguf or Qwen3.6-27B-Q6_K.gguf instead of Qwen3.6-27B-Q4_K_M.gguf ?
• Do I need PyTorch installed or not?
• How to define model temperature and thinking mode?

[davide221]

@slavap glad #178 fixed it.

1. Q6_K / Q8_0 work just swap --target. Accept rate should be similar if not better.
2. PyTorch not needed. Transformers only loads tokenizer.
3. Temperature: "temperature": 0.7 in request body. Thinking enable per-request via chat_template_kwargs {"enable_thinking": true}.