Output texts keeps on repeating infinitely [Tried with llama.cpp server running on ubuntu 22.04]

[masifamu]

I have been trying to extract texts from the PDF but the output keeps on repeating, seems like the model is stuck somewhere. I am writing here to get some help on this, I don't understand the intricate details of unlimited OCR/llama.cpp setups. If someone has already gone through this, please help.

I am using it on CPU only Dell Optiplex 7060 with 16GB RAM setup. Let me know if I need to put more info here. I have attached the logs as well for the reference.

Thanks.

logs.txt

[nitinrajg]

Hi @masifamu! This continuous repeating loop happens when llama-server runs without explicit stop sequences or deterministic sampling parameters.

Here are a few steps to fix this:

1. Add Stop Sequences in Request/Server:
   Pass stop tokens in your API request payload so the server halts generation upon completion:

   {
     "max_tokens": 4096,
     "temperature": 0.0,
     "stop": ["<|endoftext|>", "<|im_end|>"]
   }

2. Set Low/Zero Temperature & Repetition Penalty:
   Set --temp 0 (greedy decoding) and --repeat-penalty 1.1 in your llama-server startup command or API options to prevent autoregressive token loops.

3. Use Expected Prompt Formats:
   Ensure your prompt includes proper prefix tokens:

   • Structured layout/boxes: <|grounding|>Convert the document to markdown.
   • Plain text: Free OCR.

[masifamu]

Thank you very much for the response and these useful suggestions.

1. This is how I am using the JSON on the llama-server localhost webUI. Is this how am I supposed to use this JSON?
2. In the server command I have included these two args as well --temp 0 & --repeat-penalty 1.1.
3. Tried with <|grounding|>Convert the document to markdown.
   OR
   Free OCR.

These improved the performance of the model a little, but I still see the repetition, not as often as without these suggestions.
Is there anything else that I should try to make it more reliable?

[nitinrajg]

Glad the previous suggestion helped you somewhat!

To completely resolve the remaining repetition, try these targeted adjustments:

1. Expand Context Window & Penalty Range on Server Launch
By default, llama-server uses a small context size and scans only the last 64 tokens for repetition. When full PDF pages exceed this, the server gets stuck in loops. Start your server with:

./llama-server -m <model.gguf> --mmproj <mmproj.gguf> -c 8192 --repeat-last-n 512 --repeat-penalty 1.2 --temp 0

2. Update the WebUI JSON Configuration
Yes, you placed the JSON in the right box in the webUI screenshot. Update it to include broader stop sequences and frequency penalties:

{
  "max_tokens": 4096,
  "temperature": 0.0,
  "repeat_penalty": 1.2,
  "repeat_last_n": 512,
  "frequency_penalty": 0.3,
  "presence_penalty": 0.1,
  "stop": [
    "<|endoftext|>",
    "<|im_end|>",
    "<|end|>",
    "</s>"
  ]
}

3. Check Image Resolution / DPI
If the input PDF contains low-resolution, blurry, or tiny text, the vision encoder enters high-uncertainty loops. Ensure input pages are rasterized clearly at 300 DPI before sending them to the model.

[masifamu]

Thanks for the suggestions.

I have tried those suggestions as it is, but I am still seeing some garbled text in the parsed output, this is happening even after increasing the DPI of the pdf, but yes you are right, increasing DPI helps. The new set of suggestions stops the repetition, thanks.

Here is how I run the llama server: Please check, I may be doing something wrong.
./build/bin/llama-server -m ~/models/unlimited-ocr/Unlimited-OCR-Q8_0.gguf --mmproj ~/models/unlimited-ocr/mmproj-Unlimited-OCR-F16.gguf --chat-template deepseek-ocr --no-jinja --host 0.0.0.0 --port 8080 -t 12 -ngl 0 -c 8192 --temp 0 --repeat-penalty 1.2 --repeat-last-n 512

NOTE: I have configured the JSON in the UI settings under developer option.