Add LLM Auto Context Compaction

[howard0su]

Summary

Server-side context compaction that automatically compresses conversation history when prompts overflow the context window. Uses the prefill-drafter (Qwen3-0.6B) as a dedicated summarization backend — fast and doesn't block the main model.

3-Layer Pipeline

Layer	Strategy	Cost
1	Edit compaction — strip blocks, truncate/dedupe tool outputs	CPU, <1ms
2	Self-summarization — use prefill drafter model condenses older history into summary	GPU (0.6B), ~1-3s
3	Hard truncation — progressive tail-keeping as last resort	CPU

Key Design Decisions

• Enabled by default — triggered by client context_management in request body
• Prefill-drafter reused — when --prefill-drafter is present, Qwen3-0.6B handles summarization (avoids tying up the 27B+ target)
• Summary length scales with max_ctx — max_ctx / 10, clamped [256, 2048]. No separate knob needed.
• Stateless — no per-session state; compaction operates on the provided messages array only

Files Changed (9 files, +2162 / -39)

New:

• server/src/server/compaction.h/.cpp — compaction logic
• harness/test_compaction.py — integration tests
• docs/llm-context-compaction.md — research report (38KB)
• docs/small-model-compression.md — small model research (16KB)

Modified:

• server/src/server/http_server.h/.cpp — threshold check in route_request(), pipeline in worker_loop(), usage reporting
• server/src/server/server_main.cpp — CLI flags, compaction backend creation
• server/CMakeLists.txt — added source file

API

Request body (Responses API):

 { "context_management": [{"type": "compaction", "compact_threshold": 4096}] }

Response includes:

 { "usage": { "compacted_tokens_saved": 1044 } }

Testing

 python harness/test_compaction.py --server-bin build/dflash_server \
   --model <target.gguf> --prefill-drafter server/models/Qwen3-0.6B-BF16.gguf