DGX Spark GGUF llama.cpp Template

Docker Compose template for serving sharded GGUF models on NVIDIA DGX Spark with llama.cpp and an OpenAI-compatible /v1 endpoint.

The included example profile serves stepfun-ai/Step-3.7-Flash-GGUF using the Q4_K_S GGUF shards and the StepFun llama.cpp step3.7 branch. Use it as a known-good starting point, then copy the service block in docker-compose.yml for your own model profile.

What Is Included

• CUDA 13.1 llama.cpp Docker build tuned for GB10 / Blackwell
• Docker Compose service with host networking, GPU access, and a healthcheck
• Automatic Hugging Face download for sharded GGUFs and optional multimodal projector
• Ignore rules for tokens, model weights, logs, and generated benchmark output

Prerequisites

On the DGX Spark:

• Docker with NVIDIA Container Toolkit
• Docker Compose v2
• Enough free disk space for the selected GGUF shards
• Hugging Face access to the model repository, if the model is gated

Quick Start On DGX Spark

docker compose up -d --build

For gated Hugging Face models, copy .env.example to .env and set HF_TOKEN before starting Compose.

The first startup builds llama.cpp, downloads missing model files into models/, and starts llama-server. The service listens on http://<spark-host>:8000/v1 by default.

Check service status:

docker compose ps
docker compose logs -f step37-flash-q4-k-s

From the Spark itself, the health endpoint is:

curl http://127.0.0.1:8000/v1/models

Smoke-test the OpenAI-compatible chat endpoint:

curl -fsS http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "step37-flash-q4-k-s",
    "messages": [
      {"role": "user", "content": "Write one short sentence about DGX Spark."}
    ],
    "temperature": 0.2,
    "max_tokens": 64,
    "stream": false
  }'

Stop the service:

docker compose down --remove-orphans

Example Model Profile

The default service is step37-flash-q4-k-s.

• Hugging Face repo: stepfun-ai/Step-3.7-Flash-GGUF
• Quantization: Q4_K_S
• Primary GGUF: Q4_K_S/Step-3.7-flash-Q4_K_S-00001-of-00003.gguf
• Required shards:
  • Q4_K_S/Step-3.7-flash-Q4_K_S-00001-of-00003.gguf
  • Q4_K_S/Step-3.7-flash-Q4_K_S-00002-of-00003.gguf
  • Q4_K_S/Step-3.7-flash-Q4_K_S-00003-of-00003.gguf
• Optional projector: mmproj-step3.7-flash-f16.gguf
• llama.cpp source: https://github.com/stepfun-ai/llama.cpp.git
• llama.cpp ref: step3.7

To enable multimodal serving for this profile:

ENABLE_MMPROJ=1 docker compose up -d --build

Set ENABLE_MMPROJ=1 in .env if you want multimodal serving to persist across future Compose commands.

Adapting The Template

For a new model, copy the step37-flash-q4-k-s service in docker-compose.yml and change these environment values:

• PROFILE_NAME: local folder under models/
• GGUF_REPO: Hugging Face repository
• GGUF_FILE: primary GGUF file passed to llama-server
• GGUF_INCLUDE: optional Hugging Face include glob, useful for shards
• GGUF_REQUIRED_FILES: space-separated files that must exist before serving
• MMPROJ_FILE: optional multimodal projector file
• SERVED_MODEL_NAME: model name exposed by the OpenAI-compatible API

If your model requires upstream llama.cpp instead of a vendor branch, set these in .env:

LLAMACPP_REPO=https://github.com/ggml-org/llama.cpp.git
LLAMACPP_REF=master

For model-specific runtime flags, use:

LLAMACPP_EXTRA_ARGS="--your-flag value"

Benchmarking From A Local Workstation

This template intentionally does not include benchmark orchestration scripts. Point your preferred OpenAI-compatible benchmark tool at:

http://<spark-host>:8000/v1

Generated benchmark output should stay out of git.

Useful Environment Variables

Variable	Default	Purpose
HF_TOKEN	blank	Hugging Face token for gated models
LLAMACPP_PORT	8000	Host port for the OpenAI-compatible server
MAX_MODEL_LEN	32768	llama.cpp context size
LLAMA_THREADS	20	CPU threads for llama.cpp
BATCH_SIZE	2048	llama.cpp batch size
UBATCH_SIZE	1024	llama.cpp micro-batch size
CACHE_TYPE_K	q8_0	KV cache K type
CACHE_TYPE_V	q8_0	KV cache V type
GPU_LAYERS	99	Number of layers offloaded to GPU
FLASH_ATTN	on	llama.cpp flash attention setting
ENABLE_MMPROJ	0	Enable optional multimodal projector
AUTO_DOWNLOAD	1	Download missing model files at container startup
CMAKE_CUDA_ARCHITECTURES	121	CUDA architecture for DGX Spark GB10

Publishing Checklist

• Confirm LICENSE matches your intended open-source license.
• Run docker compose config.
• Confirm .env, models/, logs/, and generated results/ files are not tracked.
• Replace or expand the example profile if this repository should target a different default model.