OpenAI/Anthropic-compatible API server

[maxiboch]

Overview

Add an HTTP API server to maxi-ml that exposes OpenAI and Anthropic-compatible endpoints. This makes maxi-ml a drop-in replacement for any tool that supports those APIs (LM Studio, Continue.dev, Cursor, Open WebUI, Chatbox, etc).

Endpoints

Required (OpenAI format)

POST /v1/chat/completions

• Input: {"model": "...", "messages": [...], "temperature": 0.7, "max_tokens": 128, "stream": true/false}
• Output (non-streaming): {"id": "chatcmpl-xxx", "object": "chat.completion", "choices": [{"message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": N, "completion_tokens": N, "total_tokens": N}}
• Output (streaming): SSE stream of data: {"choices": [{"delta": {"content": "tok"}}]} chunks, terminated by data: [DONE]
• This is the most important endpoint — 90% of clients only use this

POST /v1/completions

• Legacy text completion format
• Input: {"model": "...", "prompt": "...", "max_tokens": 128}
• Same response structure but with text instead of message

POST /v1/embeddings

• Input: {"model": "...", "input": ["text1", "text2"]}
• Output: {"data": [{"embedding": [0.1, 0.2, ...], "index": 0}], "usage": {...}}
• Maps to our existing MiaEmbedModel / GpuBgeM3Model

POST /v1/audio/transcriptions

• Input: multipart form with file (audio) and model
• Output: {"text": "transcribed text"}
• Maps to our ASR pipeline

GET /v1/models

• Lists available models (auto-detected from model directories)
• Output: {"data": [{"id": "Qwen3-0.6B", "object": "model", "owned_by": "local"}]}

Nice-to-have (Anthropic format)

POST /v1/messages

• Anthropic Messages API format
• Input: {"model": "...", "messages": [...], "max_tokens": 128}
• Similar to OpenAI but different response schema

Health/Status

GET /health — returns 200 if server is running
GET /v1/cache/stats — KV cache utilization, loaded models, memory usage

Architecture

Crate: maxi-ml-server (new crate in crates/)

crates/maxi-ml-server/
├── Cargo.toml
├── src/
│   ├── lib.rs
│   ├── server.rs          # axum router setup, bind, TLS
│   ├── routes/
│   │   ├── chat.rs         # /v1/chat/completions
│   │   ├── completions.rs  # /v1/completions
│   │   ├── embeddings.rs   # /v1/embeddings
│   │   ├── audio.rs        # /v1/audio/transcriptions
│   │   ├── models.rs       # /v1/models
│   │   └── health.rs       # /health, /v1/cache/stats
│   ├── types/
│   │   ├── openai.rs       # OpenAI request/response types (serde)
│   │   ├── anthropic.rs    # Anthropic request/response types
│   │   └── streaming.rs    # SSE streaming helpers
│   └── model_manager.rs    # Loads/manages models, maps model IDs to backends

Dependencies

• axum — HTTP framework (async, tower-based)
• tokio — async runtime
• serde / serde_json — JSON serialization
• uuid — request ID generation
• tower-http — CORS middleware

Integration Points

The server calls into existing maxi-ml infrastructure:

// Chat completions → existing generate pipeline
use maxi_ml_core::generate::{generate_dyn, CausalLM, SamplingConfig};

// Embeddings → existing embed pipeline  
use mia_embed::SmartMiaEmbedModel;

// ASR → existing whisper pipeline
use asr_models::WhisperModel;

// Model loading → existing weight loading
use maxi_ml_core::qwen::Qwen3Model;
use maxi_ml_backend_metal::pipeline::GpuQwen3Model;

Streaming Implementation

SSE streaming for chat completions:

async fn chat_completions_stream(
    model: &dyn CausalLM,
    request: ChatCompletionRequest,
) -> impl IntoResponse {
    let (tx, rx) = tokio::sync::mpsc::channel(32);
    
    tokio::spawn(async move {
        // Run generation, send each token as SSE event
        for token in generate_stream(model, &request) {
            let chunk = ChatCompletionChunk { delta: token, ... };
            tx.send(format!("data: {}\n\n", serde_json::to_string(&chunk)?)).await;
        }
        tx.send("data: [DONE]\n\n").await;
    });
    
    Sse::new(ReceiverStream::new(rx))
}

Model Manager

Auto-discovers models from configured directories:

struct ModelManager {
    model_dirs: Vec<PathBuf>,
    loaded: HashMap<String, LoadedModel>,
    default_backend: ComputeBackend,
    strategy: ExecutionStrategy,
}

enum LoadedModel {
    Cpu(Qwen3Model),
    Gpu(GpuQwen3Model),
    Embed(SmartMiaEmbedModel),
}

CLI Integration

Add to maxi-ml-cli:

maxi-ml serve [OPTIONS]
  --port 8080              HTTP port (default: 11434 for Ollama compat)
  --host 0.0.0.0           Bind address
  --model-dir <PATH>       Model directory (can specify multiple)
  --backend auto            Compute backend
  --strategy balanced       Execution strategy
  --api-key <KEY>          Optional API key for auth
  --cors                   Enable CORS for browser clients

Testing

All of this can be unit tested without GPU hardware:

• Mock the CausalLM trait with a stub that returns fixed tokens
• Test request parsing, response formatting, SSE streaming
• Test model listing, error handling, parameter validation
• Integration tests with reqwest against the running server

Acceptance Criteria

1. curl http://localhost:8080/v1/chat/completions -d '{"model":"Qwen3-0.6B","messages":[{"role":"user","content":"hi"}]}' returns valid OpenAI response
2. Streaming works with stream: true
3. /v1/models lists auto-detected models
4. /v1/embeddings works with embed models
5. Any OpenAI-compatible client (Continue.dev, Open WebUI) can connect and chat