From c13c32755a950efc5a7ad91836c8152229761fde Mon Sep 17 00:00:00 2001 From: Simo Lin <25425177+slin1237@users.noreply.github.com> Date: Sat, 1 Aug 2026 16:16:14 -0700 Subject: [PATCH] docs(readme): align README with the shipped feature set Every claim now traces to code or the supported product surface: - 8 -> 10 routing policies (adds passthrough, least_load) - 40+ -> 90+ Prometheus metrics; OTel W3C trace propagation into engines over HTTP and gRPC - Drop unverifiable claims: sub-millisecond routing decisions, zero-copy tokenization (now: cached tokenization with zero-copy cache hits) - Supported Backends reshaped to one row per category so the two lists no longer read as row pairings; engines gain MLX, Ollama reclassified as an OpenAI-compatible server; providers list the dedicated adapters (OpenAI, Anthropic, Gemini, xAI) followed by OCI Generative AI, AWS Bedrock, and Azure OpenAI (reached via their OpenAI-compatible modes) and the generic passthrough - API coverage now lists the shipped surface: Rerank, Classify, Conversations, Gemini Interactions, Realtime (WebSocket + WebRTC), audio transcription, tokenize/detokenize; MCP execution scoped to Responses + Messages where it actually runs - New Why/Features rows: priority admission scheduling, PD/EPD disaggregation with DP-aware routing, K8s discovery, 21 tool + 16 reasoning parsers; OIDC correctly scoped to the control plane - Quick Start: add Helm (oci://ghcr.io/smg-project/charts/smg) and the protoc prerequisite for cargo install - All docs links moved from smg-docs GitHub blobs to the live site (lightseek.org/smg); docs badge retargeted from the dead MkDocs GitHub Pages site; Docker badge now points at Docker Hub where stable tags are actually published Signed-off-by: Simo Lin <25425177+slin1237@users.noreply.github.com> --- README.md | 69 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 36 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index a1f2c2e01..125ebd9d4 100644 --- a/README.md +++ b/README.md @@ -10,17 +10,17 @@

Release - Docker + Docker PyPI License - Docs + Docs Discord Slack Ask DeepWiki PyTorch Blog

-Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows. +Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.

SMG architecture: clients flow through the gateway layer and router layer to gRPC workers, HTTP workers, and external APIs @@ -30,13 +30,13 @@ Engine-agnostic, high-performance model-routing gateway for large-scale LLM depl | | | |:--------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **🚀 Maximize GPU Utilization** | Cache-aware routing understands your inference engine's KV cache state—whether vLLM, TensorRT-LLM, TokenSpeed, or SGLang—to reuse prefixes and reduce redundant computation. | -| **🔌 One API, Any Backend** | Route to self-hosted models (vLLM, TensorRT-LLM, TokenSpeed, SGLang) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint. | -| **⚡ Built for Speed** | Native Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running. | -| **🔒 Enterprise Control** | Multi-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure. | -| **📊 Full Observability** | 40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer. | +| **🚀 Maximize GPU Utilization** | Cache-aware routing tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX — with load modeling that accounts for queued token work and KV pressure. | +| **🔌 One API, Any Backend** | Route to self-hosted engines over HTTP or gRPC, or to OpenAI, Anthropic, Gemini, and xAI — plus any OpenAI-compatible endpoint — through a single unified gateway. | +| **⚡ Built for Speed** | Native Rust with streaming gRPC pipelines, cached tokenization with zero-copy cache hits, prefill/decode disaggregation (including a separate encode stage for vision), and DP-aware routing for data-parallel engines. | +| **🔒 Enterprise Control** | Priority admission scheduling with preemption and per-tenant controls, API-key auth with OIDC on the control plane, WebAssembly plugins for custom logic, and chat history that never leaves your infrastructure. | +| **📊 Full Observability** | 90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into the engines over both HTTP and gRPC, and structured JSON logs with request correlation. | -**API Coverage:** OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution. +**API Coverage:** OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs. ## Quick Start @@ -46,10 +46,13 @@ Engine-agnostic, high-performance model-routing gateway for large-scale LLM depl # Docker docker pull lightseekorg/smg:latest +# Kubernetes (Helm) +helm install smg oci://ghcr.io/smg-project/charts/smg + # Python pip install smg -# Rust +# Rust (needs protoc) cargo install smg ``` @@ -79,40 +82,40 @@ That's it. SMG is now load-balancing requests across your workers. ## Supported Backends -| Self-Hosted | Cloud Providers | -|-------------|--------------------------------| -| vLLM | OpenAI | -| TensorRT-LLM | Anthropic | -| TokenSpeed | Google Gemini | -| SGLang | Oracle Generative AI Service | -| Ollama | Azure OpenAI | -| Any OpenAI-compatible server | Any OpenAI-compatible provider | +| | | +|:--|:--| +| **Self-Hosted Engines** | vLLM · SGLang · TokenSpeed · TensorRT-LLM · MLX (Apple Silicon) · any OpenAI-compatible server (e.g. Ollama) | +| **Cloud Providers** | OpenAI · Anthropic · Google Gemini · xAI · OCI Generative AI · AWS Bedrock · Azure OpenAI · any OpenAI-compatible provider (Groq, Together, …) | ## Features | Feature | Description | |---------|-------------| -| **[8 Routing Policies](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/routing/load-balancing.md)** | cache_aware, round_robin, power_of_two, consistent_hashing, prefix_hash, manual, random, bucket | -| **[gRPC Pipeline](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/architecture/grpc-pipeline.md)** | Native gRPC with streaming, reasoning extraction, and tool call parsing | -| **[MCP Integration](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/extensibility/mcp.md)** | Connect external tool servers via Model Context Protocol | -| **[High Availability](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/architecture/high-availability.md)** | Mesh networking with SWIM protocol for multi-node deployments | -| **[Chat History](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/data/chat-history.md)** | Pluggable storage: PostgreSQL, Oracle, Redis, or in-memory | -| **[WASM Plugins](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/extensibility/wasm-plugins.md)** | Extend with custom WebAssembly logic | -| **[Resilience](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/reliability/index.md)** | Circuit breakers, retries with backoff, rate limiting | +| **[10 Routing Policies](https://lightseek.org/smg/concepts/routing/load-balancing)** | cache_aware, least_load, power_of_two, consistent_hashing, prefix_hash, bucket, round_robin, random, manual, passthrough | +| **[gRPC Pipeline](https://lightseek.org/smg/concepts/architecture/grpc-pipeline)** | Native streaming gRPC to the engines with prefill/decode and encode disaggregation and DP-aware routing | +| **[Kubernetes Discovery](https://lightseek.org/smg/getting-started/service-discovery)** | Native pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery | +| **[Model Parsers](https://lightseek.org/smg/getting-started/tokenization-and-parsing)** | 21 tool-call parsers and 16 reasoning parsers with automatic model detection — DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and more | +| **[MCP Integration](https://lightseek.org/smg/concepts/extensibility/mcp)** | Tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging | +| **[High Availability](https://lightseek.org/smg/concepts/architecture/high-availability)** | Mesh networking with SWIM gossip and CRDT-replicated state for multi-node deployments | +| **[Chat History](https://lightseek.org/smg/concepts/data/chat-history)** | Pluggable storage with schema migrations: PostgreSQL, Oracle, Redis, or in-memory | +| **[WASM Plugins](https://lightseek.org/smg/concepts/extensibility/wasm-plugins)** | Extend request and response handling with custom WebAssembly middleware | +| **[Resilience](https://lightseek.org/smg/concepts/reliability/index)** | Circuit breakers, retries with backoff and jitter, rate limiting, and priority admission scheduling | ## Documentation +Full documentation lives at **[lightseek.org/smg](https://lightseek.org/smg/)**. + | | | |:--|:--| -| [Getting Started](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/getting-started/index.md) | Installation and first steps | -| [Architecture](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/concepts/architecture/overview.md) | How SMG works | -| [Configuration](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/reference/configuration.md) | CLI reference and options | -| [API Reference](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/reference/api/openai.md) | OpenAI-compatible endpoints | -| [Kubernetes Setup](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/getting-started/service-discovery.md) | In-cluster discovery and production setup | +| [Getting Started](https://lightseek.org/smg/getting-started) | Installation and first steps | +| [Architecture](https://lightseek.org/smg/concepts/architecture/overview) | How SMG works | +| [Configuration](https://lightseek.org/smg/reference/configuration) | CLI reference and options | +| [API Reference](https://lightseek.org/smg/reference/api/openai) | OpenAI-compatible endpoints | +| [Kubernetes Setup](https://lightseek.org/smg/getting-started/service-discovery) | In-cluster discovery and production setup | ## Contributing -We welcome contributions! See [Contributing Guide](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/contributing/index.md) for details. +We welcome contributions! See the [Contributing Guide](https://lightseek.org/smg/contributing) for details. -- [Development Setup](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/contributing/development.md) -- [Code Style](https://github.com/smg-project/smg-docs/blob/main/src/lib/content/contributing/code-style.md) +- [Development Setup](https://lightseek.org/smg/contributing/development) +- [Code Style](https://lightseek.org/smg/contributing/code-style)