A lightweight vLLM-style inference engine implemented from scratch based on the Nano-vLLM framework — a beginner-friendly teaching-oriented inference engine framework. All newly added features are documented under docs/features in plain language explaining the implementation details, and the key points that may come up in future interviews are recorded there as well.
- INT8 KV Cache Quantization —— per-token symmetric quantization that extends cacheable context length and reduces memory footprint; see pipeline and implementation details.
- AWQ Weight Quantization (W4A16) —— load and run existing AutoAWQ 4-bit quantized models (no calibration set required, direct inference); see pipeline and implementation details.
- 🚀 Fast offline inference —— inference speed comparable to vLLM
- 📖 Readable codebase —— clean implementation in ~1,200 lines of Python
- ⚡ Optimization Suite —— prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.
This project maintains two long-lived, independently evolving branch lines. Choose according to your use case:
| Branch | Positioning | Notes |
|---|---|---|
v0.2.1 |
Industrial branch | Targets production / industry-common paths, includes stable features such as INT8 KV Cache quantization and AWQ weight quantization; published as official Releases. |
sci_v0.2.1 |
Research branch | Targets research exploration, introducing various research-oriented optimizations (which may differ from industry-common practice); evolves in parallel with v0.2.1 and is generally not published as an official Release. |
Both branches are forked from
mainand maintained with independent commits. The released version ofv0.2.1is an immutable snapshot pinned to its corresponding tag (e.g.v0.2.1);sci_v0.2.1serves only as a development line / version anchor.
The project's version numbers follow Nano-vLLM (evolving from the Nano-vLLM baseline). Features supported by each version:
- INT8 KV Cache Quantization —— per-token symmetric quantization that extends cacheable context length and reduces memory footprint; see quantization/dequantization and full-pipeline details.
- AWQ Weight Quantization (W4A16) —— load and run existing AutoAWQ 4-bit quantized models (no calibration set required, direct inference); see pipeline and implementation details.
- 🚀 Fast offline inference —— inference speed comparable to vLLM
- 📖 Readable codebase —— clean implementation in ~1,200 lines of Python
- ⚡ Optimization Suite —— prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.
pip install git+https://github.com/1571859588/EvoNano-VLLM.gitTo download the model weights manually, use the following command:
huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
--local-dir ~/huggingface/Qwen3-0.6B/ \
--local-dir-use-symlinks FalseSee example.py for usage. The API mirrors vLLM's interface with minor differences in the LLM.generate method:
from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, EvoNano-VLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]| Architecture | Status | Description | Verified Model |
|---|---|---|---|
| Qwen3 | ✅ Supported | The currently implemented and verified model architecture | Qwen3-0.6B (all benchmarks and INT8 KV Cache quantization verifications in this repo are run on this model) |
| Qwen3 + AWQ (W4A16) | ✅ Supported | int4 weight quantization (AutoAWQ format, gemm, group_size=128), Reference dequantization path | Qwen3-0.6B-AWQ, Qwen3-8B-AWQ |
Note: Currently only the Qwen3 architecture is adapted. Other architectures (e.g. Llama series) require a corresponding implementation under
nanovllm/models/before use. The AWQ backend currently supports only 4-bit, zero_point=True, gemm/gemv AutoAWQ exports.
This repo ships two benchmark scripts with different purposes; do not confuse them:
| Script | Purpose | Notes |
|---|---|---|
bench.py |
General performance benchmark | Uses random tokens to stress test engine throughput/latency, for comparing against vLLM etc. |
bench_sharegpt.py |
Real-data quantization/config comparison | Uses real ShareGPT conversations as prompts to compare throughput/memory/capacity across KV quantization, weight quantization, and speculative decoding. |
tests/kv_quant/verify_kv_quant.py |
Generation quality verification | Generates once with baseline and once with int8 on the same prompts, then compares text similarity to verify quantization is lossless. |
bench.py uses randomly sampled token sequences (not real text) to stress engine throughput, for horizontal comparison against vLLM.
Test Configuration:
- Model: Qwen3-0.6B
- Total Requests: 256 sequences
- Input/Output Length: randomly sampled between 100–1024 tokens
| Inference Engine | Output Tokens | Time (s) | Throughput (tokens/s) |
|---|---|---|---|
| vLLM | 133,966 | 98.37 | 1361.84 |
| EvoNano-VLLM | 133,966 | 93.41 | 1434.13 |
| Inference Engine | Output Tokens | Time (s) | Throughput (tokens/s) |
|---|---|---|---|
| EvoNano-VLLM | 133,966 | 21.30 | 6289.08 |
Full run methods, evaluation metrics, quality verification, and result file structure are in docs/features/int8_kv_cache/int8_kv_cache.md.
| Metric | baseline (fp16, eager) | int8 KV (eager) | Change |
|---|---|---|---|
| KV Capacity (tokens) | 311,808 | 604,928 | +94% |
| KV cache memory | 33.305 GB | 32.307 GB | slightly lower (int8 + scale) |
| TTFT (mean) | 0.5342 s | 0.4985 s | comparable |
| TPOT (mean) | 21.8 ms | 66.5 ms | ~30% slower |
| Throughput | 1715.97 tok/s | 633.69 tok/s | -63% |
| Generation char overlap | — | 67.4% | quantization essentially lossless |
Data source:
benchmark_results/kv_int8_cache/sharegpt_baseline_eager.json,sharegpt_int8kv.json,kv_quant_verify.json. int8 cannot be captured by CUDA Graph and thus auto-enters eager mode, so the baseline is also run with--enforce_eagerto keep a single variable.
3. AWQ 4-bit Weight Quantization Benchmark (tests/awq/bench_one.py + tests/awq/bench_awq_vs_bf16.py)
Full run methods, unit checks, three-way comparison (A=AWQ / B=BF16 / C=force non-quantized load, expected to fail), performance bottleneck and implementation notes are in docs/features/int4_awq_w4a16/awq_benchmark.md; the loading and dequantization teaching (with real tensor shapes) is in docs/features/int4_awq_w4a16/awq_loading.md; the readable three-way comparison analysis report (with per-sample output comparison) is in benchmark_results/quantization_int_4/report.md.
Quantization mode distinction: the AWQ supported here belongs to the "load an already-quantized model" class — weights were quantized offline by AutoAWQ (
qweight/qzeros/scales), and this engine loads and runs them directly without any calibration dataset; we will later add a calibration-set-based "online quantization (PTQ)" class.
| Metric | AWQ (W4A16) | BF16 original | Note |
|---|---|---|---|
| Weight memory | 1012 MB | 1613 MB | BF16 ≈ 1.59x of AWQ |
| Prefill throughput | 208 tok/s | 227 tok/s | comparable |
| Decode throughput | 15.8 tok/s | 46.0 tok/s | AWQ ~3x slower (Reference dequant path overhead) |
| TPOT | 63 ms | 22 ms | AWQ ~3x slower |
| Metric | Value |
|---|---|
| Weight memory | 8296 MB |
| Prefill throughput | 95.5 tok/s |
| Decode throughput | 1.9 tok/s |
| TPOT | 528 ms |
Generation quality and performance-bottleneck notes are in docs/features/int4_awq_w4a16/awq_benchmark.md.
The regression_test.sh at the repo root is a one-shot regression entry point that verifies all capabilities still work after a change. It runs the following cases serially (inspect each case's output for green [PASS] / red [FAIL]):
| Case | What it verifies | Key input / output |
|---|---|---|
| A. KV Cache Quant | ShareGPT throughput baseline (fp16) + int8 KV + KV numeric check | huggingface/Qwen3-0.6B → benchmark_results/kv_int8_cache/*.json |
| B. AWQ single-model | AWQ (W4A16) loads and generates sane text; records memory/throughput | huggingface/Qwen3-0.6B-AWQ → quantization_int_4/qwen3_0.6b_awq.json |
| C. BF16 vs AWQ | A=AWQ / B=BF16 original / C=strip-quant-config load (expected to fail) | tests/awq/bench_awq_vs_bf16.py → results.json + report.md |
| D. dtype generality | BF16 original runs standalone (no longer silently treated as fp16) | huggingface/Qwen3-0.6B → quantization_int_4/qwen3_0.6b_bf16.json |
| E. TP tensor-parallel | dtype change doesn't break sharding (skips if <2 idle GPUs) | huggingface/Qwen3-0.6B TP=2 → benchmark_results/tp/... |
| F. BF16 + int8 KV | attention dequantized dtype follows correctly | huggingface/Qwen3-0.6B --kv_quant int8 |
| G. AWQ numeric checks | merged-layer packing + single-layer dequant matches HF reference | tests/awq/test_awq_load.py / test_awq_layer.py |
| H. checkpoint integrity guard | missing shard should error, not silently emit garbage (8B-AWQ shards known-missing, guard only) | huggingface/Qwen/Qwen3-8B-AWQ |
Run it:
bash regression_test.sh
# Uses CUDA_VISIBLE_DEVICES=5 (A100 40G) by default. Change the GPU var at the top of the script to switch.
# All result JSONs land under benchmark_results/ for review.Case E (TP=2) needs 2 idle GPUs; the script probes idle cards (memory used < 4GB) and auto-skips if not enough, to avoid single-GPU nccl deadlock / a permanently held port. All other cases run on a single GPU.
Before any change is merged, it must pass every feature covered by regression_test.sh (all of A–G; H is exempt due to the known upstream shard issue). Specifically:
- "Partial pass" is not mergeable — if the change touches common engine paths (loader, dtype passthrough, linear, KV quant, TP sharding, etc.), A–G must all show
[PASS]. - Case C ("strip quant config and load AWQ", expected to fail) is an expected-failure case — its failure is correct behavior and must not be treated as a regression; success of every other case is mandatory.
- Case H (8B-AWQ integrity guard) is allowed to degrade only because the upstream checkpoint shards are missing; once
model-00001-of-00002.safetensorsis restored, this case must also pass. - Recommended: wire this script into CI as a required PR check; run
bash regression_test.shlocally before submitting.
The following are subsequent optimization and feature directions, ordered by priority/dependency. The AWQ-related item is referenced in the benchmark and implementation notes above.
- AWQ parallel / fused dequantization & loading
- Replace the serial
reference_awq_lineardequantization path inawq.pywith a Triton fused kernel (unpack int4 weights, subtract zero_point, multiply scale and accumulate with the input in one pass) to avoid materializing the full FP32 weight intermediate and to reduce kernel-launch count. - Explore batched / kernel-fused filling in the loading stage (
_weight_loader/_zero_loader), replacing the per-parameter serialnarrow().copy_(). - Goal: eliminate the serial dequantization bottleneck in the 8B-AWQ decode phase (currently 1.9 tok/s) and match the fused int4 GEMM performance of vLLM / AutoAWQ.
- Validation: use the HF reference numerical parity in
tests/awq/test_awq_layer.pyas the golden reference.
- Replace the serial
- On-the-fly quantization (PTQ): support calibration-set-based quantization (e.g. automatic optimal scaling-factor search), distinct from the current "load an already-quantized model" path.
- More model architectures: add Llama-family and others under
nanovllm/models/, extending the AWQ backend's coverage. - AWQ gemv decode-specific optimization: kernel-level acceleration for the single-token (batch=1) GEMV layout used in the decode phase.
- CUDA Graph compatibility for KV Cache quantization: explore capturing the dynamic dequantization of the decode phase into Graph capture to reduce TPOT degradation under eager mode.