该Docker镜像基于最新llama.cpp主分支构建,支持多Token预测(MTP)技术,提供CUDA、Vulkan等多种GPU后端,可快速部署高性能大语言模型推理服务。
llama.cpp with Multi-Token Prediction (MTP) Support
Docker images for https://github.com/ggml-org/llama.cpp built from the latest main branch, with full Multi-Token Prediction (MTP) support including image input and recent bug fixes not yet available in official releases.
What is MTP?
Multi-Token Prediction (MTP) is a speculative decoding technique that allows the model to predict multiple tokens per forward pass, resulting in significantly faster inference with no quality loss. To use it, you need both an MTP-compatible server build and a model that includes MTP draft layers.
Available Tags
| Tag | Backend |
|---|---|
havenoammo/llama:cuda13-server | NVIDIA CUDA 13 |
havenoammo/llama:cuda12-server | NVIDIA CUDA 12 |
havenoammo/llama:vulkan-server | Vulkan (cross-platform GPU) |
havenoammo/llama:intel-server | Intel GPU (SYCL) |
havenoammo/llama:rocm-server | AMD ROCm |
Note: These images are community-built and tested primarily on CUDA 13. Other backends should work but are less tested. Feedback welcome.
Compatible Models
Unsloth MTP Models (quantized MTP layers)
- https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
- https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF
Q8 MTP Models (full Q8 MTP layers for improved prediction accuracy)
- https://huggingface.co/havenoammo/Qwen3.6-27B-MTP-UD-GGUF
- https://huggingface.co/havenoammo/Qwen3.6-35B-A3B-MTP-GGUF
The Q8 variants keep MTP layers at full Q8 quantization, which can improve speculative decoding accuracy at the cost of slightly more VRAM usage.
Quick Start
bashdocker run --gpus all --rm \ -p 8080:8080 \ -v ./models:/models \ havenoammo/llama:cuda13-server \ -m /models/Qwen3.6-27B-MTP-UD-Q8_K_XL.gguf \ --port 8080 \ --host 0.0.0.0 \ -n -1 \ --parallel 1 \ --ctx-size 262144 \ --fit-target 844 \ --mmap \ -ngl -1 \ --flash-attn on \ --metrics \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --jinja \ --chat-template-kwargs '{"preserve_thinking":true}' \ --ubatch-size 512 \ --batch-size 2048 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --spec-type mtp \ --spec-draft-n-max 3
Adjust as you see fit. What matters most for MTP is --spec-type mtp and --spec-draft-n-max 3.
Switching from Official llama.cpp Docker Images
If you are already using the official ghcr.io/ggml-org/llama.cpp images, switching is straightforward. Simply replace the image name with the appropriate tag above. The server flags and model paths remain the same.
Feedback
If you run into issues with a specific backend, feel free to open a discussion. Reports on which hardware and configurations work are especially helpful.
镜像拉取方式
您可以使用以下命令拉取该镜像。请将 <标签> 替换为具体的标签版本。如需查看所有可用标签版本,请访问 标签列表页面。
DockerHub 原生拉取命令
用户好评
来自真实用户的反馈,见证轩辕镜像的优质服务