llama

Llama Docker Image

该Docker镜像基于最新llama.cpp主分支构建,支持多Token预测(MTP)技术,提供CUDA、Vulkan等多种GPU后端,可快速部署高性能大语言模型推理服务。

下载次数: 0状态:社区镜像维护者:havenoammo仓库类型:镜像最近更新:
下载命令
点击查看

llama.cpp with Multi-Token Prediction (MTP) Support

Docker images for https://github.com/ggml-org/llama.cpp built from the latest main branch, with full Multi-Token Prediction (MTP) support including image input and recent bug fixes not yet available in official releases.


What is MTP?

Multi-Token Prediction (MTP) is a speculative decoding technique that allows the model to predict multiple tokens per forward pass, resulting in significantly faster inference with no quality loss. To use it, you need both an MTP-compatible server build and a model that includes MTP draft layers.


Available Tags

TagBackend
havenoammo/llama:cuda13-serverNVIDIA CUDA 13
havenoammo/llama:cuda12-serverNVIDIA CUDA 12
havenoammo/llama:vulkan-serverVulkan (cross-platform GPU)
havenoammo/llama:intel-serverIntel GPU (SYCL)
havenoammo/llama:rocm-serverAMD ROCm

Note: These images are community-built and tested primarily on CUDA 13. Other backends should work but are less tested. Feedback welcome.


Compatible Models

Unsloth MTP Models (quantized MTP layers)

  • https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
  • https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Q8 MTP Models (full Q8 MTP layers for improved prediction accuracy)

  • https://huggingface.co/havenoammo/Qwen3.6-27B-MTP-UD-GGUF
  • https://huggingface.co/havenoammo/Qwen3.6-35B-A3B-MTP-GGUF

The Q8 variants keep MTP layers at full Q8 quantization, which can improve speculative decoding accuracy at the cost of slightly more VRAM usage.


Quick Start

bash
docker run --gpus all --rm \
    -p 8080:8080 \
    -v ./models:/models \
    havenoammo/llama:cuda13-server \
    -m /models/Qwen3.6-27B-MTP-UD-Q8_K_XL.gguf \
    --port 8080 \
    --host 0.0.0.0 \
    -n -1 \
    --parallel 1 \
    --ctx-size 262144 \
    --fit-target 844 \
    --mmap \
    -ngl -1 \
    --flash-attn on \
    --metrics \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --jinja \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --ubatch-size 512 \
    --batch-size 2048 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --spec-type mtp \
    --spec-draft-n-max 3

Adjust as you see fit. What matters most for MTP is --spec-type mtp and --spec-draft-n-max 3.


Switching from Official llama.cpp Docker Images

If you are already using the official ghcr.io/ggml-org/llama.cpp images, switching is straightforward. Simply replace the image name with the appropriate tag above. The server flags and model paths remain the same.


Feedback

If you run into issues with a specific backend, feel free to open a discussion. Reports on which hardware and configurations work are especially helpful.

镜像拉取方式

您可以使用以下命令拉取该镜像。请将 <标签> 替换为具体的标签版本。如需查看所有可用标签版本,请访问 标签列表页面。

轩辕镜像加速拉取命令点我查看更多 llama 镜像标签

docker pull docker.xuanyuan.run/havenoammo/llama:<标签>

DockerHub 原生拉取命令

docker pull havenoammo/llama:<标签>
专业版 · 高速稳定拉取镜像
50GB 仅 ¥8/年
高速镜像下载在线技术支持专业版保障服务可用付费会员免广告