mekayelanik/vllm-cpu Docker 镜像 - 轩辕镜像 | Docker 镜像高效稳定拉取服务

Easy, fast, and cheap LLM serving for everyone
Docker Images for CPU-Only Inference

Documentation | Docker Hub | GitHub | Supported Models

---

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

---

Overview

Pre-built Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets.

Features: OpenAI-Compatible API, CPU Optimizations (AVX512, VNNI, BF16, AMX), Multi-Architecture, Health Checks, PUID/PGID.

Requirements: 4+ GB RAM (16+ recommended), 4+ CPU cores, Docker 20.10+, 2+ GB shm.

---

Image Variants

Choose the appropriate variant based on your CPU's instruction set support:

Available Tags

Each variant has two tag formats:

• Version-specific: <variant>-<version> (e.g., avx512bf16-0.12.0)
• Latest: <variant>-latest (e.g., avx512bf16-latest)

Check Your CPU Support

bash
# Check available instruction sets
lscpu | grep -E "avx512|vnni|amx"

# Or use flags
grep -o 'avx512[a-z_]*\|amx[a-z_]*' /proc/cpuinfo | sort -u

---

Quick Start

Docker CLI

bash
# Pull the image
docker pull mekayelanik/vllm-cpu:noavx512-latest

# Run with a small model
docker run -d \
  --name vllm-cpu \
  --restart unless-stopped \
  --cap-add SYS_NICE \
  --security-opt seccomp=unconfined \
  --shm-size 4g \
  -p 8000:8000 \
  -v vllm-data:/data \
  -e VLLM_MODEL=Qwen/Qwen3-0.6B \
  -e VLLM_SERVER_HOST=0.0.0.0 \
  -e VLLM_SERVER_PORT=8000 \
  mekayelanik/vllm-cpu:noavx512-latest \
  --max-model-len 8192

# Check logs
docker logs -f vllm-cpu

# Test the API
curl http://localhost:8000/v1/models

API Endpoints

---

Docker Compose Examples

Standard Bridge Network (Recommended)

This is the simplest and most portable configuration, suitable for most deployments.

yaml
# docker-compose.yml - Standard Bridge Network
# Save this file and run: docker compose up -d

services:
  vllm-cpu:
    image: mekayelanik/vllm-cpu:noavx512-latest
    container_name: vllm-cpu
    hostname: vllm-cpu
    domainname: local
    restart: unless-stopped

    # Required capabilities for CPU optimization
    cap_add:
      - SYS_NICE
    security_opt:
      - seccomp=unconfined

    # Shared memory for model loading
    shm_size: 4g

    # Port mapping
    ports:
      - "8000:8000"

    # Persistent storage
    volumes:
      - vllm-data:/data
      - vllm-cache:/data/cache
      # Optional: Mount local HuggingFace cache
      # - ${HOME}/.cache/huggingface:/data/models

    # Environment configuration
    environment:
      # User/Group mapping (optional)
      - PUID=1000
      - PGID=1000
      - TZ=UTC

      # Model configuration
      - VLLM_MODEL=Qwen/Qwen3-0.6B
      # - HF_TOKEN=your_huggingface_token  # For gated models

      # Server configuration
      - VLLM_SERVER_HOST=0.0.0.0
      - VLLM_SERVER_PORT=8000
      # - VLLM_API_KEY=your_api_key  # Optional API key

      # Logging
      - VLLM_LOGGING_LEVEL=INFO
      - VLLM_CONFIGURE_LOGGING=1

      # Memory optimization
      - MALLOC_TRIM_THRESHOLD_=***

      # CPU-specific optimization (adjust based on your system)
      # KV cache size in GB - increase for more concurrent requests
      # Default is 4GB. Formula: (RAM - Model Size - 8GB) / 2
      - VLLM_CPU_KVCACHE_SPACE=8

    # vLLM arguments (passed to entrypoint)
    command:
      - "--max-model-len"
      - "8192"
      - "--dtype"
      - "auto"

    # Health check is built into the Docker image (30s interval, 120s start period)

    # Resource limits (optional)
    deploy:
      resources:
        limits:
          memory: 16G
        reservations:
          memory: 4G

# Named volumes for persistence
volumes:
  vllm-data:
    driver: local
  vllm-cache:
    driver: local

Deploy:

bash
# Start the service
docker compose up -d

# View logs
docker compose logs -f vllm-cpu

# Check status
docker compose ps

# Stop the service
docker compose down

---

MACVLAN Network (Advanced)

MACVLAN gives the container its own IP address on your local network, making it appear as a separate physical device. This is useful for:

• Direct LAN access without port forwarding
• Running multiple vLLM instances with unique IPs
• Integration with network services that require dedicated IPs

Prerequisites:

• Linux host (MACVLAN doesn't work on Docker Desktop for Mac/Windows)
• Know your network interface name (e.g., eth0, ens18, enp0s3)
• Available IP address in your network range
• Promiscuous mode enabled on network interface (for some setups)

yaml
# docker-compose-macvlan.yml - MACVLAN Network Configuration
# Save this file and run: docker compose -f docker-compose-macvlan.yml up -d

services:
  vllm-cpu:
    image: mekayelanik/vllm-cpu:avx512bf16-latest
    container_name: vllm-cpu
    hostname: vllm-cpu
    domainname: local
    restart: unless-stopped

    # Required capabilities
    cap_add:
      - SYS_NICE
    security_opt:
      - seccomp=unconfined

    # Shared memory
    shm_size: 4g

    # MAC address (generate unique one for your network)
    mac_address: "02:42:c0:a8:01:64"

    # MACVLAN network with static IP
    networks:
      vllm-macvlan:
        ipv4_address: 192.168.1.100

    # Persistent storage
    volumes:
      - vllm-data:/data
      - vllm-cache:/data/cache
      # Mount local models directory
      - /path/to/models:/data/models:ro

    # Environment configuration
    environment:
      # User/Group mapping
      - PUID=1000
      - PGID=1000
      - TZ=UTC

      # Model configuration
      - VLLM_MODEL=Qwen/Qwen3-0.6B
      - HF_HOME=/data/models
      # - HF_TOKEN=your_huggingface_token

      # Server configuration (bind to all interfaces)
      - VLLM_SERVER_HOST=0.0.0.0
      - VLLM_SERVER_PORT=8000

      # Logging
      - VLLM_LOGGING_LEVEL=INFO
      - VLLM_CONFIGURE_LOGGING=1

      # Performance tuning
      - MALLOC_TRIM_THRESHOLD_=***
      - OMP_NUM_THREADS=8
      - MKL_NUM_THREADS=8

      # CPU-specific optimization
      # KV cache size in GB - adjust based on your RAM
      - VLLM_CPU_KVCACHE_SPACE=12

    # vLLM arguments
    command:
      - "--max-model-len"
      - "***"
      - "--dtype"
      - "bfloat16"
      - "--cpu-offload-gb"
      - "0"

    # Health check is built into the Docker image (30s interval, 120s start period)

    # Resource limits
    deploy:
      resources:
        limits:
          memory: 32G
        reservations:
          memory: 8G

# MACVLAN Network Definition
networks:
  vllm-macvlan:
    driver: macvlan
    driver_opts:
      # Change to your network interface name
      parent: eth0
    ipam:
      driver: default
      config:
        # Adjust to match your network configuration
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1
          # IP range for containers (optional, limits assignable IPs)
          ip_range: 192.168.1.100/30

# Named volumes
volumes:
  vllm-data:
    driver: local
  vllm-cache:
    driver: local

Deploy with MACVLAN:

bash
# Find your network interface: ip link show
# Create and start
docker compose -f docker-compose-macvlan.yml up -d

# Access via dedicated IP
curl [***]

Note: With MACVLAN, the host cannot directly reach the container. Create a macvlan shim interface on the host if needed.

---

High Performance Production Server

This configuration is optimized for high-load, high-concurrency production environments with maximum throughput. Designed for enterprise deployments on high-end Intel Xeon or AMD EPYC servers.

Target Hardware:

• 32+ CPU cores (64+ threads)
• 128+ GB RAM (more RAM = larger KV cache = more concurrent requests)
• NVMe storage for model files
• 10GbE+ networking

Key Optimization: VLLM_CPU_KVCACHE_SPACE

The most important setting for CPU inference is KV cache size. The default 4GB is too small for production. This config sets it to 40GB for high concurrency.

Use Cases:

• Production API serving with hundreds of concurrent users
• High-throughput batch processing
• Enterprise LLM deployments
• Multi-tenant inference services

yaml
# docker-compose-high-performance.yml - High Load Production Configuration
# Optimized for maximum throughput and concurrency
# Run: docker compose -f docker-compose-high-performance.yml up -d

services:
  vllm-cpu-prod:
    image: mekayelanik/vllm-cpu:amxbf16-latest
    container_name: vllm-cpu-prod
    hostname: vllm-prod
    domainname: local
    restart: always

    # Use host network for maximum network performance
    network_mode: host

    # Extended capabilities for performance optimization
    cap_add:
      - SYS_NICE
      - IPC_LOCK
    security_opt:
      - seccomp=unconfined

    # Large shared memory for concurrent request handling
    shm_size: 16g

    # CPU pinning for NUMA optimization (adjust to your topology)
    # cpuset: "0-31"

    # Persistent storage with performance optimizations
    volumes:
      # Use fast NVMe storage for models
      - /mnt/nvme/vllm-data:/data
      - /mnt/nvme/vllm-cache:/data/cache
      - /mnt/nvme/models:/data/models:ro
      # tmpfs for temporary files
      - type: tmpfs
        target: /tmp
        tmpfs:
          size: 4G
          mode: 1777

    # Production environment configuration
    environment:
      # Run as root for maximum performance (or set specific user)
      # - PUID=0
      # - PGID=0
      - TZ=UTC

      # Model configuration - use a powerful model
      - VLLM_MODEL=Qwen/Qwen3-8B
      - HF_HOME=/data/models
      - HF_HUB_OFFLINE=1
      # - HF_TOKEN=your_token

      # Server configuration
      - VLLM_SERVER_HOST=0.0.0.0
      - VLLM_SERVER_PORT=8000
      - VLLM_API_KEY=${VLLM_API_KEY:-}

      # ============================================================
      # CPU-SPECIFIC VLLM OPTIMIZATION (CRITICAL FOR PERFORMANCE)
      # ============================================================
      # KV Cache size in GB - CRITICAL: Default is only 4GB!
      # Set this based on your available RAM and model size
      # Formula: (Total RAM - Model Size - 8GB headroom) / 2
      # Example: 128GB RAM, 16GB model = (128-16-8)/2 = 52GB
      - VLLM_CPU_KVCACHE_SPACE=40

      # OpenMP thread binding for CPU affinity
      # "auto" = automatic binding, or specify cores like "0-31"
      # Improves cache locality and reduces context switching
      - VLLM_CPU_OMP_THREADS_BIND=0-31

      # Reserve CPU cores for framework overhead (scheduler, HTTP server)
      # These cores won't be used for inference, reducing contention
      - VLLM_CPU_NUM_OF_RESERVED_CPU=2
      # ============================================================

      # Extended timeouts for high load
      - VLLM_HTTP_TIMEOUT_KEEP_ALIVE=30
      - VLLM_ENGINE_ITERATION_TIMEOUT_S=300
      - VLLM_RPC_TIMEOUT=30000

      # Logging - reduce verbosity for performance
      - VLLM_CONFIGURE_LOGGING=1
      - VLLM_LOGGING_LEVEL=WARNING
      - VLLM_LOG_STATS_INTERVAL=60

      # Memory optimization - aggressive settings
      - MALLOC_TRIM_THRESHOLD_=65536
      - MALLOC_MMAP_THRESHOLD_=***
      - MALLOC_MMAP_MAX_=65536

      # Threading - set to (physical cores - reserved cores)
      # If you have 32 cores and reserve 2, set this to 30
      - OMP_NUM_THREADS=30
      - MKL_NUM_THREADS=30
      - OMP_PROC_BIND=close
      - OMP_PLACES=cores

      # NUMA optimization (GNU OpenMP)
      - GOMP_CPU_AFFINITY=0-29

      # Use tcmalloc for better memory performance
      - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4

      # Disable usage stats for privacy/performance
      - VLLM_NO_USAGE_STATS=1
      - VLLM_DO_NOT_TRACK=1

    # High-performance vLLM arguments
    command:
      # Context length
      - "--max-model-len"
      - "32768"
      # Data type for Intel AMX
      - "--dtype"
      - "bfloat16"
      # Scheduling for high concurrency
      - "--max-num-seqs"
      - "256"
      - "--max-num-batched-tokens"
      - "32768"
      # Disable request logging for performance
      - "--disable-log-requests"
      # Enable chunked prefill for better latency
      - "--enable-chunked-prefill"
      - "--max-chunked-prefill-len"
      - "8192"
      # Speculative decoding (if supported by model)
      # - "--speculative-model"
      # - "path/to/draft/model"
      # - "--num-speculative-tokens"
      # - "5"

    # Health check is built into the Docker image (30s interval, 120s start period)

    # No resource limits - use all available resources
    # deploy:
    #   resources:
    #     limits:
    #       memory: 120G
    #     reservations:
    #       memory: 64G

    # Logging configuration
    logging:
      driver: json-file
      options:
        max-size: "100m"
        max-file: "5"
        compress: "true"

    # Ulimits for high concurrency
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: ***
        hard: ***
      nproc:
        soft: 65535
        hard: 65535

Deploy:

bash
# Verify CPU supports AMX: lscpu | grep -i amx
# Set kernel params: sudo sysctl -w vm.swappiness=1
docker compose -f docker-compose-high-performance.yml up -d
docker stats vllm-cpu-prod

---

Environment Variables

Core Settings

Model Configuration

Server Configuration

Logging

Performance

CPU-Specific Optimization (Critical)

These environment variables are specific to vLLM CPU inference and can significantly impact performance. See vLLM CPU FAQ for details.

KV Cache Sizing Guide:

---

Volume Mounts

---

Common Arguments

---

Troubleshooting

---

Updating Images

bash
# Docker Compose
docker compose pull && docker compose up -d

# Docker CLI
docker pull mekayelanik/vllm-cpu:noavx512-latest
docker stop vllm-cpu && docker rm vllm-cpu
# Recreate with new image

# Clean old images
docker image prune -f

---

Support & License

• Issues: GitHub | Docs: docs.vllm.ai
• License: Docker Images (GPL-3.0), vLLM Project (Apache 2.0)
• Registries: Docker Hub | GHCR

---

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

---

Back to Top