rhoninseiei/sglang-qwen35-nvfp4

sglang-qwen35-nvfp4

Runtime image for serving Qwen3.5-family checkpoints quantized with NVIDIA ModelOpt NVFP4 by using SGLang on NVIDIA Blackwell GPUs.

Runtime versions

• sglang=0.5.9
• transformers=5.3.0.dev0
• base image: lmsysorg/sglang:latest-runtime

Recommended image tags:

• rhoninseiei/sglang-qwen35-nvfp4:latest
• rhoninseiei/sglang-qwen35-nvfp4:sglang0.5.9-transformers5.3.0dev0

Note:

• Docker image tags cannot contain +, so the versioned tag uses - instead of +.

What this image is for

This image is intended for scenarios similar to the following:

• Qwen3.5 multimodal or dense-family Hugging Face checkpoints
• offline quantization already completed with NVIDIA ModelOpt
• exported Hugging Face checkpoint contains hf_quant_config.json
• SGLang serving with --quantization modelopt_fp4
• Blackwell GPU systems that require --attention-backend triton

This image bakes in a newer Transformers source tree so qwen3_5 architectures can be recognized even when the upstream runtime image lags behind.

Base image

This image is derived from:

• lmsysorg/sglang:latest-runtime

Additional runtime adjustments:

• inject newer transformers source via PYTHONPATH
• install newer huggingface_hub and hf-xet
• default launch entrypoint configured for ModelOpt FP4/NVFP4 checkpoints

Supported use case validated in local testing

Validated against a locally quantized checkpoint derived from:

• crownelius/Crow-9B-Opus-4.6-Distill-Heretic_Qwen3.5

Quantization/export characteristics used in testing:

• NVIDIA ModelOpt output format
• qformat=nvfp4_mlp_only
• kv_cache_qformat=fp8
• Hugging Face export with hf_quant_config.json
• architecture recognized as Qwen3_5ForConditionalGeneration

Observed working behaviors in local tests:

• text chat in Chinese, Japanese, English, and mixed-language prompts
• multimodal requests with image understanding
• simple concurrent requests
• long-context retrieval after template adjustment to suppress reasoning leakage

Requirements

• NVIDIA Blackwell GPU
• NVIDIA container runtime configured on host
• Docker with GPU support
• Quantized model directory mounted into /models

Example run

bash
docker run -d \
  --name sglang_qwen35_nvfp4 \
  --gpus all \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -p 31000:30000 \
  -v /mnt/s/LLM/models:/models \
  rhoninseiei/sglang-qwen35-nvfp4:latest

Version-pinned equivalent:

bash
docker run -d \
  --name sglang_qwen35_nvfp4 \
  --gpus all \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -p 31000:30000 \
  -v /mnt/s/LLM/models:/models \
  rhoninseiei/sglang-qwen35-nvfp4:sglang0.5.9-transformers5.3.0dev0

Optional conservative memory setting:

bash
docker run -d \
  --name sglang_qwen35_nvfp4 \
  --gpus all \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e MEM_FRACTION_STATIC=0.75 \
  -p 31000:30000 \
  -v /mnt/s/LLM/models:/models \
  rhoninseiei/sglang-qwen35-nvfp4:latest

Default entrypoint behavior:

• MODEL_PATH=/models/Crow-9B-Opus-4.6-Distill-Heretic_Qwen3.5/nvfp4_mlp_only_kv_fp8_hf
• PORT=30000
• TP=1
• --quantization modelopt_fp4
• --attention-backend triton
• --trust-remote-code

If your model lives elsewhere, override MODEL_PATH:

bash
docker run -d \
  --name sglang_qwen35_nvfp4 \
  --gpus all \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e MODEL_PATH=/models/your_model_dir \
  -p 31000:30000 \
  -v /path/to/models:/models \
  rhoninseiei/sglang-qwen35-nvfp4:latest

Notes

• This image targets ModelOpt FP4/NVFP4 style checkpoints, not arbitrary quantization formats.
• Current launch defaults assume Blackwell and use triton attention backend accordingly.
• A newer Transformers tree is injected because some upstream runtime images do not yet recognize qwen3_5 architectures out of the box.
• Model quality, instruction-following quality, and multimodal behavior still depend on the underlying checkpoint.

docker-compose example

yaml
services:
  sglang-qwen35-nvfp4:
    image: rhoninseiei/sglang-qwen35-nvfp4:sglang0.5.9-transformers5.3.0dev0
    container_name: sglang_qwen35_nvfp4
    ipc: host
    ports:
      - "31000:30000"
    environment:
      MODEL_PATH: /models/Crow-9B-Opus-4.6-Distill-Heretic_Qwen3.5/nvfp4_mlp_only_kv_fp8_hf
      TP: "1"
      # MEM_FRACTION_STATIC: "0.75"
    volumes:
      - /mnt/s/LLM/models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ulimits:
      memlock: -1
      stack: 67108864
    restart: unless-stopped

Disclaimer

• This is a convenience runtime image, not an official release from SGLang, NVIDIA, Qwen, or the model author.
• No claim is made that every Qwen3.5-derived checkpoint or every NVFP4 export will work unchanged.
• Users remain responsible for complying with all upstream licenses, model licenses, export controls, and deployment requirements.
• The image is provided as-is without any warranty of correctness, fitness, or support.