chava2311/bitnet-server

BitNet-b1.58-2B-4T Inference Server

This image provides a lightweight, optimized environment for running inference on the BitNet-b1.58-2B model using bitnet.cpp. Built with a multi-stage Dockerfile, the image is stripped of build dependencies to ensure a minimal footprint while maintaining high performance for CPU-based LLM execution.

🚀 Key Features

• Architecture: Based on Python 3.12-slim for a balance between compatibility and size.
• Ternary Weights: Leverages BitNet's 1.58-bit technology for high-efficiency inference.
• Ready-to-Use: Includes an integrated inference server and the pre-quantized i2_s model.
• Stability Patches: Includes source-level fixes for ggml-bitnet-mad.cpp and *** conversion scripts to ensure reliable deployment.

🛠️ Quick Start

You can deploy the inference server with a single command:

bash
docker run -d -p 8080:8080 --name bitnet-server \
  -e THREADS=4 \
  -e CONTEXT_SIZE=4096 \
  chava2311/bitnet-b1.58-server

⚙️ Configuration (Environment Variables)

The image is highly customizable through environment variables:

🏗️ Build Process Details

The image uses a sophisticated build pipeline:

1. Build Stage: Installs cmake, clang, and build-essential. It applies critical patches to the bitnet.cpp source code and runs the setup_env.py script to compile the ***aries and quantize the model to the i2_s format.
2. App Stage: Extract only the necessary shared libraries (libggml.so, libllama.so), the llama-server ***ary, and the quantized model, resulting in a production-ready runtime environment.

⚠️ Performance Note

As this model runs on the CPU, performance is bound by your processor's speed and the number of assigned THREADS. Note that as the chat context grows, the initial "Time to First Token" (TTFT) will increase. For high-traffic environments, *** limiting CONTEXT_SIZE to maintain low latency.

Base Image: python:3.12-slim
Core Engine: bitnet.cpp (LLM inference)
Model format: *** (i2_s quantization)