wdunn001/codec-sglang

codec-sglang

SGLang inference server with Codec token-native binary transport plus the codec-supervisor control plane. One container, GPU-accelerated, OpenAI-compatible.

License: https://github.com/wdunn001/codec-supervisor/blob/main/LICENSE for the codec-supervisor wrapper. The bundled https://github.com/sgl-project/sglang remains under Apache-2.0. Production use under BUSL-1.1 is permitted up to USD $5M annual gross revenue; above that, contact ***.

Quick start

bash
docker run -d --gpus all \
  -p 8080:8080 \
  -v codec-models:/models \
  -v codec-hf-cache:/root/.cache/huggingface \
  -e CODEC_INITIAL_MODEL=Qwen/Qwen2.5-0.5B-Instruct \
  --shm-size 8g \
  wdunn001/codec-sglang:latest

Then:

bash
# OpenAI-compatible
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"x","prompt":"Hello","max_tokens":20}'

# Codec wire format (msgpack frames of token IDs — ~200x smaller)
curl http://localhost:8080/v1/completions \
  -d '{"model":"x","prompt":"Hello","max_tokens":20,"stream":true,"stream_format":"msgpack"}'

Run your own model

Three ways:

1. Override CODEC_INITIAL_MODEL — any HF repo id:

bash
docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=meta-llama/Llama-3.1-8B-Instruct \
  -e HF_TOKEN=hf_xxx \
  wdunn001/codec-sglang:latest

2. Bind-mount a local checkpoint — fine-tunes you don't want on HF:

bash
docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=/models/my-finetune \
  -v /path/to/my-finetune:/models/my-finetune:ro \
  wdunn001/codec-sglang:latest

3. Hot-swap via admin API — supervisor stays up, model swaps:

bash
# Pull from HF
curl -X POST http://localhost:8080/admin/models/pull \
  -H "Content-Type: application/json" \
  -d '{"repo_id": "Qwen/Qwen2.5-7B-Instruct"}'

# Upload a tarball
tar -cf my-finetune.tar -C ./checkpoints/my-finetune .
curl -X POST "http://localhost:8080/admin/models/upload?name=my-finetune" \
  -F "file=@my-finetune.tar"

# Hot-swap
curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name":"my-finetune"}'

Pass CODEC_BACKEND_ARGS to tune sglang (--tp 2 --quantization fp8 ...).

What's inside

• Base: lmsysorg/sglang:latest
• Codec patches: https://github.com/sgl-project/sglang/pull/24483 (token-native binary transport) and https://github.com/sgl-project/sglang/pull/24557 (server-side ToolWatcher), applied as an editable overlay so all upstream kernels (flash-attn, sgl_kernel, triton) stay intact.
• Control plane: https://github.com/wdunn001/codec-supervisor — FastAPI admin on :8080 for model upload, HF pull, hot-swap, and an OpenAI-compatible reverse proxy.

Admin endpoints

Source & docs

• Image source: https://github.com/wdunn001/codec-supervisor
• Full docs: [***]
• Codec protocol: [***]
• Upstream PRs: https://github.com/sgl-project/sglang/pull/24483, https://github.com/sgl-project/sglang/pull/24557