ghcr.io/ggml-org/llama.cpp:server-vulkan-b9859

llama.cpp

C/C++实现的LLM推理

近期API变更

• https://github.com/ggml-org/llama.cpp/issues/9289
• https://github.com/ggml-org/llama.cpp/issues/9291

热门话题

• Hugging Face 缓存迁移：使用 -hf 下载的模型现在存储在标准 Hugging Face 缓存目录中，可与其他 HF 工具共享。
• https://github.com/ggml-org/llama.cpp/discussions/***
• https://github.com/ggml-org/llama.cpp/discussions/***
• https://github.com/ggml-org/llama.cpp/discussions/***
• llama-server 现已支持多模态：https://github.com/ggml-org/llama.cpp/pull/*** | 文档
• 用于 FIM 补全的 VS Code 扩展：https://github.com/ggml-org/llama.vscode
• 用于 FIM 补全的 Vim/Neovim 插件：https://github.com/ggml-org/llama.vim
• Hugging Face *** 编辑器：https://github.com/ggml-org/llama.cpp/discussions/9268 | https://huggingface.co/spaces/CISCai/***-editor
• 浏览器现已支持 WebGPU，查看介绍它的博客/演示 https://reeselevine.github.io/llamas-on-the-web/%E3%80%82

---

• LLaMA 🦙
• LLaMA 2 🦙🦙
• LLaMA 3 🦙🦙🦙
• https://huggingface.co/***/***-7B-v0.1
• https://huggingface.co/models?search=***-ai/***
• https://huggingface.co/databricks/dbrx-instruct
• https://huggingface.co/ai21labs
• https://huggingface.co/models?search=tiiuae/***
• https://github.com/ymcui/Chinese-LLaMA-Alpaca 和 https://github.com/ymcui/Chinese-LLaMA-Alpaca-2
• https://github.com/bofenghuang/vigogne
• https://github.com/ggml-org/llama.cpp/pull/5423
• Koala
• https://huggingface.co/models?search=baichuan-inc/Baichuan + https://huggingface.co/hiyouga/baichuan-7b-sft
• https://huggingface.co/models?search=BAAI/Aquila
• https://github.com/ggml-org/llama.cpp/pull/3187
• https://huggingface.co/smallcloudai/Refact-1_6B-fim
• https://github.com/ggml-org/llama.cpp/pull/3417
• https://github.com/ggml-org/llama.cpp/pull/3553
• https://huggingface.co/models?search=01-ai/Yi
• https://huggingface.co/***
• https://huggingface.co/models?search=deepseek-ai/deepseek
• https://huggingface.co/models?search=Qwen/Qwen
• https://github.com/ggml-org/llama.cpp/pull/3557
• https://huggingface.co/models?search=microsoft/phi
• https://github.com/ggml-org/llama.cpp/pull/***
• https://huggingface.co/gpt2
• https://github.com/ggml-org/llama.cpp/pull/5118
• https://huggingface.co/models?search=internlm2
• https://github.com/WisdomShell/codeshell
• https://ai.google.dev/***
• https://github.com/state-spaces/mamba
• https://huggingface.co/keyfan/grok-1-hf
• https://huggingface.co/models?search=xverse
• https://huggingface.co/models?search=CohereForAI/c4ai-command-r
• https://huggingface.co/models?search=sea-lion
• https://huggingface.co/GritLM/GritLM-7B + https://huggingface.co/GritLM/GritLM-8x7B
• OLMo
• OLMo 2
• https://huggingface.co/allenai/OLMoE-1B-7B-0924
• https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330
• https://github.com/Eleuther***-neox + https://github.com/EleutherAI/pythia
• https://huggingface.co/collections/***/arctic-66290090abe542894a5ac520
• https://huggingface.co/models?search=Smaug
• https://huggingface.co/LumiOpen/Poro-34B
• https://huggingface.co/1bitLLM
• https://huggingface.co/models?search=flan-t5
• https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca
• https://huggingface.co/collections/THUDM/glm-4-0414-67f3cbcb34dd9d252707cb2e
• https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce***
• https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct
• https://huggingface.co/collections/tiiuae/***mamba-7b-66b9a580324dd1598b0f6d4a
• https://huggingface.co/inceptionai/jais-13b-chat
• https://huggingface.co/collections/speakleash/bielik-11b-v23-66ee813238d9b526a072408a
• https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-***
• https://github.com/BlinkDL/RWKV-LM
• https://huggingface.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1
• https://huggingface.co/ai-sage/GigaChat-20B-A3B-instruct
• https://huggingface.co/trillionlabs/Trillion-7B-preview
• https://huggingface.co/collections/inclusionAI/ling-67c51c85b34a7ea0aba94c32
• https://huggingface.co/collections/LiquidAI/lfm2
• https://huggingface.co/collections/LiquidAI/lfm25
• https://huggingface.co/collections/LiquidAI/liquid-nanos
• https://huggingface.co/collections/tencent/hunyuan-dense-model-6890632cda26b***c9c5e7
• https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86
• https://huggingface.co/JetBrains/models?search=mellum

多模态

• https://huggingface.co/collections/liuhaotian/***-15-653aac15d994e992e2677a7e%E3%80%81https://huggingface.co/collections/liuhaotian/***-16-65b9e40155f60fd046a5ccf2
• https://huggingface.co/models?search=SkunkworksAI/Bak***
• https://huggingface.co/NousResearch/Obsidian-3B-V0.5
• https://huggingface.co/models?search=Lin-Chen/ShareGPT4V
• https://huggingface.co/models?search=mobileVLM
• https://huggingface.co/models?search=Yi-VL
• https://huggingface.co/models?search=MiniCPM
• https://huggingface.co/vikhyatk/moondream2
• https://github.com/BAAI-DCAI/Bunny
• https://huggingface.co/models?search=glm-edge
• https://huggingface.co/collections/Qwen/qwen2-vl-66cee7455501d7126940800d
• https://huggingface.co/collections/LiquidAI/lfm2-vl-68963bbc84a610f7638d5ffa

绑定

• Python: https://github.com/ddh0/easy-llama
• Python: https://github.com/abetlen/llama-cpp-python
• Go: https://github.com/go-skynet/go-llama.cpp
• Node.js: https://github.com/withcatai/node-llama-cpp
• JS/TS（llama.cpp 服务器客户端）: lgrammel/modelfusion
• JS/TS（可编程提示引擎 CLI）: https://github.com/offline-ai/cli
• JavaScript/Wasm（可在浏览器中运行）: https://github.com/tangledgroup/llama-cpp-wasm
• TypeScript/Wasm（更友好的 API，可在 npm 上获取）: https://github.com/ngxson/wllama
• Ruby: https://github.com/yoshoku/llama_cpp.rb
• Ruby: https://github.com/docusealco/rllama
• Rust（功能更丰富）: https://github.com/edgenai/llama_cpp-rs
• Rust（更友好的 API）: https://github.com/mdrokz/rust-llama.cpp
• Rust（更直接的绑定）: https://github.com/utilityai/llama-cpp-rs
• Rust（从 crates.io 自动构建）: https://github.com/ShelbyJenkins/llm_client
• C#/.NET: https://github.com/SciSharp/LLamaSharp
• C#/VB.NET（功能更丰富 - 社区许可证）: LM-Kit.NET
• Scala 3: https://github.com/donderom/llm4s
• Clojure: https://github.com/phronmophobic/llama.clj
• React Native: https://github.com/mybigday/llama.rn
• Java: https://github.com/kherud/java-llama.cpp
• Java: https://github.com/QuasarByte/llama-cpp-jna
• Zig: https://github.com/Deins/llama.cpp.zig
• Flutter/Dart: https://github.com/netdur/llama_cpp_dart
• Flutter: https://github.com/xuegao-tzx/Fllama
• PHP（基于 llama.cpp 构建的 API 绑定和功能）: https://github.com/distantmagic/resonance https://github.com/ggml-org/llama.cpp/pull/6326
• Guile Scheme: https://savannah.nongnu.org/projects/guile-llama-cpp
• Swift: https://github.com/srgtuszy/llama-cpp-swift
• Swift: https://github.com/ShenghaiWang/SwiftLlama
• Delphi: https://github.com/Embarcadero/llama-cpp-delphi
• Go（无需 CGo）: https://github.com/hybridgroup/yzma
• Android: llama.android

UIs

(若要在此列出项目，该项目应明确声明依赖于 llama.cpp)

• https://github.com/yaroslavyaroslav/OpenAI-sublime-text%EF%BC%88MIT%EF%BC%89
• https://apps.apple.com/us/app/bonzai-your-local-ai-agent/id6752847988%EF%BC%88%E4%B8%93%E6%9C%89%EF%BC%89
• https://github.com/cztomsik/ava%EF%BC%88MIT%EF%BC%89
• https://github.com/alexpinel/Dot%EF%BC%88GPL%EF%BC%89
• https://github.com/ylsdamxssjxxdd/eva%EF%BC%88MIT%EF%BC%89
• https://github.com/iohub/coLLaMA%EF%BC%88Apache-2.0%EF%BC%89
• https://github.com/janhq/jan%EF%BC%88AGPL%EF%BC%89
• https://github.com/johnbean393/Sidekick%EF%BC%88MIT%EF%BC%89
• https://github.com/zhouwg/kantv?tab=readme-ov-file%EF%BC%88Apache-2.0%EF%BC%89
• https://github.com/firatkiral/kodibot%EF%BC%88GPL%EF%BC%89
• https://github.com/ggml-org/llama.vim%EF%BC%88MIT%EF%BC%89
• https://github.com/abgulati/LARS%EF%BC%88AGPL%EF%BC%89
• https://github.com/vietanhdev/llama-assistant%EF%BC%88GPL%EF%BC%89
• https://github.com/undreamai/LlamaLib%EF%BC%88Apache-2.0%EF%BC%89
• https://github.com/guinmoon/LLMFarm?tab=readme-ov-file%EF%BC%88MIT%EF%BC%89
• https://github.com/undreamai/LLMUnity%EF%BC%88MIT%EF%BC%89
• https://lmstudio.ai/%EF%BC%88%E4%B8%93%E6%9C%89%EF%BC%89
• https://github.com/mudler/LocalAI%EF%BC%88MIT%EF%BC%89
• https://github.com/LostRuins/koboldcpp%EF%BC%88AGPL%EF%BC%89
• MindMac（专有）
• https://github.com/MindWorkAI/AI-Studio%EF%BC%88FSL-1.1-MIT%EF%BC%89
• https://github.com/Mobile-Artificial-Intelligence/maid%EF%BC%88MIT%EF%BC%89
• https://github.com/Mozilla-Ocho/llamafile%EF%BC%88Apache-2.0%EF%BC%89
• https://github.com/nat/openplayground%EF%BC%88MIT%EF%BC%89
• https://github.com/nomic-***4all%EF%BC%88MIT%EF%BC%89
• https://github.com/ollama/ollama%EF%BC%88MIT%EF%BC%89
• https://github.com/oobabooga/text-generation-webui%EF%BC%88AGPL%EF%BC%89
• https://github.com/a-ghorbani/pocketpal-ai%EF%BC%88MIT%EF%BC%89
• https://github.com/psugihara/FreeChat%EF%BC%88MIT%EF%BC%89
• https://github.com/ptsochantaris/emeltal%EF%BC%88MIT%EF%BC%89
• https://github.com/pythops/tenere%EF%BC%88AGPL%EF%BC%89
• https://github.com/containers/ramalama%EF%BC%88MIT%EF%BC%89
• https://github.com/semperai/amica%EF%BC%88MIT%EF%BC%89
• https://github.com/withcatai/catai%EF%BC%88MIT%EF%BC%89
• https://github.com/***89/autopen%EF%BC%88GPL%EF%BC%89

工具

• https://github.com/akx/ggify – 从 Hugging Face Hub 下载 PyTorch 模型并将其转换为 GGML
• https://github.com/akx/ollama-dl – 从 Ollama 库下载模型以直接与 llama.cpp 一起使用
• https://github.com/crashr/gppm – 启动利用 NVIDIA Tesla P40 或 P100 GPU 的 llama.cpp 实例，降低空闲功耗
• https://github.com/gpustack/***-parser-go/tree/main/cmd/***-parser - 查看/检查 *** 文件并估算内存使用情况
• Styled Lines（专有许可，Unity3d 游戏开发推理部分的异步包装器，带有预构建的移动和 Web 平台包装器以及模型示例）
• https://github.com/***ai/*** – 🦥 将微调及训练后的模型导出/保存为 ***（Apache-2.0）

基础设施

• https://github.com/intentee/paddler - 开源 LLMOps 平台，用于在自有基础设施中托管和扩展 AI
• https://github.com/gpustack/gpustack - 管理用于运行 LLM 的 GPU 集群
• https://github.com/onicai/llama_cpp_canister - 在 Internet Computer 上作为智能合约的 llama.cpp，使用 WebAssembly
• https://github.com/mostlygeek/llama-swap - 透明代理，为 llama-server 添加自动模型切换功能
• https://github.com/kalavai-net/kalavai-client - 众包端到端 LLM 部署，支持任意规模
• https://github.com/InftyAI/llmaz - ☸️ 简单、高级的 Kubernetes 大语言模型推理平台。
• https://github.com/defilantech/llmkube - 用于 llama.cpp 的 Kubernetes 算子，支持多 GPU 和 Apple Silicon Metal

游戏

llama-cli

一个用于访问和试验 llama.cpp 大部分功能的 CLI 工具。

以对话模式运行

具有内置聊天模板的模型会自动激活对话模式。如果未自动激活，可通过添加 -cnv 并使用 --chat-template NAME 指定合适的聊天模板来手动启用。

llama-cli -m model.gguf

#
> hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
#
> what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!

使用自定义聊天模板运行对话模式

# 使用 "chatml" 模板（使用 -h 查看支持的模板列表）
llama-cli -m model.gguf -cnv --chat-template chatml

# 使用自定义模板
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'

使用自定义语法约束输出

llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}

grammars/ 文件夹包含一些示例语法。要编写自己的语法，请查看 GBNF 指南。

如需编写更复杂的 JSON 语法，请访问 [***]

llama-server

一个轻量级、与 https://github.com/openai/openai-openapi 兼容的 LLM 服务 HTTP 服务器。

使用默认配置在 8080 端口启动本地 HTTP 服务器

llama-server -m model.gguf --port 8080

# 可通过浏览器访问基本 Web UI：http://localhost:8080
# 聊天补全端点：http://localhost:8080/v1/chat/completions

支持多用户和并行解码

# 最多 4 个并发请求，每个请求最大上下文为 4096
llama-server -m model.gguf -c 16384 -np 4

启用推测解码

# draft.gguf 模型应为目标 model.gguf 的小型变体
llama-server -m model.gguf -md draft.gguf

提供嵌入模型服务

# 使用 /embedding 端点
llama-server -m model.gguf --embedding --pooling cls -ub 8192

提供重排序模型服务

# 使用 /reranking 端点
llama-server -m model.gguf --reranking

使用语法约束所有输出

# 自定义语法
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

XCFramework

XCFramework 是适用于 iOS、visionOS、tvOS 和 macOS 的预编译库版本。它可用于 Swift 项目，无需从源代码编译库。例如：

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
name: "MyLlamaPackage",
targets: [
.executableTarget(
name: "MyLlamaPackage",
dependencies: [
"LlamaFramework"
]),
.binaryTarget(
name: "LlamaFramework",
url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
)
]
)

上述示例使用的是库的中间构建版本 b5046。可以通过更改 URL 和校验和来修改为使用不同版本。

命令补全

部分环境支持命令行补全。

Bash 补全

$ build/bin/llama-cli --completion-bash
> ~/.llama-completion.bash
$ source ~/.llama-completion.bash

也可以选择将其添加到 .bashrc 或 .bash_profile 中以自动加载。例如：

$ echo "source ~/.llama-completion.bash"
>> ~/.bashrc

依赖项

• https://github.com/yhirose/cpp-httplib - 单头文件 HTTP 服务器，供 llama-server 使用 - MIT 许可证
• https://github.com/nothings/stb - 单头文件图像格式解码器，供多模态子系统使用 - 公共领域
• https://github.com/nlohmann/json - 单头文件 JSON 库，供各种工具/示例使用 - MIT 许可证
• https://github.com/mackron/miniaudio - 单头文件音频格式解码器，供多模态子系统使用 - 公共领域
• https://github.com/sheredom/subprocess.h - 用于 C 和 C++ 的单头文件进程启动解决方案 - 公共领域