yusiwen/llama.cpp Docker 镜像 - 轩辕镜像 | Docker 镜像高效稳定拉取服务
yusiwen/llama.cppyusiwen
llama.cpp basic build w/wo CUDA/OpenCL support
8 次收藏下载次数: 0状态：社区镜像维护者：yusiwen仓库类型：镜像最近更新：11 天前
My docker image of llama.cpp.
It is a minimal build which can run on CPU/GPU for small LLM models.
Basic usages
For CPU inferencing:
bash# check version $ docker run --rm yusiwen/llama.cpp:latest /main --version version: 1879 (3e5ca79) built with cc (GCC) 9.5.0 for x86_64-linux-gnu # main $ docker run --rm -v /opt/data/ai/models:/models yusiwen/llama.cpp:latest /llama-cli -m /models/mistral-7b-v0.1.Q4_K_M.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e Log start main: build = 1879 (3e5ca79) main: built with cc (GCC) 9.5.0 for x86_64-linux-gnu main: seed = *** llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /models/mistral-7b-v0.1.Q4_K_M.gguf (version GGUF V2) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = mistralai_mistral-7b-v0.1 llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = *** llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = ***.000000 llama_model_loader: - kv 11: general.file_type u32 = 15 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 19: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V2 llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = *** llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = ***.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 4.07 GiB (4.83 BPW) llm_load_print_meta: general.name = mistralai_mistral-7b-v0.1 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.11 MiB llm_load_tensors: offloading 0 repeating layers to GPU llm_load_tensors: offloaded 0/33 layers to GPU llm_load_tensors: CPU buffer size = 4165.37 MiB ............................................................................................... llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: freq_base = ***.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CPU KV buffer size = 64.00 MiB llama_new_context_with_model: KV self size = 64.00 MiB, K (f16): 32.00 MiB, V (f16): 32.00 MiB llama_new_context_with_model: graph splits (measure): 1 llama_new_context_with_model: CPU compute buffer size = 73.00 MiB system_info: n_threads = 6 / 12 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | sampling: repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000 top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800 mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000 sampling order: CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp generate: n_ctx = 512, n_batch = 512, n_predict = 400, n_keep = 0 Building a website can be done in 10 simple steps: Step 1: Pick your website name The first step of building any website is to pick the website name you want. This is also known as a URL or domain. The most common URLs are .com, .net and .org. If you’re looking for something specific like a restaurant, then try using their local extension such as .ca for Canada. Step 2: Set up your hosting account with the right amount of bandwidth and disk space In order to set up your website on a server, you will need a hosting account. This is where all the files that make up your site live (images, videos, etc.). You can find many different companies online who offer these services at varying prices depending upon what features they offer. Some examples include GoDaddy or BlueHost. Step 3: Designing Your Site Layout – Choose Themes & Plugins To Install On WordPress Website Now that we have our hosting set up, it’s time to start designing our site layout! There are two main ways of doing this: using themes or building custom templates from scratch. Themes provide pre-made designs for you to choose from while custom template builders allow complete control over how things look like on any given page/post within the site itself – think about it like programming languages versus HTML code. Both methods have their pros and cons; however, most people prefer using themes because they offer more flexibility when changing layouts without having any coding knowledge at all! Step 4: Creating Pages For Your Website – Use WordPress Post Editor Or Create Custom Page Types On The Frontend With WooCommerce Plugin Now that you’ve designed your site layout, it’s time to start creating pages for it. There are two main ways of doing this: using the default post editor or creating custom page types on the frontend with WooCommerce plugin (if you need e-commerce features .... llama_print_timings: load time = 448.09 ms llama_print_timings: sample time = 64.36 ms / 400 runs ( 0.16 ms per token, 6215.33 tokens per second) llama_print_timings: prompt eval time = 965.08 ms / 19 tokens ( 50.79 ms per token, 19.69 tokens per second) llama_print_timings: eval time = 42130.65 ms / 399 runs ( 105.59 ms per token, 9.47 tokens per second) llama_print_timings: total time = 43288.23 ms / 418 tokens Log end
For GPU inferencing, use the image tagged with -cuda:
bash$ docker run --rm -v /opt/data/ai/models:/models yusiwen/llama.cpp:latest-cuda /llama-cli -m /models/mistral-7b-v0.1.Q4_K_M.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -ngl 50 ...
Disclaimer
This image is builded only for my personal purpose of testing LLM inference on difference CPUs and GPUs in my own automation pipelines.
Use at your own risks.
rocm/llama.cpp
rocm
暂无描述
5 次收藏8.5千+ 次下载
4 个月前更新
amperecomputingai/llama.cpp
amperecomputingai
Ampere® optimized llama.cpp with full support for GGUF models available on HuggingFace.
2 次收藏5.6千+ 次下载
1 个月前更新
dimaskiddo/llama.cpp
dimaskiddo
Debian Based LLAMA.CPP Image Repository
1.7千+ 次下载
2 个月前更新
gclub/llama.cpp
gclub
SkywardAI community forked llama.cpp repo
993 次下载
1 年前更新
dynafire/llama.cpp
dynafire
llama.cpp with env exposed config, suitable for RunPod
1.3千+ 次下载
1 年前更新
iaishow/llama.cpp
iaishow
暂无描述
4.2千+ 次下载
2 个月前更新
镜像拉取常见问题
使用与功能问题
错误码与失败问题
manifest unknown 错误：镜像不存在或标签错误
manifest unknown 错误
TLS/SSL 证书验证失败：Docker pull 时 HTTPS 证书错误
TLS 证书验证失败
DNS 解析超时：无法解析镜像仓库地址或连接超时
DNS 解析超时
410 Gone 错误：Docker 版本过低导致协议不兼容
410 错误：版本过低
402 Payment Required 错误：流量耗尽错误提示
402 错误：流量耗尽
401 UNAUTHORIZED 错误：身份认证失败或登录信息错误
身份认证失败错误
429 Too Many Requests 错误：请求频率超出专业版限制
429 限流错误
Docker login 凭证保存错误：Cannot autolaunch D-Bus（不影响登录）
凭证保存错误
账号 / 计费 / 权限
用户好评
来自真实用户的反馈，见证轩辕镜像的优质服务
oldzhang
运维工程师
Linux服务器
5
"Docker访问体验非常流畅，大镜像也能快速完成下载。"