openeuler/euler-copilot-fast-inference Docker 镜像 - 轩辕镜像 | Docker 镜像高效稳定拉取服务

Quick reference

euler-copilot-fast-inference | openEuler

An inference framework that can be used for large-scale CPU inference, based on openEuler basic image, containerized package after release, can be used for one-click deployment

Supported tags and respective Dockerfile links

Usage：

1. Start the container based on the application image

2. Access the /home/euler-copilot-fast-inference directory to get the inference process

3. Specify the large model path and execute the inference process

4. Specify the large model path as the first argument, using -i to specify the input prompt word

5. View model inference results

6. example download: Due to the large size of euler-copilot-fast-inference image, it is recommended to pull it locally separately before starting the container:

     docker pull openeuler/euler-copilot-fast-inference:{Tag}
   

   usage:

   • download qwen1_5-7b-chat-q4_0.gguf

       wget [***]
     

   • start docker，copy weight file into docker，start inference

       # cd source code
       cd euler-copilot-fast-inference
       # start docker
       docker run --name **** -it -d --net=host --privileged=true --entrypoint=bash openeuler/fast-llm:1.0.0-oe2203sp3
       # copy weight file
       docker cp qwen1_5-7b-chat-q4_0.gguf <CONTAINER ID>:/home/euler-copilot-fast-inference
       # start inference
       docker exec -it <name> bash
       cd /home/euler-copilot-fast-inference
       # example
       ./fast-llm qwen1_5-7b-chat-q4_0.gguf -t 0.0 -i "中国最高的山峰是?" -n 64
     

Question and answering

If you have any questions or want to use some special features, please submit an issue or a pull request on openeuler-docker-images.