dustynv/transformers

transformers

容器 镜像 运行 构建

HuggingFace https://huggingface.co/docs/transformers/index%E5%BA%93%E9%80%9A%E8%BF%87%E4%BE%BF%E6%8D%B7%E7%9A%84API%E6%94%AF%E6%8C%81%E5%90%84%E7%A7%8DNLP%E5%92%8C%E8%A7%86%E8%A7%89%E6%A8%A1%E5%9E%8B%EF%BC%8C%E8%A2%AB%E8%AE%B8%E5%A4%9A%E5%85%B6%E4%BB%96LLM%E5%8C%85%E6%89%80%E4%BD%BF%E7%94%A8%E3%80%82%E5%9C%A8https://huggingface.co/models%E4%B8%8A%E6%9C%89%E5%A4%A7%E9%87%8F%E4%B8%8E%E5%85%B6%E5%85%BC%E5%AE%B9%E7%9A%84%E6%A8%A1%E5%9E%8B%E3%80%82

[!NOTE]
如果您希望使用Transformer的集成https://huggingface.co/docs/transformers/main_classes/quantization#bitsandbytes-integration%E9%87%8F%E5%8C%96%EF%BC%88%60load_in_8bit/load_in_4bit%60%EF%BC%89%E6%88%96https://huggingface.co/docs/transformers/main_classes/quantization#autogptq-integration%E9%87%8F%E5%8C%96%EF%BC%8C%E8%AF%B7%E8%BF%90%E8%A1%8C%E4%BB%A5%E4%B8%8B%E5%AE%B9%E5%99%A8%EF%BC%8C%E8%BF%99%E4%BA%9B%E5%AE%B9%E5%99%A8%E5%9C%A8Transformers%E5%9F%BA%E7%A1%80%E4%B8%8A%E5%8C%85%E5%90%AB%E4%BA%86%E7%9B%B8%E5%BA%94%E7%9A%84%E5%BA%93%EF%BC%9A

• https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/auto_gptq%EF%BC%88%E4%BE%9D%E8%B5%96%E4%BA%8ETransformers%EF%BC%89
• https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/bitsandbytes%EF%BC%88%E4%BE%9D%E8%B5%96%E4%BA%8ETransformers%EF%BC%89

文本生成基准测试

替换您想要运行的https://huggingface.co/models?pipeline_tag=text-generation&sort=trending%EF%BC%88%E5%BA%94%E8%AF%A5%E6%98%AF%E5%83%8FGPT%E3%80%81Llama%E7%AD%89CausalLM%E6%A8%A1%E5%9E%8B%EF%BC%89

bash
./run.sh $(./autotag transformers) \
   huggingface-benchmark.py --model=gpt2

如果模型仓库是私有的或需要身份验证，请添加--env HUGGINGFACE_TOKEN=<您的访问令牌>

默认情况下，性能测量会生成128个新的输出标记（可以使用--tokens=N设置）

可以使用--prompt='your prompt here'更改提示

精度/量化

使用--precision参数启用量化（选项：fp32 fp16 fp4 int8，默认：fp16）

如果您使用fp4或int8，请运行上面提到的https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/bitsandbytes%E5%AE%B9%E5%99%A8%EF%BC%8C%E4%BB%A5%E4%BE%BF%E5%AE%89%E8%A3%85bitsandbytes%E5%8C%85%E8%BF%9B%E8%A1%8C%E9%87%8F%E5%8C%96%E3%80%82%E9%A2%84%E6%9C%9F%E9%80%9A%E8%BF%87Transformers%E7%9A%844%E4%BD%8D/8%E4%BD%8D%E9%87%8F%E5%8C%96%E6%AF%94FP16%E6%85%A2%EF%BC%88%E4%BD%86%E6%B6%88%E8%80%97%E6%9B%B4%E5%B0%91%E5%86%85%E5%AD%98%EF%BC%89- 更多信息请参见https://huggingface.co/docs/transformers/main_classes/quantization%E3%80%82

其他库如https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/exllama%E3%80%81https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/awq%E5%92%8Chttps://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/auto-gptq%E6%9C%89%E8%87%AA%E5%AE%9A%E4%B9%89CUDA%E5%86%85%E6%A0%B8%E5%92%8C%E6%9B%B4%E9%AB%98%E6%95%88%E7%9A%84%E9%87%8F%E5%8C%96%E6%80%A7%E8%83%BD%E3%80%82

Llama2

• 首先从[***]
• 然后创建HuggingFace账户，并请求访问其中一个Llama2模型，如https://huggingface.co/meta-llama/Llama-2-7b-hf%EF%BC%88%E8%BF%99%E6%A0%B7%E6%82%A8%E5%B0%86%E8%8E%B7%E5%BE%97%E6%89%80%E6%9C%89Llama2%E6%A8%A1%E5%9E%8B%E7%9A%84%E8%AE%BF%E9%97%AE%E6%9D%83%E9%99%90%EF%BC%89
• 从https://huggingface.co/settings/tokens%E8%8E%B7%E5%8F%96%E7%94%A8%E6%88%B7%E8%AE%BF%E9%97%AE%E4%BB%A4%E7%89%8C

bash
./run.sh --env HUGGINGFACE_TOKEN=<您的访问令牌> $(./autotag transformers) \
   huggingface-benchmark.py --model=meta-llama/Llama-2-7b-hf