lintoai/linto-platform-nlp-named-entity-recognition

linto-platform-nlp-named-entity-recognition

镜像概述

本镜像用于构建LinTO平台的命名实体识别(NER)NLP服务，基于https://github.com/linto-ai/linto-platform-nlp-core%E5%BC%80%E5%8F%91%EF%BC%8C%E5%8F%AF%E4%B8%8Ehttps://github.com/linto-ai/linto-platform-stack%E4%B8%80%E8%B5%B7%E9%83%A8%E7%BD%B2%EF%BC%8C%E4%B9%9F%E5%8F%AF%E7%8B%AC%E7%AB%8B%E9%83%A8%E7%BD%B2%EF%BC%88%E8%AF%A6%E8%A7%81%E5%BC%80%E5%8F%91%E9%83%A8%E5%88%86%EF%BC%89%E3%80%82

LinTO的NLP服务采用spaCy的核心设计理念：组件与管道。组件（位于components/目录下）与服务解耦，可轻松复用至其他spaCy项目，组件通过组织成管道实现特定NLP任务。本服务支持REST API和Celery任务两种启动方式，并提供GPU/CPU支持。

核心功能与特性

• 双服务模式：支持REST API（http模式）和Celery任务（task模式）两种部署方式
• 多语言支持：默认支持法语（fr）和英语（en），可通过配置扩展
• 计算加速：支持GPU加速（需安装NVIDIA Container Toolkit）和CPU运行模式
• 灵活部署：可独立部署或集成到LinTO平台栈
• spaCy兼容：采用spaCy组件与管道架构，支持模型扩展与定制

使用场景

• 文本分析系统中的实体识别（如识别组织、人物、地点等）
• NLP流水线中的实体提取环节
• 需要多语言实体识别的应用服务
• 微服务架构中的NLP功能模块

使用方法与配置说明

环境准备

1. 下载模型：将模型下载到主机的./assets目录（也可指定其他位置）

   bash
   cd linto-platform-nlp-named-entity-recognition/
   bash scripts/download_models.sh
   

2. 配置环境变量：复制默认环境变量文件并修改

   bash
   cp .envdefault .env
   

环境变量说明

构建镜像

bash
sudo docker build --tag lintoai/linto-platform-nlp-named-entity-recognition:latest .

或使用docker-compose构建：

bash
sudo docker-compose build

运行容器

GPU支持模式

确保已安装https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian%E5%92%8CGPU%E9%A9%B1%E5%8A%A8%EF%BC%9A

bash
sudo docker run --gpus all \
--rm -p 80:80 \
-v $PWD/assets:/app/assets:ro \
--env-file .env \
lintoai/linto-platform-nlp-named-entity-recognition:latest

CPU模式

CPU运行配置

• 移除命令中的--gpus all参数
• 在.env文件中设置USE_GPU=False

或使用docker-compose运行：

bash
sudo docker-compose up

docker-compose CPU配置

• 从docker-compose.yml中移除runtime: nvidia配置
• 在.env文件中设置USE_GPU=False

REST API使用说明

当SERVICE_MODE=http时，访问http://localhost/docs可查看交互式API文档。

API端点：http://localhost/ner/{lang}

支持的语言

请求示例

json
{
  "articles": [
    {
      "text": "Apple Inc. is an American multinational technology company that specializes in consumer electronics, computer software and online services."
    },
    {
      "text": "Apple was founded in 1976 by Steve Jobs, Steve Wozniak and Ronald Wayne to develop and sell Wozniak's Apple I personal computer."
    }
  ]
}

响应示例

json
{
  "ner": [
    {
      "text": "Apple Inc. is an American multinational technology company that specializes in consumer electronics, computer software and online services.",
      "ents": [
        {
          "text": "Apple Inc",
          "label": "ORG",
          "start": 0,
          "end": 9
        },
        {
          "text": "American",
          "label": "MISC",
          "start": 17,
          "end": 25
        }
      ]
    },
    {
      "text": "Apple was founded in 1976 by Steve Jobs, Steve Wozniak and Ronald Wayne to develop and sell Wozniak's Apple I personal computer.",
      "ents": [
        {
          "text": "Apple",
          "label": "ORG",
          "start": 0,
          "end": 5
        },
        {
          "text": "Steve Jobs",
          "label": "PER",
          "start": 29,
          "end": 39
        },
        {
          "text": "Steve Wozniak",
          "label": "PER",
          "start": 41,
          "end": 54
        },
        {
          "text": "Ronald Wayne",
          "label": "PER",
          "start": 59,
          "end": 71
        },
        {
          "text": "Wozniak",
          "label": "PER",
          "start": 92,
          "end": 99
        },
        {
          "text": "Apple I",
          "label": "MISC",
          "start": 102,
          "end": 109
        }
      ]
    }
  ]
}

Celery任务模式测试

1. 启动Redis：在本地安装并运行Redis

   bash
   redis-server --protected-mode no --bind 0.0.0.0 --loglevel debug
   

2. 配置环境变量：确保.env中设置

   SERVICE_MODE=task
   SERVICES_BROKER=redis://172.17.0.1:6379
   

3. 启动容器：使用前述docker run或docker-compose up命令启动服务

4. 测试任务：运行以下Python脚本

   python
   from docker.xuanyuan.run/celery import Celery
   celery = Celery(broker='redis://localhost:6379/0', backend='redis://localhost:6379/1')
   r = celery.send_task(
       'ner_task', 
       (
           'en', 
           [
               "Apple Inc. is an American multinational technology company that specializes in consumer electronics, computer software and online services.",
               "Apple was founded in 1976 by Steve Jobs, Steve Wozniak and Ronald Wayne to develop and sell Wozniak's Apple I personal computer."
           ],
           {"ner": {"top_n": 3}}
       ),
       queue='ner')
   r.get()
   

部署建议

推荐通过https://github.com/linto-ai/linto-platform-stack%E8%BF%9B%E8%A1%8C%E9%83%A8%E7%BD%B2%EF%BC%8C%E4%BB%A5%E8%8E%B7%E5%BE%97%E5%AE%8C%E6%95%B4%E7%9A%84%E5%B9%B3%E5%8F%B0%E5%8A%9F%E8%83%BD%E6%94%AF%E6%8C%81%E3%80%82%E7%8B%AC%E7%AB%8B%E9%83%A8%E7%BD%B2%E6%97%B6%E9%9C%80%E7%A1%AE%E4%BF%9D%E6%B6%88%E6%81%AF%E4%BB%A3%E7%90%86%EF%BC%88%E5%A6%82Redis%EF%BC%89%E5%92%8C%E6%A8%A1%E5%9E%8B%E6%96%87%E4%BB%B6%E6%AD%A3%E7%A1%AE%E9%85%8D%E7%BD%AE%E3%80%82