lintoai/linto-platform-nlp-keyphrase-extraction

linto-platform-nlp-keyphrase-extraction

镜像概述

本镜像用于构建LinTO的NLP服务：关键词提取（Keyphrase Extraction），基于https://github.com/linto-ai/linto-platform-nlp-core%E5%BC%80%E5%8F%91%E3%80%82%E5%8F%AF%E4%B8%8Ehttps://github.com/linto-ai/linto-platform-stack%E4%B8%80%E8%B5%B7%E9%83%A8%E7%BD%B2%EF%BC%8C%E4%B9%9F%E5%8F%AF%E7%8B%AC%E7%AB%8B%E9%83%A8%E7%BD%B2%EF%BC%88%E8%AF%A6%E8%A7%81%E4%B8%8B%E6%96%87%E5%BC%80%E5%8F%91%E9%83%A8%E5%88%86%EF%BC%89%E3%80%82

LinTO的NLP服务采用spaCy的核心设计理念：组件与管道。组件（位于components/目录下）与服务解耦，可轻松复用至其他spaCy项目，组件通过管道组织以实现特定NLP任务。

本服务支持两种启动方式：REST API和Celery任务，且均支持GPU与CPU模式。

核心功能与特性

• 多部署模式：支持REST API（HTTP）和Celery任务两种服务模式
• 硬件支持：兼容GPU加速（需安装NVIDIA Container Toolkit）和纯CPU运行
• 多语言支持：默认支持法语（fr）和英语（en），可通过配置扩展
• 组件化设计：基于spaCy的组件-管道架构，组件可独立复用
• 可配置性：支持通过环境变量和API请求参数自定义提取参数

使用场景

• 文本内容分析与摘要生成
• 信息检索系统中的关键词提取
• 文档自动标引与分类
• NLP处理管道中的关键短语提取环节
• 需要批量或异步处理的关键词提取任务（Celery模式）

使用方法

环境准备

1. 安装依赖工具
   确保已安装：

   • Docker及Docker Compose
   • https://git-lfs.github.com/%EF%BC%88%E7%94%A8%E4%BA%8E%E4%B8%8B%E8%BD%BD%E6%A8%A1%E5%9E%8B%E6%96%87%E4%BB%B6%EF%BC%89
   • 若使用GPU模式，需安装NVIDIA Container Toolkit及GPU驱动

2. 下载模型文件
   将模型文件下载至主机的./assets目录（可自定义路径）：

   bash
   cd linto-platform-nlp-keyphrase-extraction/
   bash scripts/download_models.sh
   

3. 配置环境变量
   复制默认环境变量配置并修改：

   bash
   cp .envdefault .env
   

   环境变量	描述	默认值
   APP_LANG	应用支持的语言列表（空格分隔）	fr en
   ASSETS_PATH_ON_HOST	主机上资产文件夹路径	./assets
   ASSETS_PATH_IN_CONTAINER	容器内模型文件挂载路径	/app/assets
   LM_MAP	语言与对应模型的JSON映射字符串	{"fr":"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2","en":"sentence-transformers/all-MiniLM-L6-v2"}
   SERVICE_MODE	服务模式，可选"http"（REST API）或"task"（Celery任务）	"http"
   CONCURRENCY	最大并发请求数	1
   USE_GPU	是否使用GPU，可选"True"或"False"	True
   SERVICE_NAME	微服务名称	kpe
   SERVICES_BROKER	微服务通信的 broker 服务器URL	"redis://localhost:6379"
   BROKER_PASS	broker服务器访问密码	None

构建镜像

使用Docker命令构建：

bash
sudo docker build --tag lintoai/linto-platform-nlp-keyphrase-extraction:latest .

或使用Docker Compose构建：

bash
sudo docker-compose build

运行容器

GPU模式

bash
sudo docker run --gpus all \
--rm -p 80:80 \
-v $PWD/assets:/app/assets:ro \
--env-file .env \
lintoai/linto-platform-nlp-keyphrase-extraction:latest

CPU模式

或使用Docker Compose运行：

bash
sudo docker-compose up

API使用说明

当SERVICE_MODE=http时，服务启动后可通过http://localhost/docs或http://localhost/redoc访问交互式API文档。

支持的语言与模型

API端点

http://localhost/kpe/{lang}，其中{lang}为语言代码（如en或fr）。

请求示例

json
{
  "articles": [
    {
      "text": "Apple Inc. is an American multinational technology company that specializes in consumer electronics, computer software and online services."
    },
    {
      "text": "Unsupervised learning is a type of machine learning in which the algorithm is not provided with any pre-assigned labels or scores for the training data. As a result, unsupervised learning algorithms must first self-discover any naturally occurring patterns in that training data set."
    }
  ]
}

响应示例

json
{
  "kpe": [
    {
      "text": "Apple Inc. is an American multinational technology company that specializes in consumer electronics, computer software and online services.",
      "keyphrases": [
        {
          "text": "apple",
          "score": 0.6539
        },
        {
          "text": "inc",
          "score": 0.3941
        },
        {
          "text": "company",
          "score": 0.2985
        },
        {
          "text": "multinational",
          "score": 0.2635
        },
        {
          "text": "electronics",
          "score": 0.2143
        }
      ]
    },
    {
      "text": "Unsupervised learning is a type of machine learning in which the algorithm is not provided with any pre-assigned labels or scores for the training data. As a result, unsupervised learning algorithms must first self-discover any naturally occurring patterns in that training data set.",
      "keyphrases": [
        {
          "text": "unsupervised",
          "score": 0.6663
        },
        {
          "text": "learning",
          "score": 0.3155
        },
        {
          "text": "algorithms",
          "score": 0.3128
        },
        {
          "text": "algorithm",
          "score": 0.2494
        },
        {
          "text": "patterns",
          "score": 0.2476
        }
      ]
    }
  ]
}

组件配置

本服务基于https://github.com/MaartenGr/KeyBERT%E5%B0%81%E8%A3%85%EF%BC%8C%E6%94%AF%E6%8C%81%E4%BB%A5%E4%B8%8B%E9%85%8D%E7%BD%AE%E5%8F%82%E6%95%B0%EF%BC%9A

运行时配置示例

可在API请求中通过component_cfg参数自定义配置：

json
{
  "articles": [
    {
      "text": "Unsupervised learning is a type of machine learning in which the algorithm is not provided with any pre-assigned labels or scores for the training data."
    }
  ],
  "component_cfg": {
    "kpe": {"keyphrase_ngram_range": [2,2], "top_n": 1}
  }
}

响应：

json
{
  "kpe": [
    {
      "text": "Unsupervised learning is a type of machine learning in which the algorithm is not provided with any pre-assigned labels or scores for the training data.",
      "keyphrases": [
        {
          "text": "unsupervised learning",
          "score": 0.7252
        }
      ]
    }
  ]
}

Celery任务模式测试

环境准备

1. 本地安装Redis并启动：

   bash
   redis-server --protected-mode no --bind 0.0.0.0 --loglevel debug
   

2. 配置.env文件：

   • 设置SERVICE_MODE=task
   • 设置SERVICES_BROKER=redis://172.17.0.1:6379（172.17.0.1为Docker默认网关）

3. 启动容器（使用上述docker run或docker-compose up命令）

测试脚本

在本地运行以下Python脚本调用Celery任务：

python
from celery import Celery
celery = Celery(broker='redis://localhost:6379/0', backend='redis://localhost:6379/1')
r = celery.send_task(
    'kpe_task', 
    (
        'en', 
        [
            "Apple Inc. is an American multinational technology company that specializes in consumer electronics, computer software and online services.",
            "Unsupervised learning is a type of machine learning in which the algorithm is not provided with any pre-assigned labels or scores for the training data."
        ],
        {"kpe": {"top_n": 3}}
    ),
    queue='kpe')
print(r.get())

高级用法

关于最大和相似度（Max Sum Similarity）、最大边际相关性（MMR）等用于多样化提取结果的高级功能，可参考KeyBERT文档和技术文章了解详细原理。