coco/concept-rw-elasticsearch

Concept Read Writer for Elasticsearch

https://circleci.com/gh/Financial-Times/concept-rw-elasticsearch/tree/master.png?style=shield](https://circleci.com/gh/Financial-Times/concept-rw-elasticsearch/tree/master)https://goreportcard.com/badge/github.com/Financial-Times/concept-rw-elasticsearch](https://goreportcard.com/report/github.com/Financial-Times/concept-rw-elasticsearch) https://coveralls.io/repos/github/Financial-Times/concept-rw-elasticsearch/badge.svg](https://coveralls.io/github/Financial-Times/concept-rw-elasticsearch)

概述和主要用途

该工具用于批量将概念写入Amazon Elasticsearch集群，并提供读取功能。

⚠️ Go的AWS SDK目前https://github.com/aws/aws-sdk-go/issues/710%EF%BC%8C%E4%BD%86%E8%87%AAv1.2.0%E8%B5%B7%E5%B7%B2%E6%9A%B4%E9%9C%B2%E7%AD%BE%E5%90%8D%E5%99%A8%EF%BC%88Signer%EF%BC%89%E3%80%82

访问AES（Amazon Elasticsearch Service）的方法：

• 基于https://github.com/smartystreets/go-aws-auth%E5%88%9B%E5%BB%BA%E4%BC%A0%E8%BE%93%E5%B1%82%EF%BC%88Transport%EF%BC%89%EF%BC%8C%E4%BD%BF%E7%94%A8v4%E7%AD%BE%E5%90%8D%E5%99%A8317%E3%80%82
• 通过上述创建的客户端，使用https://github.com/olivere/elastic%E5%BA%93%E6%89%A7%E8%A1%8C%E4%BB%BB%E4%BD%95ES%E8%AF%B7%E6%B1%82

如果需要先设置Elasticsearch，请参见https://github.com/Financial-Times/concept-rw-elasticsearch/blob/master/mapping_readme.md%E7%9A%84%E8%AF%B4%E6%98%8E%E3%80%82

核心功能和特性

• 批量写入：支持将概念数据批量写入Elasticsearch集群
• AWS签名集成：通过v4签名器安全访问Amazon Elasticsearch Service
• 数据管理端点：提供PUT、批量PUT、GET、DELETE及指标更新等数据操作接口
• 健康检查：内置健康检查端点，监控服务及Elasticsearch集群状态
• 概念类型支持：处理多种概念类型（如组织、品牌、人物等），并特殊处理membership概念以更新关联人物信息

安装

下载源码、依赖并构建二进制文件：

go get github.com/Financial-Times/concept-rw-elasticsearch
cd $GOPATH/src/github.com/Financial-Times/concept-rw-elasticsearch
go build .

测试

单元测试

运行单元测试：

go test -race ./...

单元和集成测试

docker-compose -f docker-compose-tests.yml up -d --build && \
docker logs -f test-runner && \
docker-compose -f docker-compose-tests.yml down -v 

本地运行Elasticsearch

使用Docker本地运行Elasticsearch：

docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false"  docker.elastic.co/elasticsearch/elasticsearch:5.3.3

向Elasticsearch实例写入数据会创建分片。若运行本地独立Elasticsearch实例，可能导致状态变为YELLOW。要将状态改为GREEN，向/_settings发送PUT请求，请求体如下：

json
{
    "index" : {
        "number_of_replicas" : 0
    }
}

Elasticsearch不会输出状态变为GREEN的日志，但应用健康检查在完成此更改后会返回健康状态。

运行方法

./concept-rw-elasticsearch --aws-access-key="{访问密钥}" --aws-secret-access-key="{密钥}"

还可指定Elasticsearch端点、区域及应用运行端口。其他参数：

当前支持的概念类型："genres, topics, sections, subjects, locations, brands, organisations, people, alphaville-series, memberships"（体裁、主题、版块、主题词、地点、品牌、组织、人物、alphaville系列、成员资格）。

数据端点（DATA endpoints）

localhost:8080/{type}/{uuid}

支持的类型：organisations, brands, genres, locations, people, sections, subjects, topics, alphaville-series, memberships（组织、品牌、体裁、地点、人物、版块、主题词、主题、alphaville系列、成员资格）

成员资格（membership）概念是特殊情况，仅处理FT成员资格（organisationUUID为FT的7bcfe07b-0fb1-49ce-a5fa-e51d5c01c3e0，且membershipRoleUUID为专栏作家7ef75a6a-b6bf-4eb7-a1da-03e0acabef1b或记者33ee38a4-c677-4952-a141-2ae14da3aedd）。

成员资格不会作为独立实体写入Elasticsearch，而是修改关联的人物概念。若该人物UUID无记录，服务会在Elasticsearch中创建占位人物对象，仅设置id、lastModified和isFTAuthor字段。

-XPUT localhost:8080/{type}/{uuid}

成功PUT请求返回200；失败返回500服务器错误；JSON格式错误或路径与请求体中UUID不匹配返回400错误请求。

旧概念模型示例：

bash
curl -XPUT -H "Content-Type: application/json" -H "X-Request-Id: 123" localhost:8080/organisations/2384fa7a-d514-3d6a-a0ea-3a711f66d0d8 --data '{"uuid":"2384fa7a-d514-3d6a-a0ea-3a711f66d0d8","type":"PublicCompany","properName":"Apple, Inc.","prefLabel":"Apple, Inc.","legalName":"Apple Inc.","shortName":"Apple","hiddenLabel":"APPLE INC","formerNames":["Apple Computer, Inc."],"aliases":["Apple Inc","Apple Computers","Apple","Apple Canada","Apple Computer","Apple Computer, Inc.","APPLE INC","Apple Incorporated","Apple Computer Inc","Apple Inc.","Apple, Inc."],"industryClassification":"7a01c847-a9bd-33be-b991-c6fbd8871a46","alternativeIdentifiers":{"TME":["TnN0ZWluX09OX0ZvcnR1bmVDb21wYW55X0FBUEw=-T04="],"uuids":["2384fa7a-d514-3d6a-a0ea-3a711f66d0d8","2abff0bd-544d-31c3-899b-fba2f60d53dd"],"factsetIdentifier":"000C7F-E","leiCode":"HWUPKR0MPOU8FGXBT394"}}'

此时仅保存以下字段：uuid（转换为id）、prefLabel、aliases、type及types（由type生成），其他字段忽略。

新概念模型示例：

bash
curl -XPUT -H "Content-Type: application/json" -H "X-Request-Id: 123" localhost:8080/people/08147da5-8110-407c-a51c-a91855e6b071 --data '{
     "prefUUID": "08147da5-8110-407c-a51c-a91855e6b071",
     "prefLabel": "Anna Whitwham",
     "type": "Person",
     "aliases": [
         "Anna Whitwham"
     ],
     "isAuthor": true,
     "sourceRepresentations": [
      {
             "uuid": "08147da5-8110-407c-a51c-a91855e6b071",
             "prefLabel": "Anna Whitwham",
             "authority": "Smartlogic",
             "authorityValue": "9c2bbb54-6b1c-4b11-b005-a31ffe3b9ee7",
             "aliases": [
                 "Anna Whitwham"
             ],
             "descriptionXML": "This is replacement Anna",
             "type": "Person",
             "emailAddress": "anna@ft.com",
             "facebookPage": "https://www.facebook.com/AnnaFT",
             "twitterHandle": "@JSmithFT",
             "_imageURL": "/Anna.jpg"
         },
         {
             "uuid": "a725fc67-db99-30c5-b37e-9ca0b47edf95",
             "prefLabel": "Anna Whitwham",
             "type": "Person",
             "authority": "TME",
             "authorityValue": "YmUwNTk1YWUtMzdhNy00NmQ4LTg4NzYtYzZmYzgzNTAzYmYy-UE4=",
             "lastModifiedEpoch": 1508313355,
             "aliases": [
                 "Anna Whitwham"
             ]
         }
     ]
 }'

-XPUT localhost:8080/bulk/{type}/{uuid}

请求将按批量处理器配置批量执行。若应用成功接收请求，始终返回200；若写入Elasticsearch失败，请求会被记录（请查看应用日志）。

bash
curl -XPUT -H "Content-Type: application/json" -H "X-Request-Id: 123" localhost:8080/bulk/organisations/2384fa7a-d514-3d6a-a0ea-3a711f66d0d8 --data '{"uuid":"2384fa7a-d514-3d6a-a0ea-3a711f66d0d8","type":"PublicCompany","properName":"Apple, Inc.","prefLabel":"Apple, Inc.","legalName":"Apple Inc.","shortName":"Apple","hiddenLabel":"APPLE INC","formerNames":["Apple Computer, Inc."],"aliases":["Apple Inc","Apple Computers","Apple","Apple Canada","Apple Computer","Apple Computer, Inc.","APPLE INC","Apple Incorporated","Apple Computer Inc","Apple Inc.","Apple, Inc."],"industryClassification":"7a01c847-a9bd-33be-b991-c6fbd8871a46","alternativeIdentifiers":{"TME":["TnN0ZWluX09OX0ZvcnR1bmVDb21wYW55X0FBUEw=-T04="],"uuids":["2384fa7a-d514-3d6a-a0ea-3a711f66d0d8","2abff0bd-544d-31c3-899b-fba2f60d53dd"],"factsetIdentifier":"000C7F-E","leiCode":"HWUPKR0MPOU8FGXBT394"}}'

-XGET localhost:8080/{type}/{uuid}

读取已写入的数据。若未找到，返回404。

bash
curl -H "X-Request-Id: 123" localhost:8080/organisations/2384fa7a-d514-3d6a-a0ea-3a711f66d0d8

返回字段包括：Id、ApiUrl、PrefLabel、Types、DirectType、Aliases（若存在）。

-XDELETE localhost:8080/{type}/{uuid}

未向客户端开放，仅用于内部测试。成功删除返回204，未找到返回404。

bash
curl -XDELETE -H "X-Request-Id: 123" localhost:8080/organisations/2384fa7a-d514-3d6a-a0ea-3a711f66d0d8

-XPUT localhost:8080/{type}/{uuid}/metrics

请求体包含JSON格式的概念指标（如{"metrics":{"annotationsCount":1234, "prevWeekAnnotationsCount": 123}}），该端点会增量更新概念的指标数据，覆盖之前的指标，但不改变文档其他部分。

bash
curl -XPUT -H'X-Request-Id: tid_example' http://localhost:8080/organisations/2384fa7a-d514-3d6a-a0ea-3a711f66d0d8/metrics --data '{"metrics":{"annotationsCount":1234, "prevWeekAnnotationsCount": 123}}'

健康检查端点（HEALTH endpoints）

localhost:8080/__health

提供标准FT输出，指示连接状态和集群健康状况。

localhost:8080/__health-details

提供ES集群的详细健康状态，匹配https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html%E7%9A%84%E5%93%8D%E5%BA%94%E3%80%82%E8%8B%A5%E6%9C%8D%E5%8A%A1%E4%B8%8D%E5%8F%AF%E7%94%A8%E6%88%96%E6%97%A0%E6%B3%95%E8%BF%9E%E6%8E%A5Elasticsearch%EF%BC%8C%E8%BF%94%E5%9B%9E503%E3%80%82

localhost:8080/__gtg

应用健康时返回200，不健康时返回503 Service Unavailable。===SHORT_DESC=== 用于批量将概念写入Amazon Elasticsearch集群并提供读取功能的工具，支持AWS签名访问，适用于概念数据的批量导入与管理。 ===FULL_DESC===# Concept Read Writer for Elasticsearch

概述和主要用途

该工具用于批量将概念写入Amazon Elasticsearch集群，并提供读取功能。由于AWS SDK for Go目前不支持Elasticsearch数据平面API，但自v1.2.0起暴露签名器（Signer），因此采用以下方式访问Amazon Elasticsearch服务（AES）：基于https://github.com/smartystreets/go-aws-auth%E5%88%9B%E5%BB%BA%E4%BC%A0%E8%BE%93%E5%B1%82%EF%BC%88%E4%BD%BF%E7%94%A8v4%E7%AD%BE%E5%90%8D%E5%99%A8%EF%BC%89%EF%BC%8C%E5%B9%B6%E9%80%9A%E8%BF%87https://github.com/olivere/elastic%E5%BA%93%E6%89%A7%E8%A1%8CES%E8%AF%B7%E6%B1%82%E3%80%82%E5%A6%82%E9%9C%80%E8%AE%BE%E7%BD%AEElasticsearch%EF%BC%8C%E5%8F%AF%E5%8F%82%E8%80%83https://github.com/Financial-Times/concept-rw-elasticsearch/blob/master/mapping_readme.md%E3%80%82

核心功能和特性

• 批量操作：支持批量写入概念数据到Elasticsearch集群
• AWS集成：通过v4签名器安全访问Amazon Elasticsearch服务
• 多端点支持：提供数据操作端点（PUT、批量PUT、GET、DELETE、指标更新）和健康检查端点
• 概念类型处理：支持多种概念类型（组织、品牌、人物等），特殊处理membership概念以更新关联人物信息
• 健康监控：内置健康检查接口，实时监控服务及Elasticsearch集群状态

安装

下载源码、依赖并构建二进制文件：

bash
go get github.com/Financial-Times/concept-rw-elasticsearch
cd $GOPATH/src/github.com/Financial-Times/concept-rw-elasticsearch
go build .

测试

单元测试

bash
go test -race ./...

单元和集成测试

bash
docker-compose -f docker-compose-tests.yml up -d --build && \
docker logs -f test-runner && \
docker-compose -f docker-compose-tests.yml down -v 

本地运行Elasticsearch

使用Docker启动本地Elasticsearch：

bash
docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:5.3.3

写入数据会创建分片，独立实例可能导致状态变为YELLOW。如需改为GREEN，向/_settings发送PUT请求：

json
{"index":{"number_of_replicas":0}}

运行方法

基础运行命令：

bash
./concept-rw-elasticsearch --aws-access-key="{访问密钥}" --aws-secret-access-key="{密钥}"

主要参数

| 参数 | 说明 | 默认值 | |----------------------|----------------------------------------------------------------