bzon/prometheus-msteams

概述

一个轻量级Go Web服务器，可接收来自Prometheus Alert Manager的POST告警消息，并通过传入的webhook URL将其发送到Microsoft Teams频道。

快速开始

安装

选项1：使用Docker运行

bash
docker run -d -p 2000:2000 \
    --name="promteams" \
    -e TEAMS_INCOMING_WEBHOOK_URL="https://outlook.office.com/webhook/xxx" \
    -e PROMTEAMS_DEBUG="true" \
    docker.io/bzon/prometheus-msteams:latest

配置Prometheus Alert Manager

默认情况下，prometheus-msteams会创建一个请求URI处理器/alertmanager。

yaml
route:
  group_by: ['alertname']
  group_interval: 30s
  repeat_interval: 30s
  group_wait: 30s
  receiver: 'prometheus-msteams'

receivers:
- name: 'prometheus-msteams'
  webhook_configs: # https://prometheus.io/docs/alerting/configuration/#webhook_config 
  - send_resolved: true
    url: 'http://localhost:2000/alertmanager' # prometheus-msteams代理地址

若尚未运行Prometheus并希望尝试该工具，可参考https://github.com/stefanprodan%E7%9A%84https://github.com/stefanprodan/dockprom%EF%BC%8C%E5%BF%AB%E9%80%9F%E5%9C%A8%E5%8D%95%E5%8F%B0%E6%9C%BA%E5%99%A8%E4%B8%8A%E5%AE%89%E8%A3%85%E6%9C%AC%E5%9C%B0Prometheus%E7%8E%AF%E5%A2%83%E3%80%82

模拟Prometheus告警发送至Teams频道

创建prom-alert.json文件，内容如下：

json
{
    "version": "4",
    "groupKey": "{}:{alertname=\"high_memory_load\"}",
    "status": "firing",
    "receiver": "teams_proxy",
    "groupLabels": {
        "alertname": "high_memory_load"
    },
    "commonLabels": {
        "alertname": "high_memory_load",
        "monitor": "master",
        "severity": "warning"
    },
    "commonAnnotations": {
        "summary": "Server High Memory usage"
    },
    "externalURL": "http://docker.for.mac.host.internal:9093",
    "alerts": [
        {
            "labels": {
                "alertname": "high_memory_load",
                "instance": "10.80.40.11:9100",
                "job": "docker_nodes",
                "monitor": "master",
                "severity": "warning"
            },
            "annotations": {
                "description": "10.80.40.11 reported high memory usage with 23.28%.",
                "summary": "Server High Memory usage"
            },
            "startsAt": "2018-03-07T06:33:21.873077559-05:00",
            "endsAt": "0001-01-01T00:00:00Z"
        }
    ]
}

执行以下命令发送模拟告警：

bash
curl -X POST -d @prom-alert.json http://localhost:2000/alertmanager

Teams频道应会收到告警消息。

发送告警至多个Teams频道

创建配置文件

创建prometheus-msteams配置文件：

yaml
connectors:
- high_priority_channel: "https://outlook.office.com/webhook/xxxx/aaa/bbb"
- low_priority_channel: "https://outlook.office.com/webhook/xxxx/aaa/ccc"

注意：high_priority_channel和low_priority_channel为示例处理器名称。

运行Docker容器时，需挂载配置文件并设置CONFIG_FILE环境变量：

bash
docker run -d -p 2000:2000 \
    --name="promteams" \
    -v /tmp/config.yml:/tmp/config.yml \
    -e CONFIG_FILE="/tmp/config.yml" \
    -e PROMTEAMS_DEBUG="true" \
    docker.io/bzon/prometheus-msteams:latest

若以二进制运行，使用--config参数：

bash
PROMTEAMS_DEBUG="true" prometheus-msteams server \
	-l localhost \
	-p 2000 \
	--config /tmp/config.yml

这将创建/high_priority_channel和/low_priority_channel两个请求URI处理器。

配置Prometheus Alert Manager

结合上述配置文件，Alert Manager配置示例如下：

yaml
route:
  group_by: ['alertname']
  group_interval: 30s
  repeat_interval: 30s
  group_wait: 30s
  receiver: 'low_priority_receiver'  # 默认/ fallback请求处理器
  routes:
   - match:
     severity: critical
     receiver: high_priority_receiver
   - match:
     severity: warning
     receiver: low_priority_receiver

receivers:
- name: 'high_priority_receiver'
  webhook_configs:
    - send_resolved: true
      url: 'http://localhost:2000/high_priority_channel' # 请求处理器1
- name: 'low_priority_receiver'
  webhook_configs:
    - send_resolved: true
      url: 'http://localhost:2000/low_priority_channel' # 请求处理器2

调试

为便于调试，设置环境变量PROMTEAMS_DEBUG=true，可查看从Prometheus接收的JSON体及发送至Microsoft Teams的JSON体：

json
2018/03/19 11:21:23 Request received from Prometheus Alert Manager
{
   "version": "4",
   "groupKey": "{}:{alertname=\"high_memory_load\"}",
   "status": "firing",
   "receiver": "teams_proxy",
   "groupLabels": {
     "alertname": "high_memory_load"
   },
   "commonLabels": {
     "alertname": "high_memory_load",
     "monitor": "master",
     "severity": "warning"
   },
   "commonAnnotations": {
     "summary": "Server High Memory usage"
   },
   "externalURL": "[***]",
   "alerts": [
     {
       "labels": {
         "alertname": "high_memory_load",
         "instance": "10.80.40.11:9100",
         "job": "docker_nodes",
         "monitor": "master",
         "severity": "warning"
       },
       "annotations": {
         "description": "10.80.40.11 reported high memory usage with 23.28%.",
         "summary": "Server High Memory usage"
       },
       "startsAt": "2018-03-07T06:33:21.873077559-05:00",
       "endsAt": "0001-01-01T00:00:00Z"
     }
   ]
 }
2018/03/19 11:21:23 Created a card for Microsoft Teams
{
   "@type": "MessageCard",
   "@context": "[***]",
   "themeColor": "8C1A1A",
   "summary": "Server High Memory usage",
   "title": "Prometheus Alert (firing)",
   "sections": 
     {
       "activityTitle": "[10.80.40.11 reported high memory usage with 23.28%.",
       "facts": [
         {
           "name": "description",
           "value": "10.80.40.11 reported high memory usage with 23.28%."
         },
         {
           "name": "summary",
           "value": "Server High Memory usage"
         },
         {
           "name": "job",
           "value": "docker_nodes"
         },
         {
           "name": "monitor",
           "value": "master"
         },
         {
           "name": "severity",
           "value": "warning"
         },
         {
           "name": "alertname",
           "value": "high_memory_load"
         },
         {
           "name": "instance",
           "value": "10.80.40.11:9100"
         }
       ],
       "markdown": false
     }
   ]
 }