dailyhotel/elb-to-elasticsearch

elb-to-elasticsearch镜像说明

镜像概述

本镜像基于Fluentd构建，旨在将AWS ELB（弹性负载均衡器）/ALB（应用负载均衡器）的访问日志高效传输至Elasticsearch，支持直接传输或通过Kafka作为日志流缓冲的架构，满足云环境下日志集中化管理与分析需求。

核心功能

1. 从AWS S3桶读取ELB/ALB访问日志
2. 直接将日志发送至Elasticsearch进行存储与检索
3. 支持通过Kafka作为中间缓冲层，实现日志流的解耦与削峰
4. 提供Kubernetes部署配置示例，便于容器化环境快速集成

使用场景

• AWS云环境中ELB/ALB访问日志的集中存储与分析
• 需要通过Kafka缓冲日志流的高并发业务场景
• Kubernetes集群内的日志收集与转发需求

配置说明

镜像通过环境变量配置关键参数，主要包括：

• AWS相关：AWS_ACCESS_KEY_ID（访问密钥ID）、AWS_SECRET_ACCESS_KEY（秘密访问密钥）、AWS_REGION（区域）、AWS_S3_BUCKETNAME（日志存储桶名）、AWS_S3_PREFIX（桶内日志前缀）
• Elasticsearch相关：ES_HOSTNAME（主机地址）、ES_USERNAME（用户名）、ES_PASSWORD（密码）、ES_PORT（端口）、ES_SCHEME（通信协议，如https）
• Kafka相关：KAFKA_BROKERS（Broker列表）、KAFKA_ZK（Zookeeper地址）、KAFKA_TOPIC（目标主题）、KAFKA_CONSUMER_GROUP（消费者组）

部署示例

直接传输至Elasticsearch（Kubernetes）

yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elb-logshipper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elb-logshipper
    spec:
      containers:
      - image: dailyhotel/elb-to-elasticsearch:latest
        imagePullPolicy: Always
        name: elb-logshipper
        env:
          - name: AWS_ACCESS_KEY_ID
            value: MY_KEY
          - name: AWS_SECRET_ACCESS_KEY
            value: MY_SECRET
          - name: AWS_REGION
            value: ap-northeast-1
          - name: AWS_S3_BUCKETNAME
            value: my-access-logs
          - name: AWS_S3_PREFIX
            value: dev
          - name: ES_HOSTNAME
            value: my.es.io
          - name: ES_USERNAME
            value: fluentd
          - name: ES_PASSWORD
            value: MY_PASSWORD
          - name: ES_PORT
            value: "9243"
          - name: ES_SCHEME
            value: https

通过Kafka缓冲（Kubernetes）

1. S3到Kafka转发

yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elb-to-kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elb-to-kafka
        role: elb-accesslogs
    spec:
      containers:
      - image: dailyhotel/elb-to-elasticsearch:s3-to-kafka
        imagePullPolicy: Always
        name: elb-to-kafka
        env:
          - name: AWS_ACCESS_KEY_ID
            value: MY_KEY
          - name: AWS_SECRET_ACCESS_KEY
            value: MY_SECRET
          - name: AWS_REGION
            value: ap-northeast-1
          - name: AWS_S3_BUCKETNAME
            value: my-access-logs
          - name: AWS_S3_PREFIX
            value: dev
          - name: KAFKA_BROKERS
            value: kafka-0.broker:9092,kafka-1.broker:9092,kafka-2.broker:9092
          - name: KAFKA_ZK
            value: zookeeper:2181
          - name: KAFKA_TOPIC
            value: elb-accesslogs

2. Kafka到Elasticsearch转发

yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elb-to-es
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elb-to-es
        role: elb-accesslogs
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      containers:
      - image: dailyhotel/elb-to-elasticsearch:kafka-to-es
        imagePullPolicy: Always
        name: elb-to-es
        env:
          - name: KAFKA_BROKERS
            value: kafka-0.broker:9092,kafka-1.broker:9092,kafka-2.broker:9092
          - name: KAFKA_ZK
            value: zookeeper:2181
          - name: KAFKA_TOPIC
            value: elb-accesslogs
          - name: KAFKA_CONSUMER_GROUP
            value: elb-to-es-group
          - name: ES_HOSTNAME
            value: my.es.io
          - name: ES_USERNAME
            value: fluentd
          - name: ES_PASSWORD
            value: MY_PASSWORD
          - name: ES_PORT
            value: "9243"
          - name: ES_SCHEME
            value: https
          - name: RETRY_LIMIT
            value: "1"
          - name: NUM_THREADS
            value: "2"

参考链接

• https://github.com/shinsaka/fluent-plugin-elb-log
• http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html
• https://github.com/coreos/alb-ingress-controller/issues/90