kylemcc/kube-filebeat

kube-filebeat

!License BSD

kube-filebeat是运行https://github.com/elastic/beats/tree/master/filebeat和https://github.com/kylemcc/kube-gen的Docker容器，通过监听Kubernetes API事件，基于Pod注解生成filebeat配置，实现集群内应用日志的自动收集与Logstash转发。

注意：本项目为实验性质，依赖Docker文件系统层机制，仅支持Docker 1.10.0及以上版本，可能随时失效。

部署方式

需在所有需收集日志的节点运行，推荐通过DaemonSet部署：

yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: "kube-filebeat"
  annotations:
    description: "基于注解的自动日志发送器"
spec:
  template:
    spec:
      containers:
        - name: "kube-filebeat"
          image: "kylemcc/kube-filebeat:latest"
          env:
            - name: LOGSTASH_HOSTS
              value: logstash.default.svc.cluster.local:5044
            - name: KUBERNETES_API_URL
              value: http://10.1.2.3:8080
          volumeMounts:
            - name: docker
              mountPath: /var/lib/docker
          imagePullPolicy: "Always"
      restartPolicy: "Always"
      volumes:
        - name: docker
          hostPath:
            path: /var/lib/docker

配置说明

通过Pod注解定义日志收集规则：

单容器Pod配置

yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kube_filebeat: >
      [
          {
              "log": "/var/log/example-app/output.log",
              "ignore_older": "24h",
              "close_older": "24h",
              "fields": {"app": "example-app","version": "1.2.3"},
              "multiline": {"pattern": "^(([[:alpha:]]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2})|([0-9]{4}-[0-9]{2}-[0-9]{2}))","negate": true,"match": "after"}
          },
          {
              "log": "/var/log/nginx/access.log",
              "exclude_lines": [".*Go-http-client/1\\.1.*"],
              "ignore_older": "24h",
              "close_older": "24h",
              "fields": {"app": "example-app","version": "1.2.3","type": "access_log"}
          }
      ]
  name: example-app
spec:
  containers:
    - image: example-app:1.2.3
      name: example-app

多容器Pod配置

需指定容器名称：

yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kube_filebeat: >
      [
          {"container": "example-app","log": "/var/log/app/logfile",...},
          {"container": "nginx","log": "/var/log/nginx/access.log",...}
      ]
spec:
  containers:
    - image: example-app:1.2.3
      name: example-app
    - image: nginx:latest
      name: nginx