shirolimit/pod-meta-exporter

pod-meta-exporter

!https://img.shields.io/docker/cloud/build/shirolimit/pod-meta-exporter !https://img.shields.io/docker/cloud/automated/shirolimit/pod-meta-exporter !https://img.shields.io/github/license/shirolimit/pod-meta-exporter

这是一个跟踪节点上的Pod并将Pod元数据导出到文件的工具，旨在解决当前版本Kubernetes Fluent Bit过滤器的限制。

核心功能

• 使用Kubernetes watch API实时跟踪节点上的Pod
• 将Pod元数据导出为<namespace>_<pod_name>.meta格式的文件
• 支持通过命令行参数或环境变量配置节点名称、目标目录和元数据保留周期
• 配合Fluent Bit过滤器从文件系统加载元数据，解决短生命周期Pod元数据获取失败问题

使用场景

适用于Kubernetes集群中需要通过Fluent Bit处理日志，但遇到短生命周期Pod元数据缺失的场景，作为Sidecar容器与Fluent Bit配合使用。

配置说明

构建方式

使用make命令快速构建：

bash
$ make build

运行参数

运行示例：

bash
$ pod-meta-exporter --node-name 'my-node' --destination-dir '/var/kube_meta' --retention-period 5m

与Fluent Bit配合部署

1. 配置Fluent Bit过滤器加载本地元数据：

[FILTER]
    Name                            kubernetes
    Match                           kube.*
    Kube_URL                        https://kubernetes.default.svc:443
    Kube_Tag_Prefix                 kube.var.log.containers.  
    Merge_Log                       On
    Keep_Log                        Off
    Labels                          On
    Annotations                     On
    Kube_meta_preload_cache_dir     /var/kube_meta

2. 添加Sidecar容器到Fluent Bit DaemonSet：

yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fluent-bit
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:latest
          command: 
            - "/fluent-bit/bin/fluent-bit"
          args: 
            - "-c"
            - "/fluent-bit/etc/fluent-bit.conf"
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: k8s-meta
              mountPath: /var/kube_meta
              readOnly: true
        - name: pod-meta-exporter
          image: shirolimit/pod-meta-exporter:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: EXPORTER_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: EXPORTER_RETENTION_PERIOD
              value: "5m"
            - name: EXPORTER_DESTINATION_DIR
              value: "/var/kube_meta"
          volumeMounts:
            - name: k8s-meta
              mountPath: /var/kube_meta
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: k8s-meta
          emptyDir: {}
      serviceAccountName: fluent-bit-service-account