dparrott/kafka-exporter

kafka_exporter

![Build Status]([] ![License]([]

Prometheus的Kafka指标导出器。如需Kafka的其他指标，请查看https://github.com/prometheus/jmx_exporter%E3%80%82

目录

• 兼容性
• 依赖项
• 下载
• 编译
  • 构建二进制文件
  • 构建Docker镜像
• 运行
  • 运行二进制文件
  • 运行Docker镜像
• 配置标志
  • 注意事项
• 指标
  • Brokers
  • Topics
  • 消费者组
  • 消费者延迟
• Grafana仪表板
• 贡献

• ---

• 许可证

兼容性

支持Apache Kafka 0.10.1.0版本及更高版本。

依赖项

• Prometheus
• https://shopify.github.io/sarama
• Golang

下载

可从https://github.com/davidmparrott/kafka_exporter/releases%E9%A1%B5%E9%9D%A2%E4%B8%8B%E8%BD%BD%E4%BA%8C%E8%BF%9B%E5%88%B6%E6%96%87%E4%BB%B6%E3%80%82

编译

构建二进制文件

shell
make

构建Docker镜像

shell
make docker

Docker Hub镜像

shell
docker pull dparrott/kafka-exporter:latest

可直接使用此镜像，无需自行构建 https://hub.docker.com/r/dparrott/kafka-exporter

运行

运行二进制文件

shell
kafka_exporter --kafka.server=kafka:9092 [--kafka.server=another-server ...]

运行Docker镜像

shell
docker run -ti --rm -p 9308:9308 dparrott/kafka-exporter --kafka.server=kafka:9092 [--kafka.server=another-server ...]

配置标志

此镜像可通过不同标志进行配置

注意事项

布尔值由https://github.com/alecthomas/kingpin/blob/master/README.md#boolean-values%E7%AE%A1%E7%90%86%EF%BC%8C%E6%AF%8F%E4%B8%AA%E5%B8%83%E5%B0%94%E6%A0%87%E5%BF%97%E9%83%BD%E6%9C%89%E4%B8%80%E4%B8%AA%E5%90%A6%E5%AE%9A%E5%BD%A2%E5%BC%8F%EF%BC%9A%60--和--no-`。

例如：

若需禁用sasl.handshake，可添加标志--no-sasl.handshake

指标

关于暴露的Prometheus指标说明。

有关底层指标的详细信息，请参见Apache Kafka。

Brokers

指标详情

指标输出示例

txt
# HELP kafka_brokers Number of Brokers in the Kafka Cluster.
# TYPE kafka_brokers gauge
kafka_brokers 3

Topics

指标详情

指标输出示例

txt
# HELP kafka_topic_partitions Number of partitions for this Topic
# TYPE kafka_topic_partitions gauge
kafka_topic_partitions{topic="__consumer_offsets"} 50

# HELP kafka_topic_partition_current_offset Current Offset of a Broker at Topic/Partition
# TYPE kafka_topic_partition_current_offset gauge
kafka_topic_partition_current_offset{partition="0",topic="__consumer_offsets"} 0

# HELP kafka_topic_partition_oldest_offset Oldest Offset of a Broker at Topic/Partition
# TYPE kafka_topic_partition_oldest_offset gauge
kafka_topic_partition_oldest_offset{partition="0",topic="__consumer_offsets"} 0

# HELP kafka_topic_partition_in_sync_replica Number of In-Sync Replicas for this Topic/Partition
# TYPE kafka_topic_partition_in_sync_replica gauge
kafka_topic_partition_in_sync_replica{partition="0",topic="__consumer_offsets"} 3

# HELP kafka_topic_partition_leader Leader Broker ID of this Topic/Partition
# TYPE kafka_topic_partition_leader gauge
kafka_topic_partition_leader{partition="0",topic="__consumer_offsets"} 0

# HELP kafka_topic_partition_leader_is_preferred 1 if Topic/Partition is using the Preferred Broker
# TYPE kafka_topic_partition_leader_is_preferred gauge
kafka_topic_partition_leader_is_preferred{partition="0",topic="__consumer_offsets"} 1

# HELP kafka_topic_partition_replicas Number of Replicas for this Topic/Partition
# TYPE kafka_topic_partition_replicas gauge
kafka_topic_partition_replicas{partition="0",topic="__consumer_offsets"} 3

# HELP kafka_topic_partition_under_replicated_partition 1 if Topic/Partition is under Replicated
# TYPE kafka_topic_partition_under_replicated_partition gauge
kafka_topic_partition_under_replicated_partition{partition="0",topic="__consumer_offsets"} 0

消费者组

指标详情

指标输出示例

txt
# HELP kafka_consumergroup_current_offset Current Offset of a ConsumerGroup at Topic/Partition
# TYPE kafka_consumergroup_current_offset gauge
kafka_consumergroup_current_offset{consumergroup="KMOffsetCache-kafka-manager-3806276532-ml44w",partition="0",topic="__consumer_offsets"} -1

# HELP kafka_consumergroup_lag Current Approximate Lag of a ConsumerGroup at Topic/Partition
# TYPE kafka_consumergroup_lag gauge
kafka_consumergroup_lag{consumergroup="KMOffsetCache-kafka-manager-3806276532-ml44w",partition="0",topic="__consumer_offsets"} 1

消费者延迟

指标详情

指标输出示例

# HELP kafka_consumer_lag_extrapolation Indicates that a consumer group lag estimation used extrapolation
# TYPE kafka_consumer_lag_extrapolation counter
kafka_consumer_lag_extrapolation{consumergroup="perf-consumer-74084",partition="0",topic="test"} 1
   
# HELP kafka_consumer_lag_interpolation Indicates that a consumer group lag estimation used interpolation
# TYPE kafka_consumer_lag_interpolation counter
kafka_consumer_lag_interpolation{consumergroup="perf-consumer-74084",partition="0",topic="test"} 1
   
# HELP kafka_consumer_lag_millis Current approximation of consumer lag for a ConsumerGroup at Topic/Partition
# TYPE kafka_consumer_lag_millis gauge
kafka_consumer_lag_millis{consumergroup="perf-consumer-74084",partition="0",topic="test"} 3.4457231197552e+10

Grafana仪表板

Grafana仪表板ID：7589，名称：Kafka Exporter Overview。

有关仪表板的详细信息，请参见Kafka Exporter Overview。

延迟估计

消费者组、主题和分区的延迟估计技术借鉴自https://github.com/lightbend/kafka-lag-exporter%E3%80%82

导出器启动后，开始采样要生成的下一个偏移量。插值表由这些样本构建，每个受监控消费者组的当前偏移量与表中的值进行比较。如果表中包含消费者组当前偏移量的上下界，则使用插值法；如果仅包含上界，则使用外推法。

在可配置的间隔prune.interval（默认30秒）内，会执行插值表的修剪操作。任何不再被broker列出的消费者组或主题将被移除。每个分区的偏移量数量会被修剪至max.offsets（默认1000），最旧的偏移量会被优先移除。

插值表的修剪在单独的线程上进行，通过插值表周围的锁确保线程安全。

贡献

如需向上游项目贡献，请提交https://github.com/danielqsj/kafka_exporter/pulls%E3%80%82

如需向此分支贡献，请在此提交https://github.com/davidmparrott/kafka_exporter/pulls%E3%80%82

***

如需向本项目的原始开发者***，请使用以下***链接

许可证

代码基于https://github.com/danielqsj/kafka_exporter/blob/master/LICENSE%E8%AE%B8%E5%8F%AF%E3%80%82