neowu/kafka

neowu/kafka 镜像文档

概述

neowu/kafka是用于内部项目的Kafka镜像，默认配置参数定义在server.properties中。用户可根据实际需求（如集群配置、分区数量、数据保留策略等）覆盖默认参数，以满足不同场景下的消息队列需求。

核心功能与特性

• 灵活配置：支持通过KAFKA_ARGS环境变量覆盖server.properties中的默认参数，如advertised.listeners、num.partitions、log.retention等
• 环境适配：兼容Docker Compose和Kubernetes部署环境，可与ZooKeeper无缝集成
• 资源可控：支持通过环境变量配置JVM参数（如KAFKA_HEAP_OPTS）和JMX监控（KAFKA_JMX_OPTS）

使用场景

适用于内部项目中的消息队列、实时数据流处理、日志收集与分发等场景，尤其适合需要自定义Kafka配置以满足特定业务需求的项目。

使用方法

Docker Compose部署示例

以下是使用Docker Compose部署Kafka及依赖的ZooKeeper的示例配置：

yaml
version: "3"
services:
  zookeeper:
    image: zookeeper:3.5.9
    ports:
      - 2181
    environment:
      - JMXDISABLE=true
      - ZOO_DATA_DIR=/data
      - ZOO_DATA_LOG_DIR=/datalog
      - ZOO_ADMINSERVER_ENABLED=false
  kafka:
    image: neowu/kafka:2.8.0
    ports:
      - 9092:9092
    environment:
      # 通过KAFKA_ARGS覆盖默认配置，示例设置 advertised.listeners 和分区数量
      - KAFKA_ARGS=--override advertised.listeners=PLAINTEXT://localhost:9092 --override num.partitions=3
    depends_on:
      - zookeeper

Kubernetes部署示例

以下是在Kubernetes中部署Kafka及ZooKeeper的完整配置，包含StatefulSet和Service定义：

ZooKeeper部署

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
  namespace: dev
spec:
  serviceName: zookeeper
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  updateStrategy:
    type: OnDelete
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      nodeSelector:
        agentpool: app
      containers:
        - name: zookeeper
          image: zookeeper:3.5.9
          env:
            - name: JMXDISABLE
              value: "true"
            - name: SERVER_JVMFLAGS
              value: "-Dzookeeper.jmx.log4j.disable=true"            
            - name: ZOO_DATA_DIR
              value: "/data"
            - name: ZOO_DATA_LOG_DIR
              value: "/datalog"
            - name: ZOO_ADMINSERVER_ENABLED
              value: "false"
            - name: ZOO_AUTOPURGE_PURGEINTERVAL
              value: "24"
          volumeMounts:
            - name: data
              mountPath: /data
            - name: datalog
              mountPath: /datalog
          resources:
            limits:
              memory: 256Mi
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: datalog
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  namespace: dev
spec:
  clusterIP: None
  ports:
    - port: 2181
  selector:
    app: zookeeper

Kafka部署

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: dev
spec:
  serviceName: kafka
  replicas: 1
  selector:
    matchLabels:
      app: kafka
  updateStrategy:
    type: OnDelete
  template:
    metadata:
      labels:
        app: kafka
    spec:
      nodeSelector:
        agentpool: app
      containers:
        - name: kafka
          env:
            # 如需监控，可配置JMX参数
            - name: KAFKA_JMX_OPTS
              value: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099"
            # 配置JVM堆内存
            - name: KAFKA_HEAP_OPTS
              value: "-Xms1G -Xmx1G"
            # 通过KAFKA_ARGS覆盖Kafka配置，示例设置ZooKeeper连接、日志保留策略和分区数量
            - name: KAFKA_ARGS
              value: "--override zookeeper.connect=zookeeper-0.zookeeper:2181 --override log.retention.bytes=1000000000 --override log.retention.hours=168 --override num.partitions=16"
          image: neowu/kafka:2.8.0
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: kafka
  namespace: dev
spec:
  clusterIP: None
  ports:
    - port: 9092
  selector:
    app: kafka

配置说明

• KAFKA_ARGS：用于覆盖Kafka默认配置的参数，格式为--override <key>=<value>，可指定多个参数（如示例中的advertised.listeners、num.partitions等）
• KAFKA_HEAP_OPTS：设置JVM堆内存大小，如-Xms1G -Xmx1G
• KAFKA_JMX_OPTS：配置JMX监控参数，用于集成监控工具（如Prometheus、Grafana）