sbloodys/hadoop

Apache Hadoop Docker镜像文档

镜像概述

本镜像为Apache Hadoop的Docker化实现，支持amd64和arm64两种架构，可快速部署完整的Hadoop集群环境。通过Docker Compose可便捷配置和启动Hadoop核心组件，适用于开发、测试及小型数据处理场景。

核心功能和特性

• 多架构支持：同时提供amd64和arm64架构镜像，适配不同硬件环境
• 完整组件集成：包含Hadoop核心组件（NameNode、DataNode、ResourceManager、NodeManager）
• 灵活配置：通过环境变量自定义Hadoop配置文件（core-site.xml、hdfs-site.xml等）
• 健康检查：内置服务健康检查机制，确保集群组件正常运行
• 持久化支持：可配置数据持久化存储，保障数据不丢失

使用场景

• Hadoop集群开发与测试环境搭建
• 分布式计算框架学习与实验
• 小型数据处理任务的分布式部署
• 教学环境中的Hadoop集群演示

使用方法

前提条件

• 已安装Docker和Docker Compose
• 网络环境允许容器间通信

部署步骤

1. 创建配置文件

创建docker-compose-hadoop.yaml文件，内容如下：

yaml
version: "3"

services:
  namenode:
    image: sbloodys/hadoop:3.3.6
    hostname: namenode
    container_name: namenode
    command: [ "hdfs", "namenode" ]
    ports:
      - 19870:9870
    env_file:
      - docker-compose.config
    environment:
      ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "1"
    tty: true
    stdin_open: true
    restart: always
    healthcheck:
      test: [ "CMD", "curl", "http://namenode:9870" ]
      interval: 5s
      timeout: 5s
      retries: 120
  datanode:
    image: sbloodys/hadoop:3.3.6
    hostname: datanode
    container_name: datanode
    command: [ "hdfs", "datanode" ]
    env_file:
      - docker-compose.config
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "1"
    tty: true
    stdin_open: true
    restart: always
    healthcheck:
      test: [ "CMD", "curl", "http://datanode:9864" ]
      interval: 5s
      timeout: 5s
      retries: 120
    depends_on:
      namenode:
        condition: service_healthy
  resourcemanager:
    image: sbloodys/hadoop:3.3.6
    hostname: resourcemanager
    container_name: resourcemanager
    command: [ "yarn", "resourcemanager" ]
    ports:
      - 18088:8088
    env_file:
      - docker-compose.config
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "1"
    tty: true
    stdin_open: true
    restart: always
    healthcheck:
      test: [ "CMD", "curl", "http://resourcemanager:8088" ]
      interval: 5s
      timeout: 5s
      retries: 120
  nodemanager:
    image: sbloodys/hadoop:3.3.6
    hostname: nodemanager
    container_name: nodemanager
    command: [ "yarn", "nodemanager" ]
    env_file:
      - docker-compose.config
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "1"
    tty: true
    stdin_open: true
    restart: always
    depends_on:
      resourcemanager:
        condition: service_healthy
    healthcheck:
      test: [ "CMD", "curl", "http://nodemanager:8042" ]
      interval: 5s
      timeout: 5s
      retries: 120

创建docker-compose.config配置文件，内容如下：

bash
HADOOP_HOME=/opt/hadoop
CORE-SITE.XML_fs.default.name=hdfs://namenode
CORE-SITE.XML_fs.defaultFS=hdfs://namenode
CORE-SITE.XML_hadoop.http.staticuser.user=hadoop
HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:8020
HDFS-SITE.XML_dfs.replication=1
MAPRED-SITE.XML_mapreduce.framework.name=yarn
MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.map.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager
YARN-SITE.XML_yarn.nodemanager.pmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.delete.debug-delay-sec=600
YARN-SITE.XML_yarn.nodemanager.vmem-check-enabled=false
YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-applications=10000
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-am-resource-percent=0.1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.queues=default
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.user-limit-factor=1
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.maximum-capacity=100
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.state=RUNNING
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_submit_applications=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_administer_queue=*
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.node-locality-delay=40
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings=
CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings-override.enable=false

2. 启动集群

在配置文件所在目录执行以下命令启动Hadoop集群：

bash
docker-compose -f docker-compose-hadoop.yaml up -d

3. 访问Web界面

• HDFS管理界面：http://localhost:***
• YARN资源管理界面：http://localhost:***

配置说明

主要环境变量说明

服务说明

• namenode：HDFS主节点，负责元数据管理，暴露9870端口用于Web管理
• datanode：HDFS数据节点，负责数据存储，依赖namenode健康状态
• resourcemanager：YARN资源管理器，负责资源分配，暴露8088端口用于Web管理
• nodemanager：YARN节点管理器，负责单节点资源管理，依赖resourcemanager健康状态

参考链接

• 镜像仓库：https://github.com/SbloodyS/hadoop