SeaTunnel是一款下一代超高性能、分布式的海量数据集成工具,它专注于高效处理大规模数据的抽取、转换与加载(ETL)任务,支持多种异构数据源与目标系统的无缝对接,具备优异的横向扩展能力和运行稳定性,适用于各类企业级大数据平台环境,能够为数据整合、流转与分析提供快速、可靠且易维护的技术支撑。
Apache SeaTunnel
SeaTunnel is a very easy-to-use, ultra-high-performance, distributed data integration platform that supports real-time synchronization of massive data. It can synchronize tens of billions of data stably and efficiently every day, and has been used in production by nearly 100 companies.
Set Up With Docker
Set Up With Docker In Local Mode
Zeta Engine
Download
shelldocker pull apache/seatunnel:<version_tag>
How to submit job in local mode
shell# Run fake source to console sink docker run --rm -it apache/seatunnel:<version_tag> ./bin/seatunnel.sh -m local -c config/v2.batch.config.template # Run job with custom config file docker run --rm -it -v /<The-Config-Directory-To-Mount>/:/config apache/seatunnel:<version_tag> ./bin/seatunnel.sh -m local -c /config/fake_to_console.conf # Example # If you config file is in /tmp/job/fake_to_console.conf docker run --rm -it -v /tmp/job/:/config apache/seatunnel:<version_tag> ./bin/seatunnel.sh -m local -c /config/fake_to_console.conf # Set JVM options when running docker run --rm -it -v /tmp/job/:/config apache/seatunnel:<version_tag> ./bin/seatunnel.sh -DJvmOption="-Xms4G -Xmx4G" -m local -c /config/fake_to_console.conf
Set Up With Docker In Cluster Mode
there has 2 ways to create cluster within docker.
Use Docker Directly
create a network
shelldocker network create seatunnel-network
start the nodes
- start master node
shell## start master and export 5801 port docker run -d --name seatunnel_master \ --network seatunnel-network \ --rm \ -p 5801:5801 \ apache/seatunnel \ ./bin/seatunnel-cluster.sh -r master
- get created container ip
shelldocker inspect seatunnel_master
run this command to get the pod ip.
- start worker node
shell# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST` docker run -d --name seatunnel_worker_1 \ --network seatunnel-network \ --rm \ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ apache/seatunnel \ ./bin/seatunnel-cluster.sh -r worker ## start worker2 # you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST` docker run -d --name seatunnel_worker_2 \ --network seatunnel-network \ --rm \ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ apache/seatunnel \ ./bin/seatunnel-cluster.sh -r worker
Scale your Cluster
run this command to start master node.
shell# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST` docker run -d --name seatunnel_master \ --network seatunnel-network \ --rm \ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ apache/seatunnel \ ./bin/seatunnel-cluster.sh -r master
run this command to start worker node.
shell# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST` docker run -d --name seatunnel_worker_1 \ --network seatunnel-network \ --rm \ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ apache/seatunnel \ ./bin/seatunnel-cluster.sh -r worker
Use Docker-compose
docker cluster mode is only support zeta engine.
The docker-compose.yaml file is :
yamlversion: '3.8' services: master: image: apache/seatunnel container_name: seatunnel_master environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4 entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r master " ports: - "5801:5801" networks: seatunnel_network: ipv4_address: 172.16.0.2 worker1: image: apache/seatunnel container_name: seatunnel_worker_1 environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4 entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r worker " depends_on: - master networks: seatunnel_network: ipv4_address: 172.16.0.3 worker2: image: apache/seatunnel container_name: seatunnel_worker_2 environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4 entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r worker " depends_on: - master networks: seatunnel_network: ipv4_address: 172.16.0.4 networks: seatunnel_network: driver: bridge ipam: config: - subnet: 172.16.0.0/24
run docker-compose up -d command to start the cluster.
You can use docker logs -f seatunne_master, docker logs -f seatunnel_worker_1 to check the node log.
And when you call http://localhost:5801/hazelcast/rest/maps/system-monitoring-information, you will see there are 2 nodes as we excepted.
After that, you can use client or restapi to submit job to this cluster.
Scale your Cluster
If you want to increase cluster node, like add a new work node.
yamlversion: '3.8' services: master: image: apache/seatunnel container_name: seatunnel_master environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4 entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r master " ports: - "5801:5801" networks: seatunnel_network: ipv4_address: 172.16.0.2 worker1: image: apache/seatunnel container_name: seatunnel_worker_1 environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4 entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r worker " depends_on: - master networks: seatunnel_network: ipv4_address: 172.16.0.3 worker2: image: apache/seatunnel container_name: seatunnel_worker_2 environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4 entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r worker " depends_on: - master networks: seatunnel_network: ipv4_address: 172.16.0.4 #### ## add new worker node #### worker3: image: apache/seatunnel container_name: seatunnel_worker_3 environment: - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4,172.16.0.5 # add ip to here entrypoint: > /bin/sh -c " /opt/seatunnel/bin/seatunnel-cluster.sh -r worker " depends_on: - master networks: seatunnel_network: ipv4_address: 172.16.0.5 # use a not used ip networks: seatunnel_network: driver: bridge ipam: config: - subnet: 172.16.0.0/24
and run docker-compose up -d command, the new worker node will start, and the current node won't restart.
Job Operation on cluster
use docker as a client
- submit job :
shell# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST` docker run --name seatunnel_client \ --network seatunnel-network \ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ --rm \ apache/seatunnel \ ./bin/seatunnel.sh -c config/v2.batch.config.template
- list job
shell# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST` docker run --name seatunnel_client \ --network seatunnel-network \ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ --rm \ apache/seatunnel \ ./bin/seatunnel.sh -l
more command please refer https://seatunnel.apache.org/docs/seatunnel-engine/user-command
use rest api
please refer https://seatunnel.apache.org/docs/seatunnel-engine/rest-api-v1
更多相关 Docker 镜像与资源
以下是 apache/seatunnel 相关的常用 Docker 镜像,适用于 不同场景 等不同场景:
镜像拉取方式
您可以使用以下命令拉取该镜像。请将 <标签> 替换为具体的标签版本。如需查看所有可用标签版本,请访问 标签列表页面。
轩辕镜像加速拉取命令点我查看更多 seatunnel 镜像标签
DockerHub 原生拉取命令
用户好评
来自真实用户的反馈,见证轩辕镜像的优质服务