amd64/spark

注意事项

这是https://hub.docker.com/_/spark%E7%9A%84%60amd64%60%E6%9E%B6%E6%9E%84%E6%9E%84%E5%BB%BA%E7%9A%84%22%E6%AF%8F%E6%9E%B6%E6%9E%84%22%E4%BB%93%E5%BA%93 -- 更多信息，请参见官方镜像文档中的https://github.com/docker-library/official-images#architectures-other-than-amd64%E5%92%8C%E5%AE%98%E6%96%B9%E9%95%9C%E5%83%8FFAQ%E4%B8%AD%E7%9A%84https://github.com/docker-library/faq#an-images-source-changed-in-git-now-what%E3%80%82

快速参考

维护者

Apache Spark

获取帮助

Apache Spark™社区

支持的标签及对应的Dockerfile链接

• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java21-python3-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java21-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java21-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java21-python3-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java17-python3-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java17-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java17-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/4bd1dbce94797b5b387b784db6b378069a8b6328/4.0.0/scala2.13-java17-python3-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java17-python3-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java17-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java17-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java17-python3-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java11-python3-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java11-r-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java11-ubuntu/Dockerfile
• https://github.com/apache/spark-docker/blob/2ebf694ad45fee6f4beeeb4204bcdb01d73c988f/3.5.7/scala2.12-java11-python3-r-ubuntu/Dockerfile

快速参考（续）

问题反馈地址

[***]

支持的架构

(https://github.com/docker-library/official-images#architectures-other-than-amd64) https://hub.docker.com/r/amd64/spark/, https://hub.docker.com/r/arm64v8/spark/

发布的镜像工件详情

https://github.com/docker-library/repo-info/blob/master/repos/spark (https://github.com/docker-library/repo-info/commits/master/repos/spark)%EF%BC%88%E9%95%9C%E5%83%8F%E5%85%83%E6%95%B0%E6%8D%AE%E3%80%81%E4%BC%A0%E8%BE%93%E5%A4%A7%E5%B0%8F%E7%AD%89%EF%BC%89

镜像更新

https://github.com/docker-library/official-images/issues?q=label%3Alibrary%2Fspark https://github.com/docker-library/official-images/blob/master/library/spark (https://github.com/docker-library/official-images/commits/master/library/spark)

描述来源

https://github.com/docker-library/docs/tree/master/spark (https://github.com/docker-library/docs/commits/master/spark)

什么是Apache Spark™？

Apache Spark™是一个多语言引擎，用于在单节点机器或集群上执行数据工程、数据科学和机器学习。它提供Scala、Java、Python和R的高级API，以及支持数据分析通用计算图的优化引擎。它还支持丰富的高级工具包括Spark SQL（用于SQL和DataFrames）、pandas API on Spark（用于pandas工作负载）、MLlib（用于机器学习）、GraphX（用于图处理）和Structured Streaming（用于流处理）。

!https://raw.githubusercontent.com/docker-library/docs/a16cd1ae80c04193c029a686d3006c95edb81594/spark/logo.png

在线文档

您可以在项目网页上找到最新的Spark文档，包括编程指南。本README仅包含基本设置说明。

交互式Scala Shell

开始使用Spark的最简单方法是通过Scala shell：

console
docker run -it amd64/spark /opt/spark/bin/spark-shell

尝试以下命令，应返回1,000,000,000：

scala
scala> spark.range(1000 * 1000 * 1000).count()

交互式Python Shell

开始使用PySpark的最简单方法是通过Python shell：

console
docker run -it amd64/spark:python3 /opt/spark/bin/pyspark

运行以下命令，也应返回1,000,000,000：

python
>>> spark.range(1000 * 1000 * 1000).count()

交互式R Shell

开始在Spark上使用R的最简单方法是通过R shell：

console
docker run -it amd64/spark:r /opt/spark/bin/sparkR

Docker部署方案示例

单节点模式

console
docker run -d -p 8080:8080 --name spark-standalone amd64/spark /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master

集群模式（Docker Compose）

yaml
version: '3'
services:
  spark-master:
    image: amd64/spark
    command: bin/spark-class org.apache.spark.deploy.master.Master
    ports:
      - "7077:7077"
      - "8080:8080"
    environment:
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=7077

  spark-worker:
    image: amd64/spark
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
    depends_on:
      - spark-master
    environment:
      - SPARK_WORKER_CORES=2
      - SPARK_WORKER_MEMORY=2g
    deploy:
      replicas: 2

启动集群：docker-compose up -d

在Kubernetes上运行Spark

[***]

配置和环境变量

详见https://github.com/apache/spark-docker/blob/master/OVERVIEW.md#environment-variable

许可证

Apache Spark、Spark、Apache、Apache feather徽标和Apache Spark项目徽标是Apache软件基金会的商标。 根据Apache许可证2.0版授权。 与所有Docker镜像一样，这些镜像可能包含其他受其他许可证约束的软件（如基础发行版的Bash等及主要软件的依赖项）。 部分自动检测的许可证信息可在https://github.com/docker-library/repo-info/tree/master/repos/spark%E4%B8%AD%E6%89%BE%E5%88%B0%E3%80%82 镜像用户有责任确保对镜像的使用符合其中所有软件的相关许可证。