quilt/spark

Quilt: Spark

本文档描述如何在Quilt上运行Apache Spark并在启动时传递任务，具体将运行SparkPI示例在Quilt Spark集群上计算π值。

配置QUILT_PATH

Quilt使用QUILT_PATH环境变量定位包。请参考入门指南了解设置QUILT_PATH的说明。

要获取Quilt附带的规范，请执行quilt get github.com/NetSys/quilt。

SparkPI

示例SparkPI程序将计算π的计算密集型任务分布在计算机集群的多台机器上。

我们的sparkPI.js Quilt.js规范简化了设置运行该Spark任务所需基础设施的流程。

配置SSH认证

Quilt管理的机器使用公钥认证控制SSH访问。要读取Spark任务的结果，我们需要访问Master VM。

如果您想使用githubKey认证，请打开specs/spark/sparkPI.js并适当设置sshKeys机器属性：

javascript
var baseMachine = new Machine({
    ...
    sshKeys: githubKeys("<您的GitHub用户名>"),
    ...
});

有关配置用户提供的公钥及更多Quilt SSH认证配置信息，请参见入门指南。

构建sparkPI.js

使用quilt daemon启动Quilt守护进程。然后在另一个shell中执行quilt run github.com/NetSys/quilt/specs/spark/sparkPI.js以构建此Quilt.js规范。

Quilt将开始在您的云提供商上配置多个VM：5台作为Worker，1台作为Master。

VM启动、Quilt配置网络和Docker容器初始化需要一些时间。以下输出显示Master的公网IP为54.153.115.119：

INFO [Nov 12 09:01:57.191] Successfully booted machines.
...
INFO [Nov 12 09:02:43.819] db.Machine:
	Machine-2{Master, Amazon us-west-1 m4.large, sir-cw7genph, PublicIP=54.153.115.119, PrivateIP=172.31.5.94, Disk=32GB}
	Machine-3{Worker, Amazon us-west-1 m4.large, sir-2i78dmbg, PublicIP=52.53.167.190, PrivateIP=172.31.6.253, Disk=32GB}
...

当控制台输出中Machine描述附带Connected标签时，对应的VM已完全启动并开始与Quilt通信：

INFO [Nov 12 09:03:41.606] db.Machine:
	Machine-2{Master, Amazon us-west-1 m4.large, sir-cw7genph, PublicIP=54.153.115.119, PrivateIP=172.31.5.94, Disk=32GB, Connected}

看到Connected标签后，您可以使用quilt ssh <MACHINE_NUM>命令连接到机器，或手动使用ssh quilt@<PUBLIC_IP>。例如，本例中可使用quilt ssh 2或ssh quilt@54.153.115.119。

检查容器

要列出集群中所有活跃容器，请使用quilt containers。例如：

$ quilt containers
Container-9{run quilt/spark run master, Minion: 172.31.6.253, StitchID: 2, IP: 10.14.11.2, Mac: 02:00:0a:0e:0b:02, Labels: [spark-ms], Env: map[JOB:run-example SparkPi]}
Container-10{run quilt/spark run worker, Minion: 172.31.14.240, StitchID:5, IP:10.205.149.88, Mac:02:00:0a:cd:95:58, Labels:[spark-wk], Env:map[MASTERS:1.spark-ms.q]}
Container-11{run quilt/spark run worker, Minion:172.31.12.226, StitchID:6, IP:10.181.3.12, Mac:02:00:0a:b5:03:0c, Labels:[spark-wk], Env:map[MASTERS:1.spark-ms.q]}
Container-12{run quilt/spark run worker, Minion:172.31.0.12, StitchID:7, IP:10.211.166.114, Mac:02:00:0a:d3:a6:72, Labels:[spark-wk], Env:map[MASTERS:1.spark-ms.q]}

获取π值结果

Master Spark容器启动后，我们可通过日志查看SparkPI任务的结果。

使用从上述容器描述中获取的Spark Master节点的Stitch ID执行quilt logs <Stitch ID>。滚动浏览Spark的信息日志后，您将找到SparkPI的结果：

$ quilt logs 2
# ...
16/06/08 18:49:42 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/06/08 18:49:42 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 0.381 s
16/06/08 18:49:42 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 0.525937 s
Pi is roughly 3.13918
16/06/08 18:49:42 INFO SparkUI: Stopped Spark web UI at http://10.0.254.144:4040
16/06/08 18:49:42 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/06/08 18:49:42 INFO MemoryStore: MemoryStore cleared
16/06/08 18:49:42 INFO BlockManager: BlockManager stopped

注意： Spark集群现已启动并可用。您可通过在Master Spark容器中执行quilt exec -t <MASTER_CONTAINER_ID> spark-shell运行交互式spark-shell。要终止部署，只需执行quilt stop。 ===END===