mpioperator/tensorflow-benchmarks

MPI Operator

https://github.com/kubeflow/mpi-operator/workflows/build/badge.svg](https://github.com/kubeflow/mpi-operator/actions?query=event%3Apush+branch%3Amaster) https://goreportcard.com/badge/github.com/kubeflow/mpi-operator](https://goreportcard.com/report/github.com/kubeflow/mpi-operator)

MPI Operator能够在Kubernetes上轻松运行allreduce风格的分布式训练。有关MPI Operator的介绍及其行业应用，请查看这篇博客文章。

镜像概述和主要用途

MPI Operator是Kubeflow生态的一部分，旨在简化Kubernetes集群上分布式训练任务的管理。它通过自定义资源（CRD）MPIJob提供声明式API，允许用户定义和部署多节点MPI训练作业，支持TensorFlow、PyTorch等主流深度学习框架的分布式训练。

核心功能和特性

• 声明式API：通过MPIJob自定义资源定义分布式训练任务，简化配置和管理
• 自动资源调度：自动处理Worker节点和Launcher节点的调度与协调
• 灵活的部署选项：支持自定义容器镜像、资源限制（如GPU）、环境变量等
• 作业监控：提供作业状态跟踪和日志查看功能
• 指标暴露：内置Prometheus指标，用于监控作业创建、成功和失败情况

安装

基本部署

使用默认设置部署operator：

shell
git clone https://github.com/kubeflow/mpi-operator
cd mpi-operator
kubectl create -f deploy/v1alpha2/mpi-operator.yaml

通过Kubeflow部署

或者，按照快速入门指南部署Kubeflow。Kubeflow 0.2.0版本引入了MPI支持的alpha版本，需使用高于0.2.0的Kubeflow版本。

验证CRD安装

检查MPI Job自定义资源是否已安装：

shell
kubectl get crd

输出应包含mpijobs.kubeflow.org：

NAME                                       AGE
...
mpijobs.kubeflow.org                       4d
...

使用Kustomize安装

如果未安装，可使用https://github.com/kubernetes-sigs/kustomize%E6%B7%BB%E5%8A%A0%EF%BC%9A

bash
git clone https://github.com/kubeflow/mpi-operator
cd mpi-operator/manifests
kustomize build overlays/kubeflow | kubectl apply -f -

注意：Kubernetes v1.14及以上版本中，kustomize已成为kubectl的子命令，可使用：

bash
kubectl kustomize base | kubectl apply -f -

创建MPI Job

定义MPIJob配置文件

通过定义MPIJob配置文件创建MPI作业。查看https://github.com/kubeflow/mpi-operator/blob/master/examples/v1alpha2/tensorflow-benchmarks.yaml%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6%EF%BC%8C%E8%AF%A5%E6%96%87%E4%BB%B6%E7%94%A8%E4%BA%8E%E5%90%AF%E5%8A%A8%E5%A4%9A%E8%8A%82%E7%82%B9TensorFlow%E5%9F%BA%E5%87%86%E8%AE%AD%E7%BB%83%E4%BD%9C%E4%B8%9A%E3%80%82%E5%8F%AF%E6%A0%B9%E6%8D%AE%E9%9C%80%E6%B1%82%E4%BF%AE%E6%94%B9%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6%E3%80%82

查看示例配置：

shell
cat examples/v1alpha2/tensorflow-benchmarks.yaml

部署MPIJob

部署MPIJob资源以启动训练：

shell
kubectl create -f examples/v1alpha2/tensorflow-benchmarks.yaml

监控MPI Job

查看Pod状态

创建MPIJob资源后，可以看到与指定GPU数量匹配的Pod：

shell
kubectl get pods

查看作业状态

通过状态部分监控作业状态。以下是作业成功完成时的示例输出：

shell
kubectl get -o yaml mpijobs tensorflow-benchmarks

yaml
apiVersion: kubeflow.org/v1alpha2
kind: MPIJob
metadata:
  creationTimestamp: "2019-07-09T22:15:51Z"
  generation: 1
  name: tensorflow-benchmarks
  namespace: default
  resourceVersion: "5645868"
  selfLink: /apis/kubeflow.org/v1alpha2/namespaces/default/mpijobs/tensorflow-benchmarks
  uid: 1c5b470f-a297-11e9-964d-88d7f67c6e6d
spec:
  cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - command:
            - mpirun
            - --allow-run-as-root
            - -np
            - "2"
            - -bind-to
            - none
            - -map-by
            - slot
            - -x
            - NCCL_DEBUG=INFO
            - -x
            - LD_LIBRARY_PATH
            - -x
            - PATH
            - -mca
            - pml
            - ob1
            - -mca
            - btl
            - ^openib
            - python
            - scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
            - --model=resnet101
            - --batch_size=64
            - --variable_update=horovod
            image: mpioperator/tensorflow-benchmarks:latest
            name: tensorflow-benchmarks
    Worker:
      replicas: 1
      template:
        spec:
          containers:
          - image: mpioperator/tensorflow-benchmarks:latest
            name: tensorflow-benchmarks
            resources:
              limits:
                nvidia.com/gpu: 2
  slotsPerWorker: 2
status:
  completionTime: "2019-07-09T22:17:06Z"
  conditions:
  - lastTransitionTime: "2019-07-09T22:15:51Z"
    lastUpdateTime: "2019-07-09T22:15:51Z"
    message: MPIJob default/tensorflow-benchmarks is created.
    reason: MPIJobCreated
    status: "True"
    type: Created
  - lastTransitionTime: "2019-07-09T22:15:54Z"
    lastUpdateTime: "2019-07-09T22:15:54Z"
    message: MPIJob default/tensorflow-benchmarks is running.
    reason: MPIJobRunning
    status: "False"
    type: Running
  - lastTransitionTime: "2019-07-09T22:17:06Z"
    lastUpdateTime: "2019-07-09T22:17:06Z"
    message: MPIJob default/tensorflow-benchmarks successfully completed.
    reason: MPIJobSucceeded
    status: "True"
    type: Succeeded
  replicaStatuses:
    Launcher:
      succeeded: 1
    Worker: {}
  startTime: "2019-07-09T22:15:51Z"

查看训练日志

训练通常运行100步，在GPU集群上需要几分钟。作业启动后，通过launcher pod查看日志：

shell
PODNAME=$(kubectl get pods -l mpi_job_name=tensorflow-benchmarks,mpi_role_type=launcher -o name)
kubectl logs -f ${PODNAME}

示例日志输出：

TensorFlow:  1.14
Model:       resnet101
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  128 global
             64 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['horovod/gpu:0', 'horovod/gpu:1']
NUMA bind:   False
Data format: NCHW
Optimizer:   sgd
Variables:   horovod

...

40	images/sec: 154.4 +/- 0.7 (jitter = 4.0)	8.280
40	images/sec: 154.4 +/- 0.7 (jitter = 4.1)	8.482
50	images/sec: 154.8 +/- 0.6 (jitter = 4.0)	8.397
50	images/sec: 154.8 +/- 0.6 (jitter = 4.2)	8.450
60	images/sec: 154.5 +/- 0.5 (jitter = 4.1)	8.321
60	images/sec: 154.5 +/- 0.5 (jitter = 4.4)	8.349
70	images/sec: 154.5 +/- 0.5 (jitter = 4.0)	8.433
70	images/sec: 154.5 +/- 0.5 (jitter = 4.4)	8.430
80	images/sec: 154.8 +/- 0.4 (jitter = 3.6)	8.199
80	images/sec: 154.8 +/- 0.4 (jitter = 3.8)	8.404
90	images/sec: 154.6 +/- 0.4 (jitter = 3.7)	8.418
90	images/sec: 154.6 +/- 0.4 (jitter = 3.6)	8.459
100	images/sec: 154.2 +/- 0.4 (jitter = 4.0)	8.372
100	images/sec: 154.2 +/- 0.4 (jitter = 4.0)	8.542
----------------------------------------------------------------
total images/sec: 308.27

暴露的指标

指标关联

使用https://github.com/kubernetes/kube-state-metrics%EF%BC%8C%E5%8F%AF%E4%BB%A5%E9%80%9A%E8%BF%87%E6%A0%87%E7%AD%BE%E5%85%B3%E8%81%94%E6%8C%87%E6%A0%87%E3%80%82%E4%BE%8B%E5%A6%82%EF%BC%9A

kube_pod_info * on(pod,namespace) group_left label_replace(mpi_operator_job_infos, "pod", "$0", "launcher", ".*")

Docker镜像

Docker镜像会自动构建并推送到https://hub.docker.com/u/mpioperator%E3%80%82%E5%8F%AF%E4%BD%BF%E7%94%A8%E4%BB%A5%E4%B8%8BDockerfile%E8%87%AA%E8%A1%8C%E6%9E%84%E5%BB%BA%E9%95%9C%E5%83%8F%EF%BC%9A

• https://github.com/kubeflow/mpi-operator/blob/master/Dockerfile
• https://github.com/kubeflow/mpi-operator/blob/master/cmd/kubectl-delivery/Dockerfile

使用场景和适用范围

MPI Operator适用于需要在Kubernetes集群上运行分布式训练的场景，特别是：

• 深度学习模型的多节点训练（如ResNet、BERT等）
• 使用MPI/OpenMPI进行通信的分布式计算任务
• 需要利用GPU资源进行高性能计算的场景
• 希望通过Kubernetes管理和扩展分布式训练工作流的团队