具备 SR-IOV 功能的 NIC 通过引入物理功能(PFs)和虚拟功能(VFs)的概念来工作。
PF 由主机使用,VF 配置通过 PF 应用。每个 VF 可被视为独立的物理 NIC,并分配给一个容器。
要构建插件二进制文件:
# make
构建成功后,插件二进制文件将位于 build/ib-sriov。
build/ib-sriov
IB-SRIOV-CNI 支持 Mellanox ConnectX®-4/ConnectX®-5/ConnectX®-6 适配卡。
使用上游 mstflint 启用 SR-IOV 功能需执行以下步骤:
安装 Mstflint 包。
# yum install -y mstflint
启用 SR-IOV
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
# mstconfig -d 0000:02:00.0 set SRIOV_EN=1 NUM_OF_VFS=8
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: 0000:02:00.0
Configurations: Next Boot New
SRIOV_EN False(0) True(1)
NUM_OF_VFS 0 8
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
重启机器
# reboot
创建 SR-IOV VFs
# echo 4
> /sys/class/net/ib0/device/sriov_numvfs
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改 VF 数量,请先将数量重置为 0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
使用 Mellanox OFED 启用 SR-IOV 功能需执行以下步骤:
1- 在 NIC 固件中启用 SR-IOV。
安装 Mellanox 管理工具(MFT)或 mstflint 是前提条件,MFT 可从此处下载,mstflint 包可在各种发行版中获取,也可从此处下载。
安装 Mellanox 管理工具(MFT)或 mstflint 是前提条件,MFT 可从此处下载,mstflint 包可在各种发行版中获取,也可从此处下载。
使用 Mellanox 固件工具包在固件中启用和配置 SR-IOV
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
在所需 PCI 插槽上定位 HCA 设备
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.
...
启用 SR-IOV
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
...
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
重启机器
# reboot
2- 在 NIC 驱动中启用 SR-IOV。
# ibdev2netdev
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Down)
# echo 4
> /sys/class/net/ib0/device/sriov_numvfs
# ibdev2netdev -v
0000:02:00.0 mlx5_0 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib0 (Up)
0000:02:00.1 mlx5_1 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib1 (Down)
0000:02:00.5 mlx5_2 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib2 (Down)
0000:02:00.6 mlx5_3 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib3 (Down)
0000:02:00.7 mlx5_4 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib4 (Down)
0000:02:00.2 mlx5_5 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib5 (Down)
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改 VF 数量,请先将数量重置为 0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
# cat
> /etc/cni/net.d/10-ib-sriov.conf /sys/class/net/ib0/device/sriov_numvfs
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改VF数量,请先将数量重置为0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
要使用Mellanox OFED启用SR-IOV功能,需要执行以下步骤:
1- 在NIC固件中启用SR-IOV。
[!NOTE] 安装Mellanox管理工具(MFT)或mstflint是前提条件,MFT可从此处下载,mstflint包在各种发行版中可用,也可从https://github.com/Mellanox/mstflint%E4%B8%8B%E8%BD%BD%E3%80%82
使用Mellanox固件工具包在固件中启用和配置SR-IOV
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
在所需PCI插槽上找到HCA设备
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.
...
启用SR-IOV
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
...
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
重启机器
# reboot
2- 在NIC驱动中启用SR-IOV。
# ibdev2netdev
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Down)
# echo 4
> /sys/class/net/ib0/device/sriov_numvfs
# ibdev2netdev -v
0000:02:00.0 mlx5_0 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib0 (Up)
0000:02:00.1 mlx5_1 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib1 (Down)
0000:02:00.5 mlx5_2 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib2 (Down)
0000:02:00.6 mlx5_3 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib3 (Down)
0000:02:00.7 mlx5_4 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib4 (Down)
0000:02:00.2 mlx5_5 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib5 (Down)
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改VF数量,请先将数量重置为0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
name(字符串,必需):网络的名称type(字符串,必需):"ib-sriov"deviceID(字符串,必需):InfiniBand SR-IOV NIC的VF的有效PCI地址。例如:"0000:03:02.3"guid(字符串,可选):VF的InfiniBand Guid。ipam(字典,可选):用于此网络的IPAM配置,不支持dhcp。link_state(字符串,可选):强制VF的链路状态。允许值:auto、enable、disable。rdmaIsolation(布尔值,可选):为RDMA工作负载启用RDMA网络命名空间隔离。有关支持此操作模式的系统要求的更多信息,请参见https://github.com/Mellanox/rdma-cniibKubernetesEnabled(布尔值,可选):强制ib-sriov-cni与https://www.github.com/Mellanox/ib-kubernetes%E9%85%8D%E5%90%88%E5%B7%A5%E4%BD%9C%E3%80%82vfioPciMode(布尔值,可选):为绑定到vfio-pci驱动程序的设备(VF或PF)启用VFIO模式。启用后,CNI会跳过网络接口配置,因为VFIO设备用于直接设备分配(例如,用于kubevirt/VM工作负载)。默认为false。如果未显式设置,将根据设备的驱动程序绑定自动检测模式。[!NOTE] PF直通仅在VFIO模式下受支持。使用PF设备时,必须将其绑定到vfio-pci驱动程序,并且必须启用(或自动检测)
vfioPciMode。不支持将PF的InfiniBand接口移至Pod网络命名空间。
[!NOTE] 如果
rdmaIsolation设置为true,则不应使用https://github.com/Mellanox/rdma-cni%E3%80%82
ib-sriov支持以下https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md#dynamic-plugin-specific-fields-capabilities--runtime-configuration%EF%BC%9A
infinibandGUID(字符串):为网络接口(VF)动态分配InfiniBand GUID。# cat
> /etc/cni/net.d/10-ib-sriov.conf <<EOF
{
"cniVersion": "0.3.1",
"name": "mynet",
"type": "ib-sriov",
"deviceID": "0000:03:02.0",
"link_state": "enable",
"rdmaIsolation": true,
"ibKubernetesEnabled": false,
"ipam": {
"type": "host-local",
"subnet": "10.56.217.0/24",
"rangeStart": "10.56.217.171",
"rangeEnd": "10.56.217.181",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "10.56.217.1"
}
}
EOF
https://github.com/openshift/sriov-network-operator%E7%94%A8%E4%BA%8E%E7%AE%A1%E7%90%86%E8%8A%82%E7%82%B9%E4%B8%8A%E7%9A%84SR-IOV%E6%8E%A5%E5%8F%A3%EF%BC%88%E4%BE%8B%E5%A6%82%E6%9B%B4%E6%94%B9%E8%8A%82%E7%82%B9%E4%B8%8A%E7%9A%84VF%E6%95%B0%E9%87%8F%EF%BC%89%EF%BC%8C%E8%BF%98%E7%94%A8%E4%BA%8E%E6%9B%B4%E6%94%B9%E6%8E%A5%E5%8F%A3%E7%9A%84%E9%93%BE%E8%B7%AF%E7%B1%BB%E5%9E%8B%EF%BC%88%E4%BB%8EETH%E6%94%B9%E4%B8%BAIB%EF%BC%8C%E5%8F%8D%E4%B9%8B%E4%BA%A6%E7%84%B6%EF%BC%89%E3%80%82https://github.com/openshift/sriov-network-operator/blob/master/deploy/crds/sriovnetwork.openshift.io_v1_sriovnetworknodepolicy_cr.yaml#L38%E5%B1%95%E7%A4%BA%E4%BA%86%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8%E8%AF%A5%E6%93%8D%E4%BD%9C%E5%99%A8%E4%B8%BA%E7%BB%99%E5%AE%9A%E7%9A%84PCI%E7%89%A9%E7%90%86%E5%8A%9F%E8%83%BD%E5%9C%B0%E5%9D%80%E6%9B%B4%E6%94%B9%E9%93%BE%E8%B7%AF%E7%B1%BB%E5%9E%8B%E5%92%8CSR-IOV%E5%B1%9E%E6%80%A7%E3%80%82
InfiniBand SR-IOV CNI适用于内核5.6,该内核支持RDMA网络命名空间隔离以及VF端口和节点GUID的获取/设置。
Kubernetes部署中使用InfiniBand作为辅助网络的推荐网络拓扑如下:
探索更多轩辕镜像的使用方法,找到最适合您系统的配置方式
通过 Docker 登录认证访问私有仓库
无需登录使用专属域名
Kubernetes 集群配置 Containerd
K3s 轻量级 Kubernetes 镜像加速
VS Code Dev Containers 配置
Podman 容器引擎配置
HPC 科学计算容器配置
ghcr、Quay、nvcr 等镜像仓库
Harbor Proxy Repository 对接专属域名
Portainer Registries 加速拉取
Nexus3 Docker Proxy 内网缓存
需要其他帮助?请查看我们的 常见问题Docker 镜像访问常见问题解答 或 提交工单
docker search 限制
站内搜不到镜像
离线 save/load
插件要用 plugin install
WSL 拉取慢
安全与 digest
新手拉取配置
镜像合规机制
不支持 push
manifest unknown
no matching manifest(架构)
invalid tar header(解压)
TLS 证书失败
DNS 超时
域名连通性排查
410 Gone 排查
402 与流量用尽
401 认证失败
429 限流
D-Bus 凭证提示
413 与超大单层
来自真实用户的反馈,见证轩辕镜像的优质服务