如果你使用 DeepSeek、元宝 AI、Copilot、ChatGPT 等 AI 工具,协助配置轩辕镜像、编写 docker pull 命令、修改 Docker Compose 镜像地址、配置镜像加速、排查镜像拉取失败、分析报错日志等问题,请先让 AI 阅读并遵守轩辕镜像的规则文档。
只需在 AI 对话中先发送下面这句话即可:
请先完整阅读并严格遵守以下文档中的全部规则与要求:
https://xuanyuan.cloud/agents.md
在未充分阅读并理解该文档前,不要生成任何命令、配置、修改建议、故障排查方案或技术回答。后续所有输出都必须严格以该文档中的规范为最高优先级执行。查看 agents.md 用法指南与完整示范。国内用户首推 元宝 AI、DeepSeek 的深度思考模式,不推荐豆包 AI;Cursor 等编辑器可在对话 @ 该链接,或加入 User Rules。 若 AI 无法访问外链,可 打开说明文档 复制全文粘贴。文档会随站点更新,复制内容可能过期,建议定期检查。
具备 SR-IOV 功能的 NIC 通过引入物理功能(PFs)和虚拟功能(VFs)的概念来工作。
PF 由主机使用,VF 配置通过 PF 应用。每个 VF 可被视为独立的物理 NIC,并分配给一个容器。
要构建插件二进制文件:
# make
构建成功后,插件二进制文件将位于 build/ib-sriov。
build/ib-sriov
IB-SRIOV-CNI 支持 Mellanox ConnectX®-4/ConnectX®-5/ConnectX®-6 适配卡。
使用上游 mstflint 启用 SR-IOV 功能需执行以下步骤:
安装 Mstflint 包。
# yum install -y mstflint
启用 SR-IOV
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
# mstconfig -d 0000:02:00.0 set SRIOV_EN=1 NUM_OF_VFS=8
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: 0000:02:00.0
Configurations: Next Boot New
SRIOV_EN False(0) True(1)
NUM_OF_VFS 0 8
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
重启机器
# reboot
创建 SR-IOV VFs
# echo 4
> /sys/class/net/ib0/device/sriov_numvfs
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改 VF 数量,请先将数量重置为 0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
使用 Mellanox OFED 启用 SR-IOV 功能需执行以下步骤:
1- 在 NIC 固件中启用 SR-IOV。
安装 Mellanox 管理工具(MFT)或 mstflint 是前提条件,MFT 可从此处下载,mstflint 包可在各种发行版中获取,也可从此处下载。
安装 Mellanox 管理工具(MFT)或 mstflint 是前提条件,MFT 可从此处下载,mstflint 包可在各种发行版中获取,也可从此处下载。
使用 Mellanox 固件工具包在固件中启用和配置 SR-IOV
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
在所需 PCI 插槽上定位 HCA 设备
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.
...
启用 SR-IOV
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
...
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
重启机器
# reboot
2- 在 NIC 驱动中启用 SR-IOV。
# ibdev2netdev
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Down)
# echo 4
> /sys/class/net/ib0/device/sriov_numvfs
# ibdev2netdev -v
0000:02:00.0 mlx5_0 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib0 (Up)
0000:02:00.1 mlx5_1 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib1 (Down)
0000:02:00.5 mlx5_2 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib2 (Down)
0000:02:00.6 mlx5_3 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib3 (Down)
0000:02:00.7 mlx5_4 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib4 (Down)
0000:02:00.2 mlx5_5 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib5 (Down)
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改 VF 数量,请先将数量重置为 0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
# cat
> /etc/cni/net.d/10-ib-sriov.conf /sys/class/net/ib0/device/sriov_numvfs
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改VF数量,请先将数量重置为0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
要使用Mellanox OFED启用SR-IOV功能,需要执行以下步骤:
1- 在NIC固件中启用SR-IOV。
[!NOTE] 安装Mellanox管理工具(MFT)或mstflint是前提条件,MFT可从此处下载,mstflint包在各种发行版中可用,也可从https://github.com/Mellanox/mstflint%E4%B8%8B%E8%BD%BD%E3%80%82
使用Mellanox固件工具包在固件中启用和配置SR-IOV
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
在所需PCI插槽上找到HCA设备
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.
...
启用SR-IOV
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
...
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
重启机器
# reboot
2- 在NIC驱动中启用SR-IOV。
# ibdev2netdev
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Down)
# echo 4
> /sys/class/net/ib0/device/sriov_numvfs
# ibdev2netdev -v
0000:02:00.0 mlx5_0 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib0 (Up)
0000:02:00.1 mlx5_1 (MT4115 - MT1523X04353) CX456A - ConnectX-4 QSFP fw 12.23.1020 port 1 (ACTIVE) ==> ib1 (Down)
0000:02:00.5 mlx5_2 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib2 (Down)
0000:02:00.6 mlx5_3 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib3 (Down)
0000:02:00.7 mlx5_4 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib4 (Down)
0000:02:00.2 mlx5_5 (MT4116 - NA) fw 12.23.1020 port 1 (DOWN ) ==> ib5 (Down)
# lspci | grep Mellanox
02:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
02:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
02:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
# ip link show
...
ib2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband c6:6d:7d:dd:2a:d5 brd ff:ff:ff:ff:ff:ff
ib3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:3e:07:68:da:fb brd ff:ff:ff:ff:ff:ff
ib4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/infiniband 42:68:f2:aa:c2:27 brd ff:ff:ff:ff:ff:ff
ib5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
...
要更改VF数量,请先将数量重置为0,然后设置所需数量
echo 0
> /sys/class/net/ib0/device/sriov_numvfs
echo 8
> /sys/class/net/ib0/device/sriov_numvfs
name(字符串,必需):网络的名称type(字符串,必需):"ib-sriov"deviceID(字符串,必需):InfiniBand SR-IOV NIC的VF的有效PCI地址。例如:"0000:03:02.3"guid(字符串,可选):VF的InfiniBand Guid。ipam(字典,可选):用于此网络的IPAM配置,不支持dhcp。link_state(字符串,可选):强制VF的链路状态。允许值:auto、enable、disable。rdmaIsolation(布尔值,可选):为RDMA工作负载启用RDMA网络命名空间隔离。有关支持此操作模式的系统要求的更多信息,请参见https://github.com/Mellanox/rdma-cniibKubernetesEnabled(布尔值,可选):强制ib-sriov-cni与https://www.github.com/Mellanox/ib-kubernetes%E9%85%8D%E5%90%88%E5%B7%A5%E4%BD%9C%E3%80%82vfioPciMode(布尔值,可选):为绑定到vfio-pci驱动程序的设备(VF或PF)启用VFIO模式。启用后,CNI会跳过网络接口配置,因为VFIO设备用于直接设备分配(例如,用于kubevirt/VM工作负载)。默认为false。如果未显式设置,将根据设备的驱动程序绑定自动检测模式。[!NOTE] PF直通仅在VFIO模式下受支持。使用PF设备时,必须将其绑定到vfio-pci驱动程序,并且必须启用(或自动检测)
vfioPciMode。不支持将PF的InfiniBand接口移至Pod网络命名空间。
[!NOTE] 如果
rdmaIsolation设置为true,则不应使用https://github.com/Mellanox/rdma-cni%E3%80%82
ib-sriov支持以下https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md#dynamic-plugin-specific-fields-capabilities--runtime-configuration%EF%BC%9A
infinibandGUID(字符串):为网络接口(VF)动态分配InfiniBand GUID。# cat
> /etc/cni/net.d/10-ib-sriov.conf <<EOF
{
"cniVersion": "0.3.1",
"name": "mynet",
"type": "ib-sriov",
"deviceID": "0000:03:02.0",
"link_state": "enable",
"rdmaIsolation": true,
"ibKubernetesEnabled": false,
"ipam": {
"type": "host-local",
"subnet": "10.56.217.0/24",
"rangeStart": "10.56.217.171",
"rangeEnd": "10.56.217.181",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "10.56.217.1"
}
}
EOF
https://github.com/openshift/sriov-network-operator%E7%94%A8%E4%BA%8E%E7%AE%A1%E7%90%86%E8%8A%82%E7%82%B9%E4%B8%8A%E7%9A%84SR-IOV%E6%8E%A5%E5%8F%A3%EF%BC%88%E4%BE%8B%E5%A6%82%E6%9B%B4%E6%94%B9%E8%8A%82%E7%82%B9%E4%B8%8A%E7%9A%84VF%E6%95%B0%E9%87%8F%EF%BC%89%EF%BC%8C%E8%BF%98%E7%94%A8%E4%BA%8E%E6%9B%B4%E6%94%B9%E6%8E%A5%E5%8F%A3%E7%9A%84%E9%93%BE%E8%B7%AF%E7%B1%BB%E5%9E%8B%EF%BC%88%E4%BB%8EETH%E6%94%B9%E4%B8%BAIB%EF%BC%8C%E5%8F%8D%E4%B9%8B%E4%BA%A6%E7%84%B6%EF%BC%89%E3%80%82https://github.com/openshift/sriov-network-operator/blob/master/deploy/crds/sriovnetwork.openshift.io_v1_sriovnetworknodepolicy_cr.yaml#L38%E5%B1%95%E7%A4%BA%E4%BA%86%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8%E8%AF%A5%E6%93%8D%E4%BD%9C%E5%99%A8%E4%B8%BA%E7%BB%99%E5%AE%9A%E7%9A%84PCI%E7%89%A9%E7%90%86%E5%8A%9F%E8%83%BD%E5%9C%B0%E5%9D%80%E6%9B%B4%E6%94%B9%E9%93%BE%E8%B7%AF%E7%B1%BB%E5%9E%8B%E5%92%8CSR-IOV%E5%B1%9E%E6%80%A7%E3%80%82
InfiniBand SR-IOV CNI适用于内核5.6,该内核支持RDMA网络命名空间隔离以及VF端口和节点GUID的获取/设置。
Kubernetes部署中使用InfiniBand作为辅助网络的推荐网络拓扑如下:
来自真实用户的反馈,见证轩辕镜像的优质服务