Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1770119

Summary: [mlx5_core] testpmd: infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)
Product: Red Hat Enterprise Linux Fast Datapath Reporter: qding
Component: DPDKAssignee: Kevin Traynor <ktraynor>
DPDK sub component: sriov QA Contact: liting <tli>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: medium CC: ctrautma, goetz.waschk, jhsiao, knweiss, mleitner, qding, sscheink
Version: FDP 19.G   
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-01-12 16:52:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
XML for creating guest
none
XML for attaching VF1 to guest
none
XML for attaching VF2 to guest none

Description qding 2019-11-08 08:05:47 UTC 
Description of problem:

[root@localhost ~]# 
 -iot@localhost ~]# testpmd -l 0-8 -n 4 -w 03:00.0 -w 04:00.0 -- --rxq=2 --txq=2 
EAL: Detected 9 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:101a net_mlx5
net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:101a net_mlx5
net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=211456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
[  674.840044] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)
[  680.695218] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)
[  680.697080] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)
[  686.473923] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)
[  692.366890] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)
[  692.368550] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5)

Signal 2 received, preparing to exit...
LATENCY_STATS: failed to remove Rx callback for pid=0, qid=0
LATENCY_STATS: failed to remove Rx callback for pid=0, qid=1
LATENCY_STATS: failed to remove Tx callback for pid=0, qid=0
LATENCY_STATS: failed to remove Tx callback for pid=0, qid=1

Shutting down port 0...
Stopping ports...
Done
Closing ports...
Port 0 is now not stopped
Done

Shutting down port 1...
Stopping ports...
Done
Closing ports...
Done

Bye...

[root@localhost ~]# 

Version-Release number of selected component (if applicable):
[root@localhost ~]# rpm -qa | grep dpdk
dpdk-tools-18.11-4.el7_6.x86_64
dpdk-18.11-4.el7_6.x86_64
[root@localhost ~]# 
[root@localhost ~]# uname -r
3.10.0-1062.el7.x86_64
[root@localhost ~]# 

same version in host

How reproducible: 100%


Steps to Reproduce:
1. create guest
   virsh create g1.xml

2. create VFs on dual ports PFs
   echo 1 > /sys/class/net/p6p1/device/sriov_numvfs 
   echo 1 > /sys/class/net/p6p2/device/sriov_numvfs 

3. attach VF to VM
   virsh attach-device g1 vf1.xml
   virsh attach-device g1 vf2.xml

4. run testpmd in guest

Actual results:


Expected results:


Additional info:

- boot cmdline in host:

[root@dell-per730-04 perf]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-1101.el7.x86_64 root=/dev/mapper/rhel_dell--per730--04-root ro intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=16 ksdevice=bootif crashkernel=auto rd.lvm.lv=rhel_dell-per730-04/root rd.lvm.lv=rhel_dell-per730-04/swap console=ttyS0,115200n81
[root@dell-per730-04 perf]# 

- boot cmdline in guest:

[root@localhost ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-1062.el7.x86_64 root=/dev/mapper/rhel-root ro intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=1 rhgb console=ttyS0,115200 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS0
[root@localhost ~]# 


Same issue with two cards:
[root@dell-per730-04 perf]# lspci -m -s 04:00.0
04:00.0 "Ethernet controller" "Mellanox Technologies" "MT28800 Family [ConnectX-5 Ex]" "Mellanox Technologies" "Device 0002"
[root@dell-per730-04 perf]# 
[root@dell-per730-04 perf]# ethtool -i p6p1
driver: mlx5_core
version: 5.0-0
firmware-version: 16.26.1040 (MT_0000000009)
expansion-rom-version: 
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
[root@dell-per730-04 perf]# 

[root@dell-per730-05 perf]# lspci -m -s 84:00.0
84:00.0 "Ethernet controller" "Mellanox Technologies" "MT27800 Family [ConnectX-5]" "Mellanox Technologies" "Device 0007"
[root@dell-per730-05 perf]# 
[root@dell-per730-05 perf]# ethtool -i p2p1
driver: mlx5_core
version: 5.0-0
firmware-version: 16.26.1040 (MT_0000000012)
expansion-rom-version: 
bus-info: 0000:84:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
[root@dell-per730-05 perf]#
Comment 3 Alaa Hleihel (NVIDIA Mellanox) 2020-01-29 16:08:13 UTC 
Hi qding,

does this still reproduce?
if so, can you please provide a setup with reproduction steps?

Thanks,
Alaa
Comment 4 qding 2020-02-05 07:28:40 UTC 
Created attachment 1657761 [details]
XML for creating guest
Comment 5 qding 2020-02-05 07:29:12 UTC 
Created attachment 1657762 [details]
XML for attaching VF1 to guest
Comment 6 qding 2020-02-05 07:29:36 UTC 
Created attachment 1657766 [details]
XML for attaching VF2 to guest
Comment 7 qding 2020-02-05 07:30:05 UTC 
(In reply to Alaa Hleihel (Mellanox) from comment #3)
> Hi qding,
> 
> does this still reproduce?
> if so, can you please provide a setup with reproduction steps?
> 

1. set bootoption in host as below and reboot system if anything changed

[root@dell-per740-10 ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-1125.el7.x86_64 root=/dev/mapper/rhel_dell--per740--10-root ro crashkernel=auto intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=16 spectre_v2=retpoline rd.lvm.lv=rhel_dell-per740-10/root rd.lvm.lv=rhel_dell-per740-10/swap console=ttyS0,115200n81
[root@dell-per740-10 ~]# 


2. install required packages
   wget http://download-node-02.eng.bos.redhat.com/brewroot/packages/qemu-kvm-rhev/2.12.0/33.el7_7.8/x86_64/qemu-img-rhev-2.12.0-33.el7_7.8.x86_64.rpm
   wget http://download-node-02.eng.bos.redhat.com/brewroot/packages/qemu-kvm-rhev/2.12.0/33.el7_7.8/x86_64/qemu-kvm-common-rhev-2.12.0-33.el7_7.8.x86_64.rpm
   wget http://download-node-02.eng.bos.redhat.com/brewroot/packages/qemu-kvm-rhev/2.12.0/33.el7_7.8/x86_64/qemu-kvm-rhev-2.12.0-33.el7_7.8.x86_64.rpm
   yum -y install libvirt
   yum -y install virt-install
   yum -y install libguestfs-tools
   systemctl restart libvirtd

3. create sriov vfs
   echo 1 > /sys/bus/pci/devices/0000:5e:00.0/sriov_numvfs
   echo 1 > /sys/bus/pci/devices/0000:5e:00.1/sriov_numvfs

4. create VM and attach VFs to guest
   virsh create g1.xml 
   virsh attach-device g1 vf1.xml 
   virsh attach-device g1 vf2.xml 

   * please see the attachment for the files

5. install required packages in guest
   yum -y install http://download-node-02.eng.bos.redhat.com/brewroot/packages/dpdk/18.11.2/1.el7/x86_64/dpdk-18.11.2-1.el7.x86_64.rpm
   yum -y install http://download-node-02.eng.bos.redhat.com/brewroot/packages/dpdk/18.11.2/1.el7/x86_64/dpdk-tools-18.11.2-1.el7.x86_64.rpm
   yum -y install libibverbs

6. set bootoptions in guest as below and reboot guest if anything changed

[root@localhost ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-1118.el7.x86_64 root=/dev/mapper/rhel-root ro rhgb intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=2 console=ttyS0,115200 crashkernel=auto spectre_v2=retpoline rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS0
[root@localhost ~]# history

7. run testpmd
   
   testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -w 0000:03:00.0 -w 0000:04:00.0 -- --burst=64 -i --forward-mode=mac --auto-start --nb-cores=2 --rxq=1 --txq=1 --rxd=2048 --txd=2048 --disable-rss



Thanks
Qijun
Comment 10 Kevin Traynor 2024-01-12 16:52:21 UTC 
This is an old issue and is not present in newer versions. Closing, please reopen or log new Jira if it is seen again. Thanks!