Bug 1693299

Summary:	virsh nodedev-list --cap net can not list all the network device when modprobe ib_umad
Product:	Red Hat Enterprise Linux 8	Reporter:	Pavel Hrdina <phrdina>
Component:	libvirt	Assignee:	Pavel Hrdina <phrdina>
Status:	CLOSED ERRATA	QA Contact:	yalzhang <yalzhang>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	bhavesh_bharadiya, chhu, dyuan, jdenemar, jishao, kasmith, lhuang, nanda_kishore_chinna, phrdina, rbalakri, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	8.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-4.5.0-25.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1398111	Environment:
Last Closed:	2019-11-05 20:48:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1398111
Bug Blocks:	1574709, 1660905

Description Pavel Hrdina 2019-03-27 13:33:47 UTC

+++ This bug was initially created as a clone of Bug #1398111 +++

Description of problem:
virsh nodedev-list --cap net can not list all the network devices when modprobe ib_umad

Version-Release number of selected component (if applicable):
kernel-3.10.0-514.el7.x86_64
libvirt-2.0.0-10.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.# virsh nodedev-list --cap net
net_eno1_34_40_b5_b1_16_68
net_eno2_34_40_b5_b1_16_6a
net_ens2f0_00_0f_53_0c_7f_00
net_ens2f1_00_0f_53_0c_7f_01

2.
# ll /sys/class/net
total 0
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 eno1 -> ../../devices/pci0000:00/0000:00:01.0/0000:04:00.0/net/eno1
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 eno2 -> ../../devices/pci0000:00/0000:00:01.0/0000:04:00.1/net/eno2
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 enp0s29f0u2 -> ../../devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.0/net/enp0s29f0u2
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 ens2f0 -> ../../devices/pci0000:80/0000:80:01.0/0000:86:00.0/net/ens2f0
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 ens2f1 -> ../../devices/pci0000:80/0000:80:01.0/0000:86:00.1/net/ens2f1
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 virbr0 -> ../../devices/virtual/net/virbr0
lrwxrwxrwx. 1 root root 0 Nov 16 05:35 virbr0-nic -> ../../devices/virtual/net/virbr0-nic


3,# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 34:40:b5:b1:16:68 brd ff:ff:ff:ff:ff:ff
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether 34:40:b5:b1:16:6a brd ff:ff:ff:ff:ff:ff
4: ens2f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether 00:0f:53:0c:7f:00 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
5: ens2f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether 00:0f:53:0c:7f:01 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
6: enp0s29f0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 1000
    link/ether 36:40:b5:b2:96:6b brd ff:ff:ff:ff:ff:ff
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:7d:8d:cc brd ff:ff:ff:ff:ff:ff
8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:7d:8d:cc brd ff:ff:ff:ff:ff:ff



Actual results:
"virsh nodedev-list --cap net" can not list all the network devices as above


Expected results:
"virsh nodedev-list --cap net" can  list all the network devices


Additional info:
With the help of lhuang' debugging, we found that if the the module ib_umad modprobed, it will create some pathes :
/sys/class/infiniband
/sys/class/infiniband_cm
/sys/class/infiniband_srp
/sys/class/infiniband_verbs

It will cause libvirt to the path /sys/devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.0 to find the file "resource", but it can not be found, it will can not 
distinguish this NIC


--- Additional comment from Pavel Hrdina on 2019-03-27 14:31:45 CET ---

Upstream commit:

commit f38ef0fac0582ac0cbb749af9d3f8ba515a6084a
Author: Pavel Hrdina <phrdina>
Date:   Thu Mar 7 17:52:55 2019 +0100

    util: skip RDMA detection for non-PCI network devices

Comment 1 Pavel Hrdina 2019-05-10 08:20:58 UTC

*** Bug 1707757 has been marked as a duplicate of this bug. ***

Comment 2 Nanda Kishore Chinnaram 2019-05-13 13:09:18 UTC

Pavel,
In which build can we expect the fix?

Comment 3 Pavel Hrdina 2019-05-20 08:09:48 UTC

Hi Nanda,

That's hard to tell, this will be fixed in RHEL-8.1.0.

You can watch this BZ and once the "Status" is MODIFIED there will
be libvirt package version in "Fixed in Version".

Comment 6 yalzhang@redhat.com 2019-07-03 09:19:51 UTC

Reproduce on libvirt-4.5.0-24.module+el8.1.0+3205+41ff0a42.x86_64

# rpm -q libvirt
libvirt-4.5.0-24.module+el8.1.0+3205+41ff0a42.x86_64

# lsmod | grep ib_umad
ib_umad                28672  4
ib_core               282624  13 rdma_cm,ib_ipoib,rpcrdma,mlx4_ib,ib_srpt,ib_srp,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,ib_cm

# ll /sys/class/net
total 0
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno1 -> ../../devices/pci0000:00/0000:00:03.1/0000:01:00.0/net/eno1
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno2 -> ../../devices/pci0000:00/0000:00:03.1/0000:01:00.1/net/eno2
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno3 -> ../../devices/pci0000:00/0000:00:03.0/0000:02:00.0/net/eno3
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno4 -> ../../devices/pci0000:00/0000:00:03.0/0000:02:00.1/net/eno4
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 enp4s0f0 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.0/net/enp4s0f0
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 enp4s0f1 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.1/net/enp4s0f1
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 ib0 -> ../../devices/pci0000:00/0000:00:02.2/0000:06:00.0/net/ib0
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 ib1 -> ../../devices/pci0000:00/0000:00:02.2/0000:06:00.0/net/ib1
lrwxrwxrwx. 1 root root 0 Jun 21 19:03 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx. 1 root root 0 Jul  3 16:59 macvtap0 -> ../../devices/virtual/net/macvtap0
lrwxrwxrwx. 1 root root 0 Jul  1 15:07 virbr0 -> ../../devices/virtual/net/virbr0
lrwxrwxrwx. 1 root root 0 Jul  1 15:07 virbr0-nic -> ../../devices/virtual/net/virbr0-nic
lrwxrwxrwx. 1 root root 0 Jul  3 17:01 vnet0 -> ../../devices/virtual/net/vnet0

# virsh nodedev-list --cap net
net_eno1_18_66_da_e6_83_98
net_eno2_18_66_da_e6_83_99
net_eno3_18_66_da_e6_83_9a
net_eno4_18_66_da_e6_83_9b
net_enp4s0f0_a0_36_9f_95_0a_08
net_enp4s0f1_a0_36_9f_95_0a_0a
net_ib0_80_00_02_08_fe_80_00_00_00_00_00_00_e4_1d_2d_03_00_48_23_31
net_ib1_80_00_02_09_fe_80_00_00_00_00_00_00_e4_1d_2d_03_00_48_23_32

The non-PCI network devices like macvtap0, virbr0-nic, vnet0 do not show in nodedev-list outputs.

Test on libvirt-4.5.0-29.x86_64, the result is as expected:

# rpm -q libvirt 
libvirt-4.5.0-29.module+el8.1.0+3567+07fd3313.x86_64

# virsh nodedev-list --cap net
net_eno1_18_66_da_e6_83_98
net_eno2_18_66_da_e6_83_99
net_eno3_18_66_da_e6_83_9a
net_eno4_18_66_da_e6_83_9b
net_enp4s0f0_a0_36_9f_95_0a_08
net_enp4s0f1_a0_36_9f_95_0a_0a
net_ib0_80_00_02_08_fe_80_00_00_00_00_00_00_e4_1d_2d_03_00_48_23_31
net_ib1_80_00_02_09_fe_80_00_00_00_00_00_00_e4_1d_2d_03_00_48_23_32
net_lo_00_00_00_00_00_00
net_macvtap0_52_54_00_76_91_49
net_virbr0_nic_52_54_00_da_3f_87
net_vnet0_fe_54_00_77_92_da

The non-PCI network devices like vnet0, virbr0-nic, lo, macvtap0 can list out in nodedev-list outputs.

Comment 7 Nanda Kishore Chinnaram 2019-07-15 04:53:52 UTC

Pavel,
is this going to make into RHEL-8.0 Z-Stream?

Comment 8 Pavel Hrdina 2019-07-18 11:59:12 UTC

(In reply to Nanda Kishore Chinnaram from comment #7)
> Pavel,
> is this going to make into RHEL-8.0 Z-Stream?

Hi, there was no z-stream fix requested so we don't have any plans to fix
it there.  It will be fixed in RHEL-8.1, if you would have to request
z-stream fix.

Comment 10 errata-xmlrpc 2019-11-05 20:48:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345