Bug 1398111
| Summary: | virsh nodedev-list --cap net can not list all the network device when modprobe ib_umad | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Jingjing Shao <jishao> | |
| Component: | libvirt | Assignee: | Pavel Hrdina <phrdina> | |
| Status: | CLOSED ERRATA | QA Contact: | yalzhang <yalzhang> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 8.1 | CC: | dyuan, jdenemar, knoel, lhuang, phrdina, xuzhang, yalzhang | |
| Target Milestone: | rc | Keywords: | Upstream | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | libvirt-5.0.0-9.el8 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1693299 (view as bug list) | Environment: | ||
| Last Closed: | 2019-08-07 10:41:09 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1693299 | |||
Upstream commit:
commit f38ef0fac0582ac0cbb749af9d3f8ba515a6084a
Author: Pavel Hrdina <phrdina>
Date: Thu Mar 7 17:52:55 2019 +0100
util: skip RDMA detection for non-PCI network devices
Reproduce the bug on libvirt-2.0.0-10.el7_3.14.x86_64 with below steps: 1. fresh boot os, ib_umad is not loaded, the bug can not be reproduced: # rpm -q libvirt qemu-kvm-rhev libvirt-2.0.0-10.el7_3.14.x86_64 qemu-kvm-rhev-2.6.0-28.el7_3.18.x86_64 # virsh nodedev-list --cap net net_enp0s29u1u1u5_42_f2_e9_32_6d_9f net_enp11s0_40_f2_e9_32_6d_9d net_enp6s0_40_f2_e9_32_6d_9c net_lo_00_00_00_00_00_00 net_virbr0_nic_52_54_00_3a_6d_8e # lsmod | grep ib_umad ==> no outputs 2. load the ib_umad module, the bug can be reproduced after restart libvirtd: # modprobe ib_umad # lsmod | grep ib_umad ib_umad 22093 0 ib_core 255469 12 rdma_cm,ib_cm,iw_cm,rpcrdma,ib_srp,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert # virsh nodedev-list --cap net net_enp0s29u1u1u5_42_f2_e9_32_6d_9f net_enp11s0_40_f2_e9_32_6d_9d net_enp6s0_40_f2_e9_32_6d_9c net_lo_00_00_00_00_00_00 net_virbr0_nic_52_54_00_3a_6d_8e # systemctl restart libvirtd # virsh nodedev-list --cap net net_enp11s0_40_f2_e9_32_6d_9d net_enp6s0_40_f2_e9_32_6d_9c # grep -i enp0s29u1u1u5 /var/log/libvirt/libvirtd.log ... 2019-07-02 09:29:42.419+0000: 2374: debug : udevAddOneDevice:1135 : Discarding device -1 0x7f191c170eb0 /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1.5/2-1.1.5:1.0/net/enp0s29u1u1u5 2019-07-02 09:29:42.419+0000: 2374: debug : udevProcessDeviceListEntry:1157 : Failed to create node device for udev device '/sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1.5/2-1.1.5:1.0/net/enp0s29u1u1u5' ... After load the ib_umad module and restart libvirtd, the device enp0s29u1u1u5 and lo, virbr0-nic all disappeared in the output of nodedev-list. verify this bug on libvirt-5.0.0-11.module+el8.0.1+3459+e357ef2f.x86_64 # rpm -q libvirt libvirt-5.0.0-11.module+el8.0.1+3459+e357ef2f.x86_64 1. Make sure the rdma is enabled by checking the ib_umad is loaded: # lsmod | grep ib_umad ib_umad 28672 4 ib_core 282624 13 rdma_cm,ib_ipoib,rpcrdma,mlx4_ib,ib_srpt,ib_srp,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,ib_cm 2. Start a guest with network and direct type interface, then check the nodedev-list outputs: # ll /sys/class/net total 0 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno1 -> ../../devices/pci0000:00/0000:00:03.1/0000:01:00.0/net/eno1 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno2 -> ../../devices/pci0000:00/0000:00:03.1/0000:01:00.1/net/eno2 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno3 -> ../../devices/pci0000:00/0000:00:03.0/0000:02:00.0/net/eno3 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 eno4 -> ../../devices/pci0000:00/0000:00:03.0/0000:02:00.1/net/eno4 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 enp4s0f0 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.0/net/enp4s0f0 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 enp4s0f1 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.1/net/enp4s0f1 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 ib0 -> ../../devices/pci0000:00/0000:00:02.2/0000:06:00.0/net/ib0 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 ib1 -> ../../devices/pci0000:00/0000:00:02.2/0000:06:00.0/net/ib1 lrwxrwxrwx. 1 root root 0 Jun 21 19:03 lo -> ../../devices/virtual/net/lo lrwxrwxrwx. 1 root root 0 Jul 3 16:17 macvtap0 -> ../../devices/virtual/net/macvtap0 lrwxrwxrwx. 1 root root 0 Jul 1 15:07 virbr0 -> ../../devices/virtual/net/virbr0 lrwxrwxrwx. 1 root root 0 Jul 1 15:07 virbr0-nic -> ../../devices/virtual/net/virbr0-nic lrwxrwxrwx. 1 root root 0 Jul 3 16:07 vnet0 -> ../../devices/virtual/net/vnet0 # virsh nodedev-list --cap net net_eno1_18_66_da_e6_83_98 net_eno2_18_66_da_e6_83_99 net_eno3_18_66_da_e6_83_9a net_eno4_18_66_da_e6_83_9b net_enp4s0f0_a0_36_9f_95_0a_08 net_enp4s0f1_a0_36_9f_95_0a_0a net_ib0_80_00_02_08_fe_80_00_00_00_00_00_00_e4_1d_2d_03_00_48_23_31 net_ib1_80_00_02_09_fe_80_00_00_00_00_00_00_e4_1d_2d_03_00_48_23_32 net_lo_00_00_00_00_00_00 net_macvtap0_52_54_00_76_91_49 net_virbr0_nic_52_54_00_da_3f_87 net_vnet0_fe_54_00_77_92_da The non-PCI network devices like virbr0-nic, vnet0, macvtap0 list out, the result is as expected. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2395 |
Description of problem: virsh nodedev-list --cap net can not list all the network devices when modprobe ib_umad Version-Release number of selected component (if applicable): kernel-3.10.0-514.el7.x86_64 libvirt-2.0.0-10.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.# virsh nodedev-list --cap net net_eno1_34_40_b5_b1_16_68 net_eno2_34_40_b5_b1_16_6a net_ens2f0_00_0f_53_0c_7f_00 net_ens2f1_00_0f_53_0c_7f_01 2. # ll /sys/class/net total 0 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 eno1 -> ../../devices/pci0000:00/0000:00:01.0/0000:04:00.0/net/eno1 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 eno2 -> ../../devices/pci0000:00/0000:00:01.0/0000:04:00.1/net/eno2 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 enp0s29f0u2 -> ../../devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.0/net/enp0s29f0u2 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 ens2f0 -> ../../devices/pci0000:80/0000:80:01.0/0000:86:00.0/net/ens2f0 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 ens2f1 -> ../../devices/pci0000:80/0000:80:01.0/0000:86:00.1/net/ens2f1 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 lo -> ../../devices/virtual/net/lo lrwxrwxrwx. 1 root root 0 Nov 16 05:35 virbr0 -> ../../devices/virtual/net/virbr0 lrwxrwxrwx. 1 root root 0 Nov 16 05:35 virbr0-nic -> ../../devices/virtual/net/virbr0-nic 3,# ip l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 34:40:b5:b1:16:68 brd ff:ff:ff:ff:ff:ff 3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000 link/ether 34:40:b5:b1:16:6a brd ff:ff:ff:ff:ff:ff 4: ens2f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000 link/ether 00:0f:53:0c:7f:00 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto 5: ens2f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000 link/ether 00:0f:53:0c:7f:01 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto 6: enp0s29f0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 1000 link/ether 36:40:b5:b2:96:6b brd ff:ff:ff:ff:ff:ff 7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000 link/ether 52:54:00:7d:8d:cc brd ff:ff:ff:ff:ff:ff 8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 1000 link/ether 52:54:00:7d:8d:cc brd ff:ff:ff:ff:ff:ff Actual results: "virsh nodedev-list --cap net" can not list all the network devices as above Expected results: "virsh nodedev-list --cap net" can list all the network devices Additional info: With the help of lhuang' debugging, we found that if the the module ib_umad modprobed, it will create some pathes : /sys/class/infiniband /sys/class/infiniband_cm /sys/class/infiniband_srp /sys/class/infiniband_verbs It will cause libvirt to the path /sys/devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.0 to find the file "resource", but it can not be found, it will can not distinguish this NIC