Bug 2229859

Summary: run virsh net-list --name command hung in the latest rhel9.3
Product: Red Hat Enterprise Linux 9 Reporter: liting <tli>
Component: libvirtAssignee: Virtualization Maintenance <virt-maint>
libvirt sub component: General QA Contact: Virtualization Bugs <virt-bugs>
Status: CLOSED DUPLICATE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: hhan, pkrempa, virt-maint
Version: 9.3   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-08 08:35:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liting 2023-08-08 03:37:25 UTC
Description of problem:

Version-Release number of selected component (if applicable):
5.14.0-350.el9.x86_64

[root@dell-per740-57 perf]# rpm -qa|grep qemu
qemu-img-8.0.0-9.el9.x86_64
libvirt-daemon-driver-qemu-9.3.0-2.el9.x86_64
ipxe-roms-qemu-20200823-9.git4bd064de.el9.noarch
qemu-kvm-common-8.0.0-9.el9.x86_64
qemu-kvm-core-8.0.0-9.el9.x86_64
qemu-kvm-device-display-virtio-gpu-8.0.0-9.el9.x86_64
qemu-kvm-device-display-virtio-gpu-pci-8.0.0-9.el9.x86_64
qemu-pr-helper-8.0.0-9.el9.x86_64
qemu-kvm-tools-8.0.0-9.el9.x86_64
qemu-kvm-device-usb-redirect-8.0.0-9.el9.x86_64
qemu-kvm-docs-8.0.0-9.el9.x86_64
qemu-kvm-device-usb-host-8.0.0-9.el9.x86_64
qemu-kvm-device-display-virtio-vga-8.0.0-9.el9.x86_64
qemu-kvm-block-rbd-8.0.0-9.el9.x86_64
qemu-kvm-block-blkio-8.0.0-9.el9.x86_64
qemu-kvm-ui-opengl-8.0.0-9.el9.x86_64
qemu-kvm-ui-egl-headless-8.0.0-9.el9.x86_64
qemu-kvm-audio-pa-8.0.0-9.el9.x86_64
qemu-kvm-8.0.0-9.el9.x86_64

[root@dell-per740-57 perf]# rpm -qa|grep libvirt
libvirt-libs-9.3.0-2.el9.x86_64
libvirt-client-9.3.0-2.el9.x86_64
libvirt-daemon-common-9.3.0-2.el9.x86_64
libvirt-daemon-driver-secret-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-core-9.3.0-2.el9.x86_64
libvirt-daemon-log-9.3.0-2.el9.x86_64
libvirt-daemon-driver-qemu-9.3.0-2.el9.x86_64
libvirt-daemon-driver-network-9.3.0-2.el9.x86_64
libvirt-daemon-config-network-9.3.0-2.el9.x86_64
libvirt-daemon-lock-9.3.0-2.el9.x86_64
libvirt-daemon-driver-nwfilter-9.3.0-2.el9.x86_64
libvirt-daemon-config-nwfilter-9.3.0-2.el9.x86_64
libvirt-daemon-plugin-lockd-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-iscsi-9.3.0-2.el9.x86_64
libvirt-daemon-driver-nodedev-9.3.0-2.el9.x86_64
libvirt-daemon-proxy-9.3.0-2.el9.x86_64
libvirt-daemon-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-scsi-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-mpath-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-logical-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-disk-9.3.0-2.el9.x86_64
libvirt-daemon-driver-interface-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-rbd-9.3.0-2.el9.x86_64
libvirt-daemon-driver-storage-9.3.0-2.el9.x86_64
libvirt-9.3.0-2.el9.x86_64
python3-libvirt-9.3.0-1.el9.x86_64

How reproducible:


Steps to Reproduce:
Run ovs dpdk pvp perf case
1.Build ovs dpdk pvp topo as follows
[root@dell-per740-57 perf]# ovs-vsctl show
30b3f65d-6366-4b8e-84af-b142b11cf901
    Bridge ovsbr0
        datapath_type: netdev
        Port dpdk0
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:3b:00.0", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"}
        Port vhost0
            Interface vhost0
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser/vhost0"}
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port dpdk1
            Interface dpdk1
                type: dpdk
                options: {dpdk-devargs="0000:3b:00.1", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"}
        Port vhost1
            Interface vhost1
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser/vhost1"}
    ovs_version: "3.1.2"

2. start the guest
yum install -y libvirt
yum -y install qemu-kvm
systemctl enable libvirtd
systemctl restart libvirtd
virsh net-list --name
virsh list --all

Actual results:
Run "virsh net-list --name", it hung.
Run "virsh list --all", it also hung.

[root@dell-per740-57 perf]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; preset: disabled)
     Active: active (running) since Mon 2023-08-07 22:18:46 EDT; 1h 16min ago
TriggeredBy: ● libvirtd.socket
             ● libvirtd-ro.socket
             ● libvirtd-admin.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
    Process: 15106 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
   Main PID: 15106 (code=exited, status=0/SUCCESS)
      Tasks: 2 (limit: 32768)
     Memory: 35.1M
        CPU: 589ms
     CGroup: /system.slice/libvirtd.service
             ├─2283 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libex>
             └─2284 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libex>

Aug 07 22:18:46 dell-per740-57.rhts.eng.pek2.redhat.com systemd[1]: Starting Virtualization daemon...
Aug 07 22:18:46 dell-per740-57.rhts.eng.pek2.redhat.com systemd[1]: Started Virtualization daemon.
Aug 07 22:18:46 dell-per740-57.rhts.eng.pek2.redhat.com dnsmasq[2283]: read /etc/hosts - 2 addresses
Aug 07 22:18:46 dell-per740-57.rhts.eng.pek2.redhat.com dnsmasq[2283]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addr>
Aug 07 22:18:46 dell-per740-57.rhts.eng.pek2.redhat.com dnsmasq-dhcp[2283]: read /var/lib/libvirt/dnsmasq/default.hostsfile

[root@dell-per740-57 perf]# systemctl status virtqemud.socket
○ virtqemud.socket
     Loaded: masked (Reason: Unit virtqemud.socket is masked.)
     Active: inactive (dead)
[root@dell-per740-57 perf]# systemctl status admin.socket
Unit admin.socket could not be found.
[root@dell-per740-57 perf]# systemctl status virtnetworkd.socket
○ virtnetworkd.socket
     Loaded: masked (Reason: Unit virtnetworkd.socket is masked.)
     Active: inactive (dead)

Expected results:
Run "virsh net-list --name", it work well.
Run "virsh list --all", it work well.

Additional info:
It also exist on 5.14.0-348.el9.x86_64, it has no issue on 5.14.0-339.el9.x86_64
job:
https://beaker.engineering.redhat.com/jobs/8160472

Comment 2 Han Han 2023-08-08 05:15:30 UTC
It looks like https://bugzilla.redhat.com/show_bug.cgi?id=2225667 . Please update to systemd-252-17.el9 and above. Then reboot the machine and check it again.

Comment 3 liting 2023-08-08 07:08:56 UTC
(In reply to Han Han from comment #2)
> It looks like https://bugzilla.redhat.com/show_bug.cgi?id=2225667 . Please
> update to systemd-252-17.el9 and above. Then reboot the machine and check it
> again.

I remove the systemd-libs-252-16.el9.x86_64 and install systemd-libs-252-17.el9.x86_64 failed. Do you know how to install it successfully? Thanks
[root@dell-per740-55 ~]# rpm -qa|grep systemd
systemd-rpm-macros-252-16.el9.noarch
systemd-libs-252-16.el9.x86_64
python3-systemd-234-18.el9.x86_64
systemd-pam-252-16.el9.x86_64
systemd-udev-252-16.el9.x86_64
rpm-plugin-systemd-inhibit-4.16.1.3-25.el9.x86_64
systemd-container-252-16.el9.x86_64
[root@dell-per740-55 ~]# rpm -e systemd-pam-252-16.el9.x86_64
warning: Unable to get systemd shutdown inhibition lock: Sender is not authorized to send message

[root@dell-per740-55 ~]# rpm -e systemd-libs-252-16.el9.x86_64 --nodeps
warning: Unable to get systemd shutdown inhibition lock: Sender is not authorized to send message

[root@dell-per740-55 ~]# wget https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/systemd/252/17.el9/x86_64/systemd-252-17.el9.x86_64.rpm

[root@dell-per740-55 ~]# rpm -ivh systemd-252-17.el9.x86_64.rpm 
error: Failed to dlopen /usr/lib64/rpm-plugins/systemd_inhibit.so libsystemd.so.0: cannot open shared object file: No such file or directory
error: error reading from file systemd-252-17.el9.x86_64.rpm

Comment 5 Peter Krempa 2023-08-08 08:35:48 UTC

*** This bug has been marked as a duplicate of bug 2225667 ***