Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2041488

Summary:	virsh command timeouts cause sos report to take over 25 minutes
Product:	Red Hat Enterprise Linux 8	Reporter:	Doreen Alongi <dalongi>
Component:	sos	Assignee:	Pavel Moravec <pmoravec>
Status:	CLOSED ERRATA	QA Contact:	Miroslav Hradílek <mhradile>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.6	CC:	agk, bmr, mhradile, plambri, sbradley, theute
Target Milestone:	rc	Keywords:	OtherQA, Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	sos-4.2-13.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-10 15:16:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Doreen Alongi 2022-01-17 13:42:51 UTC

Description of problem:

When running the sosreport test from the redhat-certification suite, it was observed that the test is taking over 25 minutes to run on RHEL 8.6 systems. This is not the case with RHEL 8.5 or RHEL 9.0 systems

I have reproduced the issue outside of redhat-certification with the sos report command. The virsh commands are timing out after 300s each

[plugin:virsh] command 'virsh -r net-list' timed out after 300s
[plugin:virsh] command 'virsh -r nwfilter-list' timed out after 300s
[plugin:virsh] command 'virsh -r pool-list' timed out after 300s
[plugin:wireless] skipped command 'iw list': required kmods missing: cfg80211.
[plugin:wireless] skipped command 'iw dev': required kmods missing: cfg80211.
[plugin:wireless] skipped command 'iwconfig': required kmods missing: cfg80211.
[plugin:wireless] skipped command 'iwlist scanning': required kmods missing: cfg80211.
Running plugins. Please wait ...

Finishing plugins [Running: virsh] ager]
Plugin virsh timed out

[plugin:virsh] command 'virsh -r list --all' timed out after 300s

Version-Release number of selected component (if applicable):
RHEL 8.6
sos 4.2-9.el8

How reproducible:
100%

Steps to Reproduce:
1.run sos report on RHEL 8.6 system
2.
3.

Actual results:
virsh commands timeout

Expected results:
virsh commands should complete quickly and sosreport should take under 5 minutes

Additional info:
Pavel Moravec has identified the issue and has a fix

emulate TTY / run timeout with --foreground option

Comment 1 Doreen Alongi 2022-01-17 17:39:44 UTC

FYI this sosreport test is run every time any redhat-certification test is executed and there is no way to disable this.  If a partner has a failed test, the sosreport helps our cert-ops engineers check their system for configuration issues.  It's a mandatory test.

Having this test take 25+ minutes every time we run any test is severely slowing down our testing of redhat-certification on RHEL 8.6 systems.  We also run nightly automation where 10+ tests are run, each one waiting 25+ minutes for sosreport to run thus tying up many beaker systems for a long time.

Can we please get this fix into RHEL 8.6 ASAP.  We run this sosreport test constantly on x86_64, ppc64le, s390x and aarch64 systems so we could act as OtherQE for this fix

Thanks for your consideration.  Doreen

Comment 2 Pavel Moravec 2022-01-18 11:48:26 UTC

Agreed with QE to contain the fix in 8.6 / in the next respin (week or two timeframe), under the condition of OtherQE.

I will clone it to RHEL9 as well now, to have 8.6 and 9.0 in par.

Comment 3 Pavel Moravec 2022-01-26 08:49:14 UTC

One missing (sanity check) fix identified, will respin.

Comment 4 Doreen Alongi 2022-02-08 12:07:25 UTC

We have run our sosreport test using sos 4.2-13.el8 on x86_64, aarch64, ppc64le and s390x with no issues.  We are no longer seeing the 25 minutes timeouts

Example on aarch64 system.  This took just over 2 minutes with no timeout error messages

 [root@ ~]# date
Tue Feb  8 07:00:29 EST 2022
[root@ ~]# sosreport
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report '

sosreport (version 4.2)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.7r_imve0 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

        Distribution Website : https://www.redhat.com/
        Commercial Support   : https://www.access.redhat.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: tcp_diag, unix_diag, udp_diag, netlink_diag, inet_diag, af_packet_diag.   Use '--allow-system-changes' to enable collection.
[plugin:systemd] skipped command 'resolvectl status': required services missing: systemd-resolved.  
[plugin:systemd] skipped command 'resolvectl statistics': required services missing: systemd-resolved.  
[plugin:wireless] skipped command 'iw list': required kmods missing: cfg80211.  
[plugin:wireless] skipped command 'iw dev': required kmods missing: cfg80211.  
[plugin:wireless] skipped command 'iwconfig': required kmods missing: cfg80211.  
[plugin:wireless] skipped command 'iwlist scanning': required kmods missing: cfg80211.  
 Running plugins. Please wait ...

  Starting 1/95  alternatives    [Running: alternatives]                          Starting 2/95  anaconda        [Running: alternatives anaconda]                 Starting 3/95  anacron         [Running: alternatives anaconda anacron]         Starting 4/95  apache          [Running: alternatives anaconda anacron apache]  Starting 5/95  ata             [Running: alternatives anaconda apache ata]      Starting 6/95  auditd          [Running: alternatives anaconda apache auditd]   Starting 7/95  block           [Running: alternatives apache auditd block]      Starting 8/95  boot            [Running: alternatives apache block boot]        Starting 9/95  ceph            [Running: apache block boot ceph]                Starting 10/95 cgroups         [Running: block boot ceph cgroups]               Starting 11/95 chrony          [Running: block boot cgroups chrony]             Starting 12/95 console         [Running: boot cgroups chrony console]           Starting 13/95 cron            [Running: boot cgroups console cron]             Starting 14/95 crypto          [Running: boot cgroups console crypto]           Starting 15/95 date            [Running: boot cgroups crypto date]              Starting 16/95 dbus            [Running: boot cgroups date dbus]                Starting 17/95 devicemapper    [Running: boot cgroups date devicemapper]        Starting 18/95 devices         [Running: boot cgroups date devices]             Starting 19/95 dnf             [Running: boot cgroups date dnf]                 Starting 20/95 dracut          [Running: boot cgroups dnf dracut]               Starting 21/95 ebpf            [Running: cgroups dnf dracut ebpf]               Starting 22/95 filesys         [Running: cgroups dnf dracut filesys]            Starting 23/95 firewall_tables [Running: cgroups dnf dracut firewall_tables]    Starting 24/95 firewalld       [Running: cgroups dnf dracut firewalld]          Starting 25/95 fwupd           [Running: dnf dracut firewalld fwupd]            Starting 26/95 gluster         [Running: dnf dracut firewalld gluster]          Starting 27/95 grub2           [Running: dnf firewalld grub2]                   Starting 28/95 gssproxy        [Running: dnf firewalld grub2 gssproxy]          Starting 29/95 hardware        [Running: dnf firewalld grub2 hardware]          Starting 30/95 host            [Running: dnf firewalld grub2 host]              Starting 31/95 hts             [Running: firewalld grub2 host hts]              Starting 32/95 i18n            [Running: firewalld grub2 host i18n]             Starting 33/95 iscsi           [Running: firewalld grub2 host iscsi]            Starting 34/95 jars            [Running: firewalld grub2 host jars]             Starting 35/95 kdump           [Running: firewalld grub2 host kdump]            Starting 36/95 kernel          [Running: firewalld grub2 kdump kernel]          Starting 37/95 keyutils        [Running: firewalld grub2 kernel keyutils]       Starting 38/95 krb5            [Running: firewalld grub2 kernel krb5]           Starting 39/95 kvm             [Running: firewalld grub2 kernel kvm]            Starting 40/95 ldap            [Running: firewalld grub2 kvm ldap]              Starting 41/95 libraries       [Running: firewalld grub2 kvm libraries]         Starting 42/95 libvirt         [Running: firewalld grub2 kvm libvirt]           Starting 43/95 login           [Running: firewalld grub2 kvm login]             Starting 44/95 logrotate       [Running: firewalld grub2 kvm logrotate]         Starting 45/95 logs            [Running: firewalld grub2 kvm logs]              Starting 46/95 lvm2            [Running: firewalld kvm logs lvm2]               Starting 47/95 md              [Running: kvm logs lvm2 md]                      Starting 48/95 memory          [Running: kvm logs lvm2 memory]                  Starting 49/95 multipath       [Running: kvm logs lvm2 multipath]               Starting 50/95 networking      [Running: kvm logs lvm2 networking]              Starting 51/95 networkmanager  [Running: kvm logs networking networkmanager]    Starting 52/95 nfs             [Running: kvm networking networkmanager nfs]     Starting 53/95 nis             [Running: kvm networking networkmanager nis]     Starting 54/95 nss             [Running: kvm networking networkmanager nss]     Starting 55/95 ntb             [Running: kvm networking networkmanager ntb]     Starting 56/95 numa            [Running: kvm networking networkmanager numa]    Starting 57/95 nvme            [Running: kvm networking networkmanager nvme]    Starting 58/95 openhpi         [Running: kvm networking networkmanager openhpi  Starting 59/95 openssl         [Running: kvm networking networkmanager openssl  Starting 60/95 pam             [Running: kvm networking networkmanager pam]     Starting 61/95 pci             [Running: kvm networking networkmanager pci]     Starting 62/95 perl            [Running: kvm networking pci perl]               Starting 63/95 pmem            [Running: kvm networking pci pmem]               Starting 64/95 postfix         [Running: kvm networking pmem postfix]           Starting 65/95 process         [Running: kvm networking pmem process]           Starting 66/95 processor       [Running: kvm networking process processor]      Starting 67/95 python          [Running: networking process processor python]   Starting 68/95 release         [Running: process processor python release]      Starting 69/95 rpm             [Running: process processor python rpm]          Starting 70/95 sar             [Running: process python rpm sar]                Starting 71/95 scsi            [Running: process python rpm scsi]               Starting 72/95 selinux         [Running: process rpm scsi selinux]              Starting 73/95 services        [Running: process rpm selinux services]          Starting 74/95 sos_extras      [Running: process rpm selinux sos_extras]        Starting 75/95 soundcard       [Running: process rpm selinux soundcard]         Starting 76/95 ssh             [Running: process rpm selinux ssh]               Starting 77/95 sssd            [Running: process rpm selinux sssd]              Starting 78/95 subscription_manager [Running: process rpm selinux subscription  Starting 79/95 sudo            [Running: process selinux subscription_manager   Starting 80/95 sunrpc          [Running: process selinux subscription_manager   Starting 81/95 system          [Running: process selinux subscription_manager   Starting 82/95 systemd         [Running: selinux subscription_manager system s  Starting 83/95 sysvipc         [Running: selinux subscription_manager systemd   Starting 84/95 teamd           [Running: selinux subscription_manager systemd   Starting 85/95 tuned           [Running: selinux subscription_manager systemd   Starting 86/95 udev            [Running: selinux subscription_manager systemd   Starting 87/95 unpackaged      [Running: selinux subscription_manager systemd   Starting 88/95 usb             [Running: selinux subscription_manager systemd   Starting 89/95 vhostmd         [Running: selinux subscription_manager systemd   Starting 90/95 virsh           [Running: selinux subscription_manager systemd   Starting 91/95 wireless        [Running: selinux subscription_manager virsh wi  Starting 92/95 x11             [Running: selinux subscription_manager virsh x1  Starting 93/95 xen             [Running: selinux subscription_manager virsh xe  Starting 94/95 xfs             [Running: selinux subscription_manager virsh xf  Finishing plugins              [Running: selinux subscription_manager xfs]      Starting 95/95 yum             [Running: selinux subscription_manager xfs yum]  Finishing plugins              [Running: selinux xfs yum]                       Finishing plugins              [Running: selinux yum]                           Finishing plugins              [Running: selinux]                                       
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
	/var/tmp/sosreport-hp-moonshot-03-c27-2022-02-08-qhguezm.tar.xz

 Size	14.17MiB
 Owner	root
 sha256	524ffac90f45ee9117d001b8edad83a4f5273fedd5916874b7b8d74fe0b9e831

Please send this file to your support representative.

[root@ ~]# date
Tue Feb  8 07:02:38 EST 2022

Comment 8 errata-xmlrpc 2022-05-10 15:16:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:1998