Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1754562

Summary: sosreport hangs after executing all plugins
Product: Red Hat Enterprise Linux 7 Reporter: Eduard Barrera <ebarrera>
Component: sosAssignee: Pavel Moravec <pmoravec>
Status: CLOSED DUPLICATE QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: agk, bmr, jjansky, ktordeur, michael.ballon, plambri, sbradley
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-11 13:43:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eduard Barrera 2019-09-23 14:56:35 UTC
Description of problem:

The process is stucking right after finishing running plugin portion:
.
.
.
  Finishing plugins              [Running: virsh yum]
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-arp-spoofing'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-ip-multicast'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-ip-spoofing'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-mac-broadcast'
[plugin:yum] collecting output of 'yum list installed'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-mac-spoofing'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-other-l2-traffic'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml no-other-rarp-traffic'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml qemu-announce-self'
[plugin:virsh] collecting output of 'virsh -r nwfilter-dumpxml qemu-announce-self-rarp'
[plugin:virsh] collecting output of 'virsh -r pool-list'
  Finishing plugins              [Running: yum]
[plugin:yum] collecting output of 'package-cleanup --dupes'
[plugin:yum] collecting output of 'package-cleanup --problems'
[plugin:yum] collecting string ...'search-disabled-repos' as 'plugin-names'
[plugin:openstack_neutron] collecting path '/var/lib/config-data/puppet-generated/neutron/etc/neutron/'
[plugin:openstack_neutron] collecting path '/var/log/neutron/'
[plugin:openstack_neutron] collecting path '/var/lib/config-data/puppet-generated/neutron/etc/default/neutron-server'
[plugin:openstack_neutron] collecting path '/var/log/containers/httpd/neutron-api/'
[plugin:openstack_neutron] collecting path '/var/lib/config-data/puppet-generated/neutron/etc/my.cnf.d/tripleo.cnf'
[plugin:openstack_neutron] collecting path '/etc/neutron/'
[plugin:openstack_neutron] failed to stat '/etc/neutron/plugins/ml2/ml2_conf_sriov.ini'

  Finished running plugins


/usr/lib/python2.7/site-packages/sos

   1471             self.setup()
   1472             self.collect()   <=== Here we print "Finished running plugins"
   1473             if not self.opts.noreport:
   1474                 self.report()
   1475                 self.html_report()
   1476                 self.plain_report()
   1477             self.postproc()
   1478             self.version()
   1479             return self.final_work()
   1480


Version-Release number of selected component (if applicable):
unsure

How reproducible:
unsure

Steps to Reproduce:
1. unsure 
2.
3.

Actual results:
sosreports can't be generated after updating openstack

Expected results:


Additional info:

Comment 2 Pavel Moravec 2019-09-23 15:00:49 UTC
maybe bz 1704957 ? how many files could have been collected? Could we get /var/tmp/sos.* working directory?

Comment 3 Pavel Moravec 2020-02-04 14:32:07 UTC
(In reply to Pavel Moravec from comment #2)
> maybe bz 1704957 ? how many files could have been collected? Could we get
> /var/tmp/sos.* working directory?

Reiteration of the needinfo.

Until said elsehow, we suppose this is a dup of 1704957 and will be closed when 7.8 is GA.

Comment 4 Mike Ballon 2020-05-05 19:41:54 UTC
I had the same result, sosreport would hang after "Finished running plugins". In my case the last plugin that ran was yum, I tried skipping it without luck.

While the running the sosreport in -vvv I noticed just screen and screen output from cgroups plugin, I exclude that and sosreport was able to complete.

Example success: sosreport -vvv --skip-plugin cgroups

Comment 5 Pavel Moravec 2020-05-05 20:39:35 UTC
(In reply to Mike Ballon from comment #4)
> I had the same result, sosreport would hang after "Finished running
> plugins". In my case the last plugin that ran was yum, I tried skipping it
> without luck.
> 
> While the running the sosreport in -vvv I noticed just screen and screen
> output from cgroups plugin, I exclude that and sosreport was able to
> complete.
> 
> Example success: sosreport -vvv --skip-plugin cgroups

Doesn't either directory

            "/proc/cgroups",
            "/sys/fs/cgroup",
            "/etc/sysconfig/cgconfig",
            "/etc/sysconfig/cgred",

contain tens of thousands of files (or even more)? If so, then https://bugzilla.redhat.com/show_bug.cgi?id=1704957 really applies and an upgrade to sos-3.8-* helps here.

Comment 6 Mike Ballon 2020-05-05 21:30:53 UTC
I will try and update sos when allowed, although the file counts are all low.

I did notice the host has a rather evil amount of mounts though.

# mount | wc -l
666

# mount | grep -c '\.snapshot'
556

# ls /etc/sysconfig/cgred | wc -l
1
# cat /proc/cgroups | wc -l
13
# ls /sys/fs/cgroup | wc -l
15
# ls /etc/sysconfig/cgconfig | wc -l
ls: cannot access /etc/sysconfig/cgconfig: No such file or directory
0

Comment 7 Pavel Moravec 2020-05-06 11:23:23 UTC
(In reply to Mike Ballon from comment #6)
> I will try and update sos when allowed, although the file counts are all low.
> 
> I did notice the host has a rather evil amount of mounts though.
> 
> # mount | wc -l
> 666
> 
> # mount | grep -c '\.snapshot'
> 556
> 
> # ls /etc/sysconfig/cgred | wc -l
> 1
> # cat /proc/cgroups | wc -l
> 13
> # ls /sys/fs/cgroup | wc -l
> 15
> # ls /etc/sysconfig/cgconfig | wc -l
> ls: cannot access /etc/sysconfig/cgconfig: No such file or directory
> 0


OK looking forward for the results of updated sos. FYI the number of subdirs isnt the key problem, but number of files in the dirs can be - you shall rather count:

find /sys/fs/cgroup | wc

or similar.

Comment 8 Mike Ballon 2020-05-06 15:11:59 UTC
# find /sys/fs/cgroup | wc
  73251   73251 7518495

Updating sos appears to have resolved my issue.

# rpm -qa | grep sos
sos-3.8-6.el7.noarch

Thanks!

Comment 9 Pavel Moravec 2020-05-11 13:43:55 UTC
(In reply to Mike Ballon from comment #8)
> # find /sys/fs/cgroup | wc
>   73251   73251 7518495
> 
> Updating sos appears to have resolved my issue.
> 
> # rpm -qa | grep sos
> sos-3.8-6.el7.noarch
> 
> Thanks!

Thanks for the confirmation.

Many tens of thousands of files being collected by a single plugin (and html report enabled, that is the default) is the trigger of bz1704957 resolved in 3.8.

Closing as duplicate of the BZ, then.

*** This bug has been marked as a duplicate of bug 1704957 ***

Comment 10 Red Hat Bugzilla 2023-09-14 05:43:39 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days