(In reply to Alexander Chuzhoy from comment #0)
> The sosreport is missing /var/log/containers/nova/nova-conductor.log
> 
> [root@controller-0 ~]# ls /var/log/containers/nova/
> nova-api.log    nova-api.log.3  nova-api.log.6        
> nova-api-metadata.log.1  nova-api-metadata.log.4  nova-consoleauth.log 
> nova-placement-api.log
> nova-api.log.1  nova-api.log.4  nova-api.log.7        
> nova-api-metadata.log.2  nova-compute.log         nova-manage.log      
> nova-rowsflush.log
> nova-api.log.2  nova-api.log.5  nova-api-metadata.log 
> nova-api-metadata.log.3  nova-conductor.log       nova-novncproxy.log  
> nova-scheduler.log
> 
> Extracted sosreport collected from controller-0 (executed sosreport --batch):
> 
> (undercloud) [stack@undercloud-0 nova]$ pwd
> /home/stack/sosreports/sosreport-controller-0-20180619140001/var/log/
> containers/nova
> (undercloud) [stack@undercloud-0 nova]$ ls
> nova-api.log

Can you provide --debug output when collecting the sosreport and also try again with --all-logs?

Tried on OSP14 with: 'sosreport --batch --debug --all-logs'

Same result:
Only 'mysql  rabbitmq  redis' dirs are under /var/log/containers

Since you nuked all the BZ template fields we don't know what version you are running.

Support for this was added in sos-3.5 which was included in rhel-7.5. If you are running an earlier version please update and try again.

If you are running some z-stream build then it may be the case that nobody has requested a backport of these changes to the affected release (assuming it is still supported).

If you are running an up-to-date version please attach the sos log from the archive (sos_logs/sos.log), or the terminal output when run with -vv.

Version:
sos-3.5-9.el7_5.noarch

Created attachment 1480093 [details]
sos.log

Debug output:
[root@controller-0 ~]# sosreport --batch --debug --all-logs                                                                                                                         [103/1828]

sosreport (version 3.5)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.AGqQfe and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment.
 Running plugins. Please wait ...

  Running 1/105: anaconda...        
  Running 2/105: anacron...        
  Running 3/105: apache...        
  Running 4/105: auditd...        
  Running 5/105: autofs...        
  Running 6/105: block...        
  Running 7/105: boot...        
  Running 8/105: ceph...        
  Running 9/105: cgroups...        
  Running 10/105: chrony...        
  Running 11/105: corosync...        
  Running 12/105: cron...        
  Running 13/105: crypto...  
  Running 14/105: dbus...        
  Running 15/105: devicemapper...        
  Running 16/105: devices...        
  Running 17/105: dlm...        
  Running 18/105: docker...        
  Running 19/105: dracut...        
  Running 20/105: etcd...        
  Running 21/105: filesys...        
  Running 22/105: firewalld...        
  Running 23/105: general...        
  Running 24/105: gluster...        
  Running 25/105: grub2...        
  Running 26/105: haproxy...        
  Running 27/105: hardware...        
  Running 28/105: hts...        
  Running 29/105: i18n...        
  Running 30/105: ipa...        
  Running 31/105: ipmitool...        
  Running 32/105: iscsi...        
  Running 33/105: jars...        
  Running 34/105: java...        
  Running 35/105: kdump...        
  Running 36/105: keepalived...        
  Running 37/105: kernel...        
  Running 38/105: keyutils...        
  Running 39/105: krb5...        
  Running 40/105: kvm...        
  Running 41/105: last...        
  Running 42/105: ldap...        
  Running 43/105: libraries...        
  Running 44/105: libvirt...        
  Running 45/105: logrotate...        
  Running 46/105: logs...        
  Running 47/105: lsbrelease...        
  Running 48/105: lvm2...        
  Running 49/105: md...        
  Running 50/105: megacli...        
  Running 51/105: memory...        
  Running 52/105: mrggrid...    
 Running 53/105: mrgmessg...        
  Running 54/105: multipath...        
  Running 55/105: mysql...        
  Running 56/105: networking...        
  Running 57/105: nfs...        
  Running 58/105: nis...        
  Running 59/105: nss...        
  Running 60/105: ntb...        
  Running 61/105: ntp...        
  Running 62/105: numa...        
  Running 63/105: oddjob...        
  Running 64/105: opendaylight...        
  Running 65/105: openhpi...        
  Running 66/105: openshift...        
  Running 67/105: openssl...        
  Running 68/105: openstack_instack...        
  Running 69/105: openstack_manila...        
  Running 70/105: openswan...        
  Running 71/105: os_net_config...        
  Running 72/105: pacemaker...        
  Running 73/105: pam...        
  Running 74/105: pci...        
  Running 75/105: perl...        
  Running 76/105: postfix...        
  Running 77/105: process...        
  Running 78/105: processor...        
  Running 79/105: puppet...        
  Running 80/105: python...        
  Running 81/105: rabbitmq...        
  Running 82/105: redis...        
  Running 83/105: rpm...        
  Running 84/105: samba...        
  Running 85/105: scsi...        
  Running 86/105: selinux...        
  Running 87/105: services...        
  Running 88/105: snmp...        
  Running 89/105: soundcard...        
  Running 90/105: ssh...        
  Running 91/105: sssd...        
 Running 92/105: subscription_manager...        
  Running 93/105: system...        
  Running 94/105: systemd...        
  Running 95/105: sysvipc...        
  Running 96/105: teamd...        
  Running 97/105: tuned...        
  Running 98/105: udev...        
  Running 99/105: usb...        
  Running 100/105: vhostmd...        
  Running 101/105: virsh...        
  Running 102/105: x11...        
  Running 103/105: xen...        
  Running 104/105: xfs...        
  Running 105/105: yum...        

Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-controller-0-20180831131705.tar.xz

The checksum is: fed81d50238c2a9d76793c852ffc5835

Please send this file to your support representative.

(In reply to Alexander Chuzhoy from comment #5)
> Version:
> sos-3.5-9.el7_5.noarch

(In reply to Alexander Chuzhoy from comment #7)
> Debug output:
> [root@controller-0 ~]# sosreport --batch --debug --all-logs                 
> [103/1828]
>
> [..]       
>
>   Running 68/105: openstack_instack...        
>   Running 69/105: openstack_manila...        

Thanks Sasha, so this shows that the plugins aren't even running on OSP 14. On IRC you were able to provide the following package list that confirms this is due to a number of openstack-* packages no longer being shipped in the overcloud images:

[root@controller-0 ~]# rpm -qa|grep openstack
puppet-openstacklib-13.2.0-0.20180724214016.a47c20e.el7ost.noarch
python-openstackclient-lang-3.16.0-0.20180801103525.f77ca68.el7ost.noarch
python2-openstacksdk-0.17.1-0.20180801104359.f28a1b0.el7ost.noarch
python2-openstackclient-3.16.0-0.20180801103525.f77ca68.el7ost.noarch
openstack-heat-agents-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
puppet-openstack_extras-13.2.0-0.20180728124012.4540a49.el7ost.noarch
openstack-selinux-0.8.15-0.20180524134826.b63283a.el7ost.noarch

This results in the check_enabled methods returning False, disabling the plugins. It appears that we didn't update these methods within the plugins when we introduced support for containers.

Just for reference, the output in comment #5 is not what we need - the --debug option enables the Python debugger, as well as _additional_ log messages when sos is run in verbose mode. You need the -vv option that was requested in comment #4 to enable verbose logging to the terminal.

Anyway, since we only need one of the sos.log or the terminal output we can start working from the log file provided - thanks.

In this case, the openstack_nova plugin was not executed at all.

In the version of sos that you are running, the package triggers for this plugin are as follows:

    packages = (
        'openstack-nova-common',
        'openstack-nova-network',
        'openstack-nova-conductor',
        'openstack-nova-conductor',
        'openstack-nova-scheduler',
        'openstack-nova-console',
        'openstack-nova-novncproxy',
        'openstack-nova-compute',
        'openstack-nova-api',
        'openstack-nova-cert',
        'openstack-nova-cells',
        'openstack-nova-objectstore',
        'python-nova',
        'python-novaclient',
        'novnc'
    )

Unfortunately, the contributor who made these changes seems to have misunderstood the mechanism, and has also implemented a check_enabled() method:

    def check_enabled(self):
        self.nova = self.is_installed("openstack-nova-common")
        return self.nova

This is completely bogus: it makes zero sense to declare a whole package list, and to then override check_enabled in a way that tests for only one of those packages.

As an initial test you can force the plugin to run from the command line:

  # sosreport -vv --batch -o openstack_nova

This will just run the one named plugin but it should allow you to confirm whether the logs are then collected correctly. Note that only current logs will be collected unless --all-logs is given.

If you'd like to test an actual fix for this problem you can edit /usr/lib/python2.7/site-packages/sos/plugins/openstack_nova.py and delete the entire check_enabled() method from the RedHatNova class (you can also delete the 'nova' member from the class, but this isn't important for testing).

Alternately apply the attached patch to the same path.

Talking this over with Lee some more on IRC, there is another problem here beyond the bug in the check_enabled() methods.

Since parts of OpenStack are now in containers they are not visible to the host package manager (which the default check_enabled() looks at). We need a completely new mechanism to either test for specific named containers, or to inspect container package sets for a match. This is a significant piece of work and is not appropriate for consideration at this late stage in 7.6.

As a workaround I suggested that we enable all the openstack_* plugins in 7.6 if any host package exists that would suggest that this is an OpenStack system. This will cause some extra plugins to run in some cases, but this is harmless, and it will ensure that data is collected in 7.6 even without a proper container enablement check.

Talking to Lee it sounds like 'openstack-selinux' would be the best package to check for for the time being.

(In reply to Bryn M. Reeves from comment #13)
> Talking this over with Lee some more on IRC, there is another problem here
> beyond the bug in the check_enabled() methods.
> 
> Since parts of OpenStack are now in containers they are not visible to the
> host package manager (which the default check_enabled() looks at). We need a
> completely new mechanism to either test for specific named containers, or to
> inspect container package sets for a match. This is a significant piece of
> work and is not appropriate for consideration at this late stage in 7.6.
> 
> As a workaround I suggested that we enable all the openstack_* plugins in
> 7.6 if any host package exists that would suggest that this is an OpenStack
> system. This will cause some extra plugins to run in some cases, but this is
> harmless, and it will ensure that data is collected in 7.6 even without a
> proper container enablement check.
> 
> Talking to Lee it sounds like 'openstack-selinux' would be the best package
> to check for for the time being.

Updated the upstream PR accordingly.

We didnt catch re-spin on time before blocker+ is required, and the BZ severity/priority does not qualify it to be RHEL7.6 blocker. Hence deferring the BZ to RHEL7.7 and potential/probable inclusion in 7.6.z.

*** Bug 1624450 has been marked as a duplicate of this bug. ***

Hello,
could you please verify the BZ against below build? Thanks in advance.


A yum repository for the build of sos-3.7-1.el7 (task 20748546) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/sos-3.7-1.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/sos-3.7-1.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/sos-3.7-1.el7.noarch.rpm

Build output will be available for the next 21 days.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2295