Red Hat Bugzilla – Bug 1506908
further updates to sos openstack plugin in containerized deployment
Last modified: 2018-04-10 14:07:13 EDT
Is this BZ / sos PR 1130 requested to be in RHEL 7.4.z? Or just in 7.5? (Or even later?) Please raise relevant flags accordingly. (not that cloning of z-stream BZs is done by PM only, so I am removing the keyword cloned)
(In reply to Pavel Moravec from comment #1) > Is this BZ / sos PR 1130 requested to be in RHEL 7.4.z? Or just in 7.5? (Or > even later?) Please raise relevant flags accordingly. Thanks for the pointer, we'd need that in 7.4.z + 7.5 . Added the 7.4.z flag > > (not that cloning of z-stream BZs is done by PM only, so I am removing the > keyword cloned) Sure seems that is different from OSP workfow. Will keep that in mind.
Fixed via sos 3.5 rebase.
Martin, could you please provide verification steps or even better verify the BZ (see #c4)?
Thanks for verification. I suggest dealing with the missing bits within this BZ. In particular: - PR 1136 needs merging and backporting - new PR for "nova-manage *" commands to run only when the service is running - are the three commands timeouting / needed to be run that way (is the list of cmds exact?)? self.add_cmd_output("nova-manage db version") self.add_cmd_output("nova-manage fixed list") self.add_cmd_output("nova-manage floating list") The two caught exceptions unrelated to the BZ: I would like to check them and file as independent BZs. Could you pls. provide at least sos_logs/*-plugin-errors.txt (or whole sosreport or access to the system)?
(In reply to Pavel Moravec from comment #10) > Thanks for verification. I suggest dealing with the missing bits within this > BZ. In particular: > > - PR 1136 needs merging and backporting > > - new PR for "nova-manage *" commands to run only when the service is running > - are the three commands timeouting / needed to be run that way (is the > list of cmds exact?)? > self.add_cmd_output("nova-manage db version") > self.add_cmd_output("nova-manage fixed list") > self.add_cmd_output("nova-manage floating list") yes, this is correct. > The two caught exceptions unrelated to the BZ: I would like to check them > and file as independent BZs. Could you pls. provide at least > sos_logs/*-plugin-errors.txt (or whole sosreport or access to the system)? # cat etcd-plugin-errors.txt Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1252, in setup plug.setup() File "/usr/lib/python2.7/site-packages/sos/plugins/etcd.py", line 43, in setup self.add_cmd_output(['%s %s' % (self.cmd, sub) for sub in subcmd]) NameError: global name 'subcmd' is not defined Seems this is a limitation of my env? # cat sos_logs/lsbrelease-plugin-errors.txt Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1300, in collect plug.collect() File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 876, in collect self._collect_cmd_output() File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 851, in _collect_cmd_output env=env) File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 722, in get_cmd_output_now env=env) File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 598, in get_command_output env=env) File "/usr/lib/python2.7/site-packages/sos/utilities.py", line 160, in sos_get_command_output raise e OSError: [Errno 12] Cannot allocate memory
(In reply to Pavel Moravec from comment #10) > Thanks for verification. I suggest dealing with the missing bits within this > BZ. In particular: > > - PR 1136 needs merging and backporting > > - new PR for "nova-manage *" commands to run only when the service is running > - are the three commands timeouting / needed to be run that way (is the > list of cmds exact?)? > self.add_cmd_output("nova-manage db version") > self.add_cmd_output("nova-manage fixed list") > self.add_cmd_output("nova-manage floating list") https://github.com/sosreport/sos/pull/1139 > > > The two caught exceptions unrelated to the BZ: I would like to check them > and file as independent BZs. Could you pls. provide at least > sos_logs/*-plugin-errors.txt (or whole sosreport or access to the system)?
(In reply to Martin Schuppert from comment #11) > (In reply to Pavel Moravec from comment #10) > > Thanks for verification. I suggest dealing with the missing bits within this > > BZ. In particular: > > > > - PR 1136 needs merging and backporting > > > > - new PR for "nova-manage *" commands to run only when the service is running > > - are the three commands timeouting / needed to be run that way (is the > > list of cmds exact?)? > > self.add_cmd_output("nova-manage db version") > > self.add_cmd_output("nova-manage fixed list") > > self.add_cmd_output("nova-manage floating list") > > yes, this is correct. > > > The two caught exceptions unrelated to the BZ: I would like to check them > > and file as independent BZs. Could you pls. provide at least > > sos_logs/*-plugin-errors.txt (or whole sosreport or access to the system)? > > # cat etcd-plugin-errors.txt > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1252, in > setup > plug.setup() > File "/usr/lib/python2.7/site-packages/sos/plugins/etcd.py", line 43, in > setup > self.add_cmd_output(['%s %s' % (self.cmd, sub) for sub in subcmd]) > NameError: global name 'subcmd' is not defined Stupid typo: https://github.com/sosreport/sos/pull/1141 > Seems this is a limitation of my env? > # cat sos_logs/lsbrelease-plugin-errors.txt > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1300, in > collect > plug.collect() > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 876, > in collect > self._collect_cmd_output() > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 851, > in _collect_cmd_output > env=env) > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 722, > in get_cmd_output_now > env=env) > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 598, > in get_command_output > env=env) > File "/usr/lib/python2.7/site-packages/sos/utilities.py", line 160, in > sos_get_command_output > raise e > OSError: [Errno 12] Cannot allocate memory Executing/collecting either of commands: lsb_release -a lsb_release -d must fail, asked for access to the system via mail.
(In reply to Pavel Moravec from comment #14) > (In reply to Martin Schuppert from comment #11) > > (In reply to Pavel Moravec from comment #10) > > > Thanks for verification. I suggest dealing with the missing bits within this > > > BZ. In particular: > > > > > > - PR 1136 needs merging and backporting > > > > > > - new PR for "nova-manage *" commands to run only when the service is running > > > - are the three commands timeouting / needed to be run that way (is the > > > list of cmds exact?)? > > > self.add_cmd_output("nova-manage db version") > > > self.add_cmd_output("nova-manage fixed list") > > > self.add_cmd_output("nova-manage floating list") > > > > yes, this is correct. > > > > > The two caught exceptions unrelated to the BZ: I would like to check them > > > and file as independent BZs. Could you pls. provide at least > > > sos_logs/*-plugin-errors.txt (or whole sosreport or access to the system)? > > > > # cat etcd-plugin-errors.txt > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1252, in > > setup > > plug.setup() > > File "/usr/lib/python2.7/site-packages/sos/plugins/etcd.py", line 43, in > > setup > > self.add_cmd_output(['%s %s' % (self.cmd, sub) for sub in subcmd]) > > NameError: global name 'subcmd' is not defined > > Stupid typo: https://github.com/sosreport/sos/pull/1141 > > > Seems this is a limitation of my env? > > # cat sos_logs/lsbrelease-plugin-errors.txt > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1300, in > > collect > > plug.collect() > > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 876, > > in collect > > self._collect_cmd_output() > > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 851, > > in _collect_cmd_output > > env=env) > > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 722, > > in get_cmd_output_now > > env=env) > > File "/usr/lib/python2.7/site-packages/sos/plugins/__init__.py", line 598, > > in get_command_output > > env=env) > > File "/usr/lib/python2.7/site-packages/sos/utilities.py", line 160, in > > sos_get_command_output > > raise e > > OSError: [Errno 12] Cannot allocate memory > > Executing/collecting either of commands: > > lsb_release -a > lsb_release -d > > must fail, asked for access to the system via mail. Access provided, check you mail. What I checked: - on the undercloud lsb release package is installed, here I do not see the issue [root@undercloud-0 ~]# rpm -qa |grep lsb redhat-lsb-core-4.1-27.el7.x86_64 redhat-lsb-submod-security-4.1-27.el7.x86_64 - on the controllers lsb release is not installed and I see the issue when I run # sosreport ... Running 50/123: logs... caught exception in plugin method "logs.collect()" writing traceback to sos_logs/logs-plugin-errors.txt Running 51/123: lsbrelease... caught exception in plugin method "lsbrelease.collect()" writing traceback to sos_logs/lsbrelease-plugin-errors.txt Both show MemoryError. Note: I also see the memory error on the undercloud in logs-plugin-errors.txt but not for the lsbplugin. Running 43/105: logs... caught exception in plugin method "logs.collect()" writing traceback to sos_logs/logs-plugin-errors.txt - I do not see it on the controller when I only run the lsb plugin - also when I install the redhat-lsb-core package I do see the issue when run sosreport with all plugins. Right now it is installed on the controller. Hope that helps.
Those plugin errors are due to bz 1183244. So to fully resolve this bug, we need to backport PRs: 1136,1139,1141
1) controller node [root@controller-0 ~]# yum update -y sos-3.5-4.el7.noarch.rpm [root@controller-0 ~]# sosreport sosreport (version 3.5) This command will collect diagnostic and configuration information from this Red Hat Enterprise Linux system and installed applications. An archive containing the collected information will be generated in /var/tmp/sos.ihEPif and may be provided to a Red Hat support representative. Any information provided to Red Hat will be treated in accordance with the published support policies at: https://access.redhat.com/support/ The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. Press ENTER to continue, or CTRL-C to quit. Please enter your first initial and last name [controller-0]: Please enter the case id that you are generating this report for []: Setting up archive ... Setting up plugins ... Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Running plugins. Please wait ... Running 51/124: logs... caught exception in plugin method "logs.collect()" writing traceback to sos_logs/logs-plugin-errors.txt Running 52/124: lsbrelease... caught exception in plugin method "lsbrelease.collect()" writing traceback to sos_logs/lsbrelease-plugin-errors.txt Running 124/124: yum... .. Creating compressed archive... Your sosreport has been generated and saved in: /var/tmp/sosreport-controller-0-20180119164415.tar.xz The checksum is: 4f953cc23ccd9d41c8bacd5bd5c514e4 Please send this file to your support representative. As from Pavel the errors above are not related to this BZ - check comment16 * container logs on HOST: [root@controller-0 sosreport-controller-0-20180119164415]# ll /var/log/containers/ total 12 drwxr-xr-x. 2 42402 42402 148 Jan 19 07:00 aodh drwxr-xr-x. 2 42405 42405 66 Jan 10 11:56 ceilometer drwxr-xr-x. 2 42415 42415 264 Jan 19 12:00 glance drwxr-xr-x. 2 42416 42416 101 Jan 10 11:56 gnocchi drwxr-xr-x. 2 42418 42418 4096 Jan 19 15:00 heat drwxr-xr-x. 2 apache apache 25 Jan 10 11:37 horizon drwxr-xr-x. 13 root root 200 Jan 10 11:27 httpd drwxr-xr-x. 2 42425 42425 4096 Jan 19 14:00 keystone drwxr-xr-x. 2 root root 27 Jan 10 11:34 memcached drwxr-xr-x. 2 42436 42436 4096 Jan 19 15:00 nova drwxr-xr-x. 2 42438 42438 45 Jan 10 11:51 panko drwxr-xr-x. 2 42445 42445 6 Jan 10 11:27 swift * container logs in SOSREPORT: [root@controller-0 sosreport-controller-0-20180119164415]# ll var/log/containers/ total 0 drwxr-xr-x. 2 root root 120 Jan 19 07:00 aodh drwxr-xr-x. 2 root root 66 Jan 10 11:56 ceilometer drwxr-xr-x. 2 root root 21 Jan 19 12:00 glance drwxr-xr-x. 2 root root 101 Jan 10 11:56 gnocchi drwxr-xr-x. 2 root root 73 Jan 19 15:00 heat drwxr-xr-x. 2 root root 25 Jan 10 11:37 horizon drwxr-xr-x. 10 root root 146 Jan 10 11:27 httpd drwxr-xr-x. 2 root root 26 Jan 19 14:00 keystone drwxr-xr-x. 2 root root 55 Jan 19 15:00 nova => expected, - we do not have a panko plugin - we do not have a memcached plugin - swift dir is empty * container configs on HOST: [root@controller-0 sosreport-controller-0-20180119164415]# ll /var/lib/config-data/puppet-generated/ | grep -v md5sum total 84 drwxr-xr-x. 4 root root 28 Jan 10 11:32 aodh drwxr-xr-x. 3 root root 17 Jan 10 11:32 ceilometer drwxr-xr-x. 3 root root 17 Jan 10 11:32 clustercheck drwxr-xr-x. 4 root root 28 Jan 10 11:34 crond drwxr-xr-x. 3 root root 17 Jan 10 11:34 glance_api drwxr-xr-x. 4 root root 28 Jan 10 11:34 gnocchi drwxr-xr-x. 3 root root 17 Jan 10 11:32 haproxy drwxr-xr-x. 4 root root 28 Jan 10 11:34 heat drwxr-xr-x. 4 root root 28 Jan 10 11:34 heat_api drwxr-xr-x. 4 root root 28 Jan 10 11:34 heat_api_cfn drwxr-xr-x. 3 root root 17 Jan 10 11:34 horizon drwxr-xr-x. 4 root root 28 Jan 10 11:34 keystone drwxr-xr-x. 3 root root 17 Jan 10 11:34 memcached drwxr-xr-x. 4 root root 29 Jan 10 11:33 mysql drwxr-xr-x. 4 root root 28 Jan 10 11:33 nova drwxr-xr-x. 4 root root 28 Jan 10 11:32 nova_placement drwxr-xr-x. 4 root root 28 Jan 10 11:34 panko drwxr-xr-x. 3 root root 17 Jan 10 11:33 rabbitmq drwxr-xr-x. 3 root root 17 Jan 10 11:32 redis drwxr-xr-x. 3 root root 17 Jan 10 11:32 swift drwxr-xr-x. 3 root root 17 Jan 10 11:32 swift_ringbuilder * container configs in SOSREPORT: [root@controller-0 sosreport-controller-0-20180119164415]# ll var/lib/config-data/puppet-generated/ | grep -v md5sum total 0 drwxr-xr-x. 3 root root 17 Jan 10 11:32 aodh drwxr-xr-x. 3 root root 17 Jan 10 11:32 ceilometer drwxr-xr-x. 4 root root 28 Jan 10 11:34 crond drwxr-xr-x. 3 root root 17 Jan 10 11:34 glance_api drwxr-xr-x. 3 root root 17 Jan 10 11:34 gnocchi drwxr-xr-x. 3 root root 17 Jan 10 11:32 haproxy drwxr-xr-x. 3 root root 17 Jan 10 11:34 heat drwxr-xr-x. 4 root root 28 Jan 10 11:34 heat_api drwxr-xr-x. 4 root root 28 Jan 10 11:34 heat_api_cfn drwxr-xr-x. 3 root root 17 Jan 10 11:34 horizon drwxr-xr-x. 4 root root 28 Jan 10 11:34 keystone drwxr-xr-x. 3 root root 17 Jan 10 11:34 memcached drwxr-xr-x. 3 root root 17 Jan 10 11:33 mysql drwxr-xr-x. 4 root root 28 Jan 10 11:33 nova drwxr-xr-x. 3 root root 17 Jan 10 11:32 nova_placement drwxr-xr-x. 3 root root 17 Jan 10 11:33 rabbitmq drwxr-xr-x. 3 root root 17 Jan 10 11:32 redis drwxr-xr-x. 3 root root 17 Jan 10 11:32 swift => expected information collected. Also the correct api process names are checked: [root@controller-0 sosreport-controller-0-20180119164415]# grep line.endswith /usr/lib/python2.7/site-packages/sos/plugins/openstack_*.py /usr/lib/python2.7/site-packages/sos/plugins/openstack_cinder.py: if line.endswith("cinder_api"): /usr/lib/python2.7/site-packages/sos/plugins/openstack_glance.py: if line.endswith("glance_api"): /usr/lib/python2.7/site-packages/sos/plugins/openstack_heat.py: if line.endswith("heat_api"): /usr/lib/python2.7/site-packages/sos/plugins/openstack_nova.py: if line.endswith("nova_api"): 2) compute node [root@compute-0 ~]# yum update -y sos-3.5-4.el7.noarch.rpm [root@compute-0 ~]# sosreport sosreport (version 3.5) This command will collect diagnostic and configuration information from this Red Hat Enterprise Linux system and installed applications. An archive containing the collected information will be generated in /var/tmp/sos.LE8SGs and may be provided to a Red Hat support representative. Any information provided to Red Hat will be treated in accordance with the published support policies at: https://access.redhat.com/support/ The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. Press ENTER to continue, or CTRL-C to quit. Please enter your first initial and last name [compute-0]: Please enter the case id that you are generating this report for []: Setting up archive ... Setting up plugins ... Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Not all environment variables set. Source the environment file for the user intended to connect to the OpenStack environment. Running plugins. Please wait ... Running 124/124: yum... .. Creating compressed archive... Your sosreport has been generated and saved in: /var/tmp/sosreport-compute-0-20180119164118.tar.xz The checksum is: 8207a007cc0302682a7c5e64fad3ad54 Please send this file to your support representative. * Container config on HOST: [root@compute-0 sosreport-compute-0-20180119164118]# ll /var/lib/config-data/puppet-generated/ total 12 drwxr-xr-x. 3 root root 17 Jan 10 11:29 ceilometer -rw-r--r--. 1 root root 33 Jan 10 11:29 ceilometer.md5sum drwxr-xr-x. 4 root root 28 Jan 10 11:29 crond -rw-r--r--. 1 root root 33 Jan 10 11:29 crond.md5sum drwxr-xr-x. 4 root root 28 Jan 10 11:30 nova_libvirt -rw-r--r--. 1 root root 33 Jan 10 11:30 nova_libvirt.md5sum * container logs in SOSREPORT: [root@compute-0 sosreport-compute-0-20180119164118]# ll var/lib/config-data/puppet-generated/ total 0 drwxr-xr-x. 3 root root 17 Jan 10 11:29 ceilometer drwxr-xr-x. 4 root root 28 Jan 10 11:29 crond drwxr-xr-x. 4 root root 28 Jan 10 11:30 nova_libvirt * container logs on host: [root@compute-0 sosreport-compute-0-20180119164118]# ll /var/log/containers/ total 4 drwxr-xr-x. 2 root root 25 Jan 10 11:48 ceilometer drwxr-xr-x. 2 42436 42436 4096 Jan 19 15:01 nova * Container logs in SOSREPORT: [root@compute-0 sosreport-compute-0-20180119164118]# ll var/log/containers/ total 0 drwxr-xr-x. 2 root root 25 Jan 10 11:48 ceilometer drwxr-xr-x. 2 root root 30 Jan 19 15:01 nova
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0963