Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1356112

Summary: sosreport runs glance-manage db_version and heat-manage db_version commands on compute nodes
Product: Red Hat Enterprise Linux 7 Reporter: Marius Cornea <mcornea>
Component: sosAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Jan Houska <jhouska>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: agk, bmr, gavin, jhouska, lyarwood, mcornea, mhradile, mschuppe, plambri, sbradley
Target Milestone: rcKeywords: OtherQA, Triaged, ZStream
Target Release: 7.4   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: sos-3.5-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 18:00:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2016-07-13 12:39:18 UTC
Description of problem:

sosreport runs glance-manage db_version and heat-manage db_version commands on compute nodes which hangs for a couple of minutes because the non controllers nodes don't get configured for these services:

During sosreport run:

[root@overcloud-compute-0 heat-admin]# ps axu | grep glance
root      9282  0.0  0.0   8584   648 pts/1    S    12:27   0:00 timeout 300s glance-manage db_version
root      9283  2.4  1.2 312976 76896 pts/1    S    12:27   0:01 /usr/bin/python2 /usr/bin/glance-manage db_version
root      9428  0.0  0.0 112648   972 pts/2    S+   12:28   0:00 grep --color=auto glance

[root@overcloud-compute-0 heat-admin]# ps axu | grep heat
root      9548  0.0  0.0   8584   652 pts/1    S    12:29   0:00 timeout 300s heat-manage db_version
root      9549  3.9  0.8 312172 52804 pts/1    S    12:29   0:01 /usr/bin/python /usr/bin/heat-manage db_version
root      9631  0.0  0.0 112648   972 pts/2    S+   12:29   0:00 grep --color=auto heat

Version-Release number of selected component (if applicable):
sos-3.2-36.el7ost.2.noarch

How reproducible:
100%

Steps to Reproduce:
1. Run sosreport on non controllers nodes

Actual results:
During the run check the heat and glance commands. 

Expected results:
There are no db_version commands ran as the services are not configured to know about the db.

Comment 4 Pavel Moravec 2017-01-19 09:21:06 UTC
(showing glance issue, equivalent one is for heat):

/usr/lib/python2.7/site-packages/sos/plugins/openstack_glance.py :

class OpenStackGlance(Plugin):
    """OpenStack Glance"""
    plugin_name = "openstack_glance"
    profiles = ('openstack', 'openstack_controller')

    option_list = []

    def setup(self):
        # Glance
        self.add_cmd_output(
            "glance-manage db_version",
            suggest_filename="glance_db_version"
        )


The "profiles = " triggers the plugin is run on a system with either openstack or openstack_controller package installed. Then everytime, the "glance-manage db_version" is called.

What shall be the trigger to call this command? Only presence of openstack_controller?

Other commands and logs collection from this plugin shall be collected on either controller or compute nodes, right? Just this command shall not be called on compute node (where openstack_controller is missing, I guess)?

Comment 6 Pavel Moravec 2017-08-30 08:28:03 UTC
(In reply to Pavel Moravec from comment #4)
> (showing glance issue, equivalent one is for heat):
> 
> /usr/lib/python2.7/site-packages/sos/plugins/openstack_glance.py :
> 
> class OpenStackGlance(Plugin):
>     """OpenStack Glance"""
>     plugin_name = "openstack_glance"
>     profiles = ('openstack', 'openstack_controller')
> 
>     option_list = []
> 
>     def setup(self):
>         # Glance
>         self.add_cmd_output(
>             "glance-manage db_version",
>             suggest_filename="glance_db_version"
>         )
> 
> 
> The "profiles = " triggers the plugin is run on a system with either
> openstack or openstack_controller package installed. Then everytime, the
> "glance-manage db_version" is called.
> 
> What shall be the trigger to call this command? Only presence of
> openstack_controller?
> 
> Other commands and logs collection from this plugin shall be collected on
> either controller or compute nodes, right? Just this command shall not be
> called on compute node (where openstack_controller is missing, I guess)?

Hello,
could you please comment here (I forgot to raise needinfo here)?

Comment 7 Marius Cornea 2017-08-30 11:34:07 UTC
(In reply to Pavel Moravec from comment #6)
> (In reply to Pavel Moravec from comment #4)
> > (showing glance issue, equivalent one is for heat):
> > 
> > /usr/lib/python2.7/site-packages/sos/plugins/openstack_glance.py :
> > 
> > class OpenStackGlance(Plugin):
> >     """OpenStack Glance"""
> >     plugin_name = "openstack_glance"
> >     profiles = ('openstack', 'openstack_controller')
> > 
> >     option_list = []
> > 
> >     def setup(self):
> >         # Glance
> >         self.add_cmd_output(
> >             "glance-manage db_version",
> >             suggest_filename="glance_db_version"
> >         )
> > 
> > 
> > The "profiles = " triggers the plugin is run on a system with either
> > openstack or openstack_controller package installed. Then everytime, the
> > "glance-manage db_version" is called.
> > 
> > What shall be the trigger to call this command? Only presence of
> > openstack_controller?
> > 
> > Other commands and logs collection from this plugin shall be collected on
> > either controller or compute nodes, right? Just this command shall not be
> > called on compute node (where openstack_controller is missing, I guess)?
> 
> Hello,
> could you please comment here (I forgot to raise needinfo here)?

I think with the new composable roles architecture we should trigger the component related commands based on whether the service is running on a particular node. We cannot assume that a node is an openstack controller anymore because the controller services can be split across different nodes. So for instance in case of glance we should check that the openstack-glance-api service status and run the glance-manage db_version command only on the nodes where it is running.

Comment 8 Pavel Moravec 2017-09-03 14:54:05 UTC
Just to confirm the change:

- additional check whether "service openstack-glance-api status" returns 0 should be added (or "service openstack-heat-api status")

- if the check returns nonzero, then just "[heat|glance]-manage db_version" command must be skipped - or whole plug-in as well must be skipped? I.e. when the service isnt running on the particular node, shall we collect stuff like

/var/log/glance/*.log
/etc/glance/*
output of "openstack image list --long"

(for glance, similar stuff is for heat)

If service isnt running on the particular node, shall or shouldnt we collect the above configs/logs/commands output?

Comment 9 Pavel Moravec 2017-09-27 10:44:07 UTC
(In reply to Pavel Moravec from comment #8)
> Just to confirm the change:
> 
> - additional check whether "service openstack-glance-api status" returns 0
> should be added (or "service openstack-heat-api status")
> 
> - if the check returns nonzero, then just "[heat|glance]-manage db_version"
> command must be skipped - or whole plug-in as well must be skipped? I.e.
> when the service isnt running on the particular node, shall we collect stuff
> like
> 
> /var/log/glance/*.log
> /etc/glance/*
> output of "openstack image list --long"
> 
> (for glance, similar stuff is for heat)
> 
> If service isnt running on the particular node, shall or shouldnt we collect
> the above configs/logs/commands output?

Bouncing needinfo, esp. if the BZ should be targeted to RHEL 7.5 (with devel freeze bit approaching already).

Comment 10 Marius Cornea 2017-09-27 11:14:57 UTC
(In reply to Pavel Moravec from comment #8)
> Just to confirm the change:
> 
> - additional check whether "service openstack-glance-api status" returns 0
> should be added (or "service openstack-heat-api status")
> 
> - if the check returns nonzero, then just "[heat|glance]-manage db_version"
> command must be skipped - or whole plug-in as well must be skipped? I.e.
> when the service isnt running on the particular node, shall we collect stuff
> like
> 
> /var/log/glance/*.log
> /etc/glance/*
> output of "openstack image list --long"
> 
> (for glance, similar stuff is for heat)
> 
> If service isnt running on the particular node, shall or shouldnt we collect
> the above configs/logs/commands output?

Sorry for the delay. To answer your question we need to:

1. check the service status(service, systemctl, etc)

2. if service is running then run all commands and collection related to that particlar service. if it's not running then do not run the commands/collection related to the service.

Comment 11 Pavel Moravec 2017-10-12 12:28:01 UTC
I asked also whether it makes sense to stop collecting logs and configs in case the service is not running. Anyway after clarification with Lee and mschuppert, we agreed it makes sense to collect them (i.e. to troubleshoot cases when service fails to start).

See linked upstream PR.

Comment 12 Pavel Moravec 2017-11-02 15:15:51 UTC
Fixed via sos 3.5 rebase.

Comment 20 errata-xmlrpc 2018-04-10 18:00:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0963