Bug 1459691 - openstack-heat: openstack-heat-api.service (and openstack-heat-api-cfn) shows failed after "openstack undercloud upgrade" completes duing minor update on OSP9
openstack-heat: openstack-heat-api.service (and openstack-heat-api-cfn) shows...
Status: CLOSED WONTFIX
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tempestconf (Show other bugs)
9.0 (Mitaka)
Unspecified Unspecified
high Severity unspecified
: async
: 9.0 (Mitaka)
Assigned To: Chandan Kumar
Martin Kopec
: ZStream
: 1459694 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-07 16:29 EDT by Alexander Chuzhoy
Modified: 2017-08-01 05:10 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-01 05:10:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sosreport from the undercloud node (15.64 MB, application/x-xz)
2017-06-07 16:39 EDT, Alexander Chuzhoy
no flags Details

  None (edit)
Description Alexander Chuzhoy 2017-06-07 16:29:34 EDT
openstack-glance: openstack-heat-api.service shows failed after "openstack undercloud upgrade" completes duing minor update on OSP9


Environment:
python-glance-12.0.0-2.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-54.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch
python-glanceclient-2.0.0-1.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-54.el7ost.noarch
openstack-puppet-modules-8.1.10-2.el7ost.noarch
python-glance-tests-12.0.0-2.el7ost.noarch
instack-undercloud-4.0.0-16.el7ost.noarch
openstack-glance-12.0.0-2.el7ost.noarch



Steps to reproduce:
1. Run "openstack undercloud upgrade"
2. Reboot the machine (if needed - new kernel)
3. run :

[root@director ~]# systemctl status openstack-heat-api.service


Result:

● openstack-heat-api.service - OpenStack Heat API Service
   Loaded: loaded (/usr/lib/systemd/system/openstack-heat-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:43 CDT; 54min ago
  Process: 5190 ExecStart=/usr/bin/heat-api (code=exited, status=1/FAILURE)
 Main PID: 5190 (code=exited, status=1/FAILURE)

Jun 07 14:33:42 director.fv1dci.org systemd[1]: Unit openstack-heat-api.service entered failed state.
Jun 07 14:33:42 director.fv1dci.org systemd[1]: openstack-heat-api.service failed.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: openstack-heat-api.service holdoff time over, scheduling restart.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-heat-api.service
Jun 07 14:33:43 director.fv1dci.org systemd[1]: Failed to start OpenStack Heat API Service.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: Unit openstack-heat-api.service entered failed state.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: openstack-heat-api.service failed.



[root@director ~]# journalctl -u openstack-heat-api.service|grep -i error
Jun 07 14:23:00 director.fv1dci.org heat-api[29433]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:01 director.fv1dci.org heat-api[29496]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:02 director.fv1dci.org heat-api[29547]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:04 director.fv1dci.org heat-api[29652]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:05 director.fv1dci.org heat-api[29724]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:06 director.fv1dci.org heat-api[29744]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:07 director.fv1dci.org heat-api[29774]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:09 director.fv1dci.org heat-api[29797]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:37 director.fv1dci.org heat-api[1529]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:38 director.fv1dci.org heat-api[2626]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:40 director.fv1dci.org heat-api[3001]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:41 director.fv1dci.org heat-api[3561]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:42 director.fv1dci.org heat-api[5190]: ERROR: Unable to locate config file [api-paste.ini]

Expected result:
openstack-heat-api.service should be UP and running.
Comment 1 Alexander Chuzhoy 2017-06-07 16:34:05 EDT
openstack-heat-api-cfn.service is also down:

[root@director ~]# systemctl status openstack-heat-api.service
● openstack-heat-api.service - OpenStack Heat API Service
   Loaded: loaded (/usr/lib/systemd/system/openstack-heat-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:43 CDT; 1h 0min ago
  Process: 5190 ExecStart=/usr/bin/heat-api (code=exited, status=1/FAILURE)
 Main PID: 5190 (code=exited, status=1/FAILURE)

Jun 07 14:33:42 director.fv1dci.org systemd[1]: Unit openstack-heat-api.service entered failed state.
Jun 07 14:33:42 director.fv1dci.org systemd[1]: openstack-heat-api.service failed.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: openstack-heat-api.service holdoff time over, scheduling restart.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-heat-api.service
Jun 07 14:33:43 director.fv1dci.org systemd[1]: Failed to start OpenStack Heat API Service.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: Unit openstack-heat-api.service entered failed state.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: openstack-heat-api.service failed.
[root@director ~]# journalctl -u openstack-heat-api.service|grep -i error
Jun 07 14:23:00 director.fv1dci.org heat-api[29433]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:01 director.fv1dci.org heat-api[29496]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:02 director.fv1dci.org heat-api[29547]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:04 director.fv1dci.org heat-api[29652]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:05 director.fv1dci.org heat-api[29724]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:06 director.fv1dci.org heat-api[29744]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:07 director.fv1dci.org heat-api[29774]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:09 director.fv1dci.org heat-api[29797]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:37 director.fv1dci.org heat-api[1529]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:38 director.fv1dci.org heat-api[2626]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:40 director.fv1dci.org heat-api[3001]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:41 director.fv1dci.org heat-api[3561]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:42 director.fv1dci.org heat-api[5190]: ERROR: Unable to locate config file [api-paste.ini]
Comment 2 Alexander Chuzhoy 2017-06-07 16:39 EDT
Created attachment 1285913 [details]
sosreport from the undercloud node
Comment 3 Alex Schultz 2017-06-07 17:44:42 EDT
We appear to be missing the following for the services:

https://github.com/rdo-packages/heat-distgit/commit/9c7d76cdcfa27335986235072ba622a173034452

starting the service by hand with these command line works, the one shipped lacks the dist.conf items so it cannot find the dist provided api-paste.ini
Comment 4 Zane Bitter 2017-06-07 17:49:55 EDT
That explains why bug 1452677 showed up in OSP10.
Comment 5 Alex Schultz 2017-06-07 17:58:11 EDT
Taking a further look, it's not that commit. It is that problem where the *-dist.conf files aren't being loaded but I'm not sure where that was accomplished previously.  I just spun up a new 9 undercloud and it's working and loading the heat-dist.conf but i'm not sure where that is being injected.
Comment 6 Alex Schultz 2017-06-08 11:13:28 EDT
So i tracked this down to python-oslo-config. In the shipped versions there's a specific code to handle the -dist.config files in /usr/lib/python2.7/site-packages/oslo_config/cfg.py

    config_files = []
    if project:
        config_files.append(_search_dirs(['/usr/share/%s/' % project],
                                         project, '-dist%s' % extension))
        config_files.append(_search_dirs(['/usr/share/%s/' % project],
                                         prog, '-dist%s' % extension))
        config_files.append(_search_dirs(cfg_dirs, project, extension))
    config_files.append(_search_dirs(cfg_dirs, prog, extension))


This is missing from the file on the system and doing an rpm verification shows that the files are not the package provided files. Additionally pip info oslo.config shows vs 4.3.0 but the package version should be 3.9.0. Someone/something upgraded oslo.config using a non-packaged version.

[root@director cmd]# pip show oslo.config                                                                                                                                                     |
Name: oslo.config
Version: 4.3.0
Summary: Oslo Configuration API
Home-page: http://docs.openstack.org/developer/oslo.config/
Author: OpenStack
Author-email: openstack-dev@lists.openstack.org
License: UNKNOWN
Location: /usr/lib/python2.7/site-packages
Requires: rfc3986, oslo.i18n, six, netaddr, debtcollector, stevedore
Comment 7 Alex Schultz 2017-06-08 11:15:08 EDT
*** Bug 1459694 has been marked as a duplicate of this bug. ***
Comment 8 Alex Schultz 2017-06-08 11:22:30 EDT
I tracked it down to the tempest-setup.sh script being used which does a sudo pip install tempestconfig. This caused a newer version of oslo.config to be installed which broke the services. 

From the tempest_output.log on the host...
2017-06-07 12:08:14 | + '[' -f sudo pip install tempestconf ']'
2017-06-07 12:08:14 | /home/stack/tempest-setup.sh: line 85: [: too many arguments
2017-06-07 12:08:14 | + sudo pip install tempestconf
2017-06-07 12:08:15 | Collecting tempestconf
2017-06-07 12:08:15 |   Downloading tempestconf-0.0.1.dev8.tar.gz
...snip...
2017-06-07 12:08:18 | Collecting oslo.config>=3.22.0 (from tempest>=14.0.0->tempestconf)
2017-06-07 12:08:18 |   Downloading oslo.config-4.3.0-py2.py3-none-any.whl (106kB)
...snip...
Comment 9 Alex Schultz 2017-06-08 11:51:19 EDT
Moving this over to python-tempestconfig as in IRC this was mentioned as missing for this version of OSP9 which is why they were pip installing tempestconf.  The recommendation would be to either provide the appropriate tempest items in package form or use a virtualenv if needing to pip install.
Comment 10 Martin Magr 2017-08-01 05:10:54 EDT
python-tempestconf has been added in OSP11 release and is available from this realease onwards. For OSP9 tempest conf is provided by tempest rpm.

Note You need to log in before you can comment on or make changes to this bug.