Bug 1459694 - openstack-glance: openstack-glance-api and openstack-glance-registry services become failed after "openstack undercloud upgrade" completes duing minor update on OSP9 [NEEDINFO]
openstack-glance: openstack-glance-api and openstack-glance-registry services...
Status: CLOSED DUPLICATE of bug 1459691
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-glance (Show other bugs)
9.0 (Mitaka)
Unspecified Unspecified
high Severity unspecified
: async
: 9.0 (Mitaka)
Assigned To: Cyril Roelandt
Avi Avraham
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-07 16:46 EDT by Alexander Chuzhoy
Modified: 2017-06-08 11:15 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-08 11:15:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
cyril: needinfo? (sasha)


Attachments (Terms of Use)
logs from the undercloud (15.64 MB, application/x-xz)
2017-06-07 16:47 EDT, Alexander Chuzhoy
no flags Details

  None (edit)
Description Alexander Chuzhoy 2017-06-07 16:46:43 EDT
openstack-glance: openstack-glance-api and openstack-glance-registry services become failed after "openstack undercloud upgrade" completes duing minor update on OSP9



Environment:
python-glance-12.0.0-2.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-54.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch
python-glanceclient-2.0.0-1.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-54.el7ost.noarch
openstack-puppet-modules-8.1.10-2.el7ost.noarch
python-glance-tests-12.0.0-2.el7ost.noarch
instack-undercloud-4.0.0-16.el7ost.noarch
openstack-glance-12.0.0-2.el7ost.noarch



Steps to reproduce:
1. Run "openstack undercloud upgrade"
2. Reboot the machine (if needed - new kernel)
3. run :

[root@director ~]# systemctl status openstack-glance-api.service
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
   Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:41 CDT; 1h 11min ago
  Process: 3719 ExecStart=/usr/bin/glance-api (code=exited, status=1/FAILURE)
 Main PID: 3719 (code=exited, status=1/FAILURE)

Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service: main process exited, code=exited, status=1/FAILURE
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service holdoff time over, scheduling restart.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-glance-api.service
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Failed to start OpenStack Image Service (code-named Glance) API server.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.



[root@director ~]# systemctl status openstack-glance-api.service
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
   Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:41 CDT; 1h 11min ago
  Process: 3719 ExecStart=/usr/bin/glance-api (code=exited, status=1/FAILURE)
 Main PID: 3719 (code=exited, status=1/FAILURE)

Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service: main process exited, code=exited, status=1/FAILURE
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service holdoff time over, scheduling restart.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-glance-api.service
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Failed to start OpenStack Image Service (code-named Glance) API server.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.
[root@director ~]# 



[root@director ~]# journalctl -u openstack-glance-registry.service|grep -i error:
Jun 07 14:22:53 director.fv1dci.org glance-registry[29098]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:53 director.fv1dci.org glance-registry[29160]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:54 director.fv1dci.org glance-registry[29201]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:55 director.fv1dci.org glance-registry[29229]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:56 director.fv1dci.org glance-registry[29268]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:35 director.fv1dci.org glance-registry[1511]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:38 director.fv1dci.org glance-registry[2468]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:39 director.fv1dci.org glance-registry[2936]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:40 director.fv1dci.org glance-registry[3413]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:41 director.fv1dci.org glance-registry[4042]: ERROR: Unable to locate paste config file for glance-registry.




[root@director ~]# journalctl -u openstack-glance-api|grep -i error:
Jun 07 14:22:53 director.fv1dci.org glance-api[29111]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:54 director.fv1dci.org glance-api[29161]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:55 director.fv1dci.org glance-api[29206]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:55 director.fv1dci.org glance-api[29236]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:56 director.fv1dci.org glance-api[29275]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:35 director.fv1dci.org glance-api[1514]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:37 director.fv1dci.org glance-api[2402]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:39 director.fv1dci.org glance-api[2881]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:40 director.fv1dci.org glance-api[3104]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:41 director.fv1dci.org glance-api[3719]: ERROR: Unable to locate paste config file for glance-api.
Comment 1 Alexander Chuzhoy 2017-06-07 16:47 EDT
Created attachment 1285914 [details]
logs from the undercloud
Comment 2 Cyril Roelandt 2017-06-08 07:21:12 EDT
So, do you have a paste config file for glance-api in /etc/glance ?

You may want to try and give the full path to this config file by editing /etc/glance/glance-api.conf:

[paste_deploy]
config_file = /path/to/the/paste/config/file
Comment 3 Alex Schultz 2017-06-08 11:15:08 EDT
I found the root cause for this host. See BZ1459691 for more details.  Someone pip installed a newer version of oslo.config which breaks our services.

*** This bug has been marked as a duplicate of bug 1459691 ***

Note You need to log in before you can comment on or make changes to this bug.