1459694 – openstack-glance: openstack-glance-api and openstack-glance-registry services become failed after "openstack undercloud upgrade" completes duing minor update on OSP9

Bug 1459694 - openstack-glance: openstack-glance-api and openstack-glance-registry services become failed after "openstack undercloud upgrade" completes duing minor update on OSP9

Summary: openstack-glance: openstack-glance-api and openstack-glance-registry services...

Keywords:
Status:	CLOSED DUPLICATE of bug 1459691
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-glance
Sub Component:
Version:	9.0 (Mitaka)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	async
Target Release:	9.0 (Mitaka)
Assignee:	Cyril Roelandt
QA Contact:	Avi Avraham
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-07 20:46 UTC by Alexander Chuzhoy
Modified:	2018-05-02 10:50 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-06-08 15:15:08 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
logs from the undercloud (15.64 MB, application/x-xz) 2017-06-07 20:47 UTC, Alexander Chuzhoy	no flags	Details
View All

Description Alexander Chuzhoy 2017-06-07 20:46:43 UTC

openstack-glance: openstack-glance-api and openstack-glance-registry services become failed after "openstack undercloud upgrade" completes duing minor update on OSP9



Environment:
python-glance-12.0.0-2.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-54.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch
python-glanceclient-2.0.0-1.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-54.el7ost.noarch
openstack-puppet-modules-8.1.10-2.el7ost.noarch
python-glance-tests-12.0.0-2.el7ost.noarch
instack-undercloud-4.0.0-16.el7ost.noarch
openstack-glance-12.0.0-2.el7ost.noarch



Steps to reproduce:
1. Run "openstack undercloud upgrade"
2. Reboot the machine (if needed - new kernel)
3. run :

[root@director ~]# systemctl status openstack-glance-api.service
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
   Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:41 CDT; 1h 11min ago
  Process: 3719 ExecStart=/usr/bin/glance-api (code=exited, status=1/FAILURE)
 Main PID: 3719 (code=exited, status=1/FAILURE)

Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service: main process exited, code=exited, status=1/FAILURE
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service holdoff time over, scheduling restart.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-glance-api.service
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Failed to start OpenStack Image Service (code-named Glance) API server.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.



[root@director ~]# systemctl status openstack-glance-api.service
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
   Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:41 CDT; 1h 11min ago
  Process: 3719 ExecStart=/usr/bin/glance-api (code=exited, status=1/FAILURE)
 Main PID: 3719 (code=exited, status=1/FAILURE)

Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service: main process exited, code=exited, status=1/FAILURE
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service holdoff time over, scheduling restart.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-glance-api.service
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Failed to start OpenStack Image Service (code-named Glance) API server.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: Unit openstack-glance-api.service entered failed state.
Jun 07 14:33:41 director.fv1dci.org systemd[1]: openstack-glance-api.service failed.
[root@director ~]# 



[root@director ~]# journalctl -u openstack-glance-registry.service|grep -i error:
Jun 07 14:22:53 director.fv1dci.org glance-registry[29098]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:53 director.fv1dci.org glance-registry[29160]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:54 director.fv1dci.org glance-registry[29201]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:55 director.fv1dci.org glance-registry[29229]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:22:56 director.fv1dci.org glance-registry[29268]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:35 director.fv1dci.org glance-registry[1511]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:38 director.fv1dci.org glance-registry[2468]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:39 director.fv1dci.org glance-registry[2936]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:40 director.fv1dci.org glance-registry[3413]: ERROR: Unable to locate paste config file for glance-registry.
Jun 07 14:33:41 director.fv1dci.org glance-registry[4042]: ERROR: Unable to locate paste config file for glance-registry.




[root@director ~]# journalctl -u openstack-glance-api|grep -i error:
Jun 07 14:22:53 director.fv1dci.org glance-api[29111]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:54 director.fv1dci.org glance-api[29161]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:55 director.fv1dci.org glance-api[29206]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:55 director.fv1dci.org glance-api[29236]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:22:56 director.fv1dci.org glance-api[29275]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:35 director.fv1dci.org glance-api[1514]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:37 director.fv1dci.org glance-api[2402]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:39 director.fv1dci.org glance-api[2881]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:40 director.fv1dci.org glance-api[3104]: ERROR: Unable to locate paste config file for glance-api.
Jun 07 14:33:41 director.fv1dci.org glance-api[3719]: ERROR: Unable to locate paste config file for glance-api.

Comment 1 Alexander Chuzhoy 2017-06-07 20:47:46 UTC

Created attachment 1285914 [details]
logs from the undercloud

Comment 2 Cyril Roelandt 2017-06-08 11:21:12 UTC

So, do you have a paste config file for glance-api in /etc/glance ?

You may want to try and give the full path to this config file by editing /etc/glance/glance-api.conf:

[paste_deploy]
config_file = /path/to/the/paste/config/file

Comment 3 Alex Schultz 2017-06-08 15:15:08 UTC

I found the root cause for this host. See BZ1459691 for more details.  Someone pip installed a newer version of oslo.config which breaks our services.

*** This bug has been marked as a duplicate of bug 1459691 ***

Comment 4 Amit Ugol 2018-05-02 10:50:31 UTC

closed, no need for needinfo.

Note You need to log in before you can comment on or make changes to this bug.