Bug 1616347 - Undercloud deployment gets stuck after starting glance_api_db_sync
Summary: Undercloud deployment gets stuck after starting glance_api_db_sync
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-glance
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ga
: ---
Assignee: Cyril Roelandt
QA Contact: Mike Abrams
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-15 16:11 UTC by Alexander Chuzhoy
Modified: 2019-11-13 22:01 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-17 07:34:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
undercloud_install.log (1.01 MB, text/plain)
2018-08-15 19:39 UTC, Filip Hubík
no flags Details

Description Alexander Chuzhoy 2018-08-15 16:11:33 UTC
Undercloud deployment gets stuck after starting glance_api_db_sync

Environment:
puppet-glance-13.2.0-0.20180728150233.a65de83.el7ost.noarch
python2-glanceclient-2.11.1-0.20180608145253.1cf00a5.el7ost.noarch
instack-undercloud-9.2.1-0.20180803181448.be5fa97.el7ost.noarch


Steps to reproduce:
Try to deploy undercloud


Result:

the deployment gets stuck.
The last container that was started: glance_api_db_sync



Repeated warnings in logs:
DeprecationWarning,
/usr/lib/python2.7/site-packages/webob/acceptparse.py:1297: DeprecationWarning: The behavior of .best_match for the Accept classes is currently being maintained for backward compatibility, but the method will be deprecated in the future, as its behavior is not specified in (and currently does not conform to) RFC 7231.

Comment 3 Filip Hubík 2018-08-15 19:37:24 UTC
I can confirm this being hit in phase1 CI, it looks like undercloud installation is stuck indefinitely trying to instantiate containers. Last entries in installation log are (full log attached):

$ openstack undercloud install  &> undercloud_install.log

...
TASK [Run docker-puppet tasks (generate config) during step 3] ********************************************************************************************************************************************************************************
skipping: [undercloud-0]

TASK [Debug output for task which failed: Run docker-puppet tasks (generate config) during step 3] ********************************************************************************************************************************************
skipping: [undercloud-0]

TASK [Start containers for step 3] ************************************************************************************************************************************************************************************************************
# process is stuck

Comment 4 Filip Hubík 2018-08-15 19:39:32 UTC
Created attachment 1476287 [details]
undercloud_install.log

Comment 6 Christian Schwede (cschwede) 2018-08-16 06:06:12 UTC
I'm debugging on Filips env right now, and one thing that looks strange to me is an empty  /usr/local/bin/kolla_glance_extend_start in the glance container. However, it should look like this IMO: https://github.com/openstack/kolla/blob/master/docker/glance/glance-api/extend_start.sh

Comment 7 Christian Schwede (cschwede) 2018-08-16 07:03:43 UTC
I checked the last two container images locally, and the latest (openstack-glance-api:2018-08-15.1) is clearly missing /usr/local/bin/kolla_glance_extend_start.

The image from before (openstack-glance-api:2018-08-14.1) still contained that file.

Comment 8 Christian Schwede (cschwede) 2018-08-16 07:04:06 UTC
Well, to be more precise, the file is there, but empty.

Comment 10 Mike Theis 2019-02-12 14:19:18 UTC
I am having this exact issue yet I don't see any sort of code or fix to close this? Can someone kindly help me understand how to get past this as it still persists even on a fresh install I did earlier this morning. Thanks!

Comment 11 Salman Khan 2019-10-25 07:27:16 UTC
Hitting alike issue in Stein as well.. any clue?

Comment 12 Cyril Roelandt 2019-11-13 22:01:52 UTC
We would need the exact commands you typed, config files, logs.


Note You need to log in before you can comment on or make changes to this bug.