Description of problem:
OSP11 -> OSP12 upgrade: split stack upgrade times out after 4h while running major-upgrade-composable-steps-docker.yaml.
Note: before running major-upgrade-composable-steps-docker.yaml I worked around BZ#1500832 by adjusting the roles_data and setting disable_upgrade_deployment flag for the compute role. From what I see this doesn't seem to behave well with split stack deployments as the stack update for the compute role just gets stuck and times out after 4h:
2017-10-11 20:42:33Z [AllNodesValidationConfig]: UPDATE_COMPLETE state changed
2017-10-12 00:36:20Z [ComputeDeployedServer]: UPDATE_FAILED UPDATE aborted
2017-10-12 00:36:20Z [overcloud]: UPDATE_FAILED Timed out
Stack overcloud UPDATE_FAILED
Heat Stack update failed.
Heat Stack update failed.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Deploy split stack OSP11
2. Workaround BZ#1500832 by setting disable_upgrade_deployment to true for compute role in deployed-server/deployed-server-roles-data.yaml
3. Run major-upgrade-composable-steps-docker.yaml to upgrade to OSP12
Upgrade gets stuck and times out after 4h
Upgrade completes fine.
Created attachment 1337688 [details]
openstack stack failures list overcloud --long
Created attachment 1337692 [details]
overcloud deploy and upgrade CLI args
just attached the overcloud stack fail list and the deploy/upgrade commands and assigning to myself for triage. I'd like to understand more about what is happening on that compute node.
* upgrade happened at Oct 11 ~ 20:08 (taking time from the boxes). I see ansible-pacemaker beign installed on the controller:
Oct 11 20:42:05 controller-0 yum: Installed: ansible-pacemaker-1.0.3-0.20170929170820.1279294.el7ost.noarch
* I don't see ansible-pacemaker being installed on the compute node, leading me to believe the upgrade init simply didn't happen there.
* I don't see any evidence of any ansible upgrade_tasks being executed anywhere even on the controller.
Most interesting issue I saw was problem reaching the undercloud VIP by os-collect-config which might point to some connectivity issue causing the hang we see ?
Oct 11 20:01:23 compute-0 os-collect-config: HTTPSConnectionPool(host='192.168.0.2', port=13808): Max retries exceeded with url: /v1/AUTH_6924b95b44284bb4b7a161c7136f5e6b/ov-edServer-vsvwpsxtavoz-deployed-server-u2vd6ruzdzpb/b957fc16-69d5-4251-9513-da1419ceeafc?temp_url_sig=ae9084afc12a9dea800aeda87e5afdf319754596&temp_url_expires=2147483586 (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x38ed850>, 'Connection to 192.168.0.2 timed out. (connect timeout=10.0)'))
Oct 11 20:08:33 compute-0 os-collect-config: ('Connection aborted.', BadStatusLine("''",))
Oct 11 20:08:33 compute-0 os-collect-config: Source [request] Unavailable.
Oct 11 20:09:33 compute-0 os-collect-config: 401 Client Error: Unauthorized for url: https://192.168.0.2:13808/v1/AUTH_6924b95b44284bb4b7a161c7136f5e6b/ov-edServer-vsvwpsxtavoz-deployed-server-u2vd6ruzdzpb/b957fc16-69d5-4251-9513-da1419ceeafc?temp_url_sig=ae9084afc12a9dea800aeda87e5afdf319754596&temp_url_expires=2147483586
Also on the controller
Oct 11 20:07:44 controller-0 os-collect-config: HTTPSConnectionPool(host='192.168.0.2', port=13808): Max retries exceeded with url: /v1/AUTH_6924b95b44284bb4b7a161c7136f5e6b/ov-edServer-q2xl3ve4y2ae-deployed-server-wi5mbmf6cqso/9187ee04-15df-4570-9f23-11d1f8e5946f?temp_url_sig=b352fa2d882de874462a84e83f82b91db2ff4753&temp_url_expires=2147483586 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x2a937d0>: Failed to establish a new connection: [Errno 113] No route to host',))
Oct 11 20:08:14 controller-0 os-collect-config: ('Connection aborted.', BadStatusLine("''",))
Oct 11 20:08:14 controller-0 os-collect-config: Source [request] Unavailable.
It seems the controller recovered from this however since the ansible-pacemaker was installed after that error so still not fully clear what the problem is.
We discussed this on upgrades scrum today too. Marius will try and see if it recreates easily. Given the time for OSP12 we thought we should also reach out to DFG:DF to see if anyone there has any ideas. In particular the os-c-c issue in comment 5 - could this be related to the fact that this is split-stack. Adding TC of DFG:DF for assignment thanks (Emilien mojo says its you sorry if it isn't ;) )
Marius, could you try again with https://review.openstack.org/#/c/511523/ ? It seems like some data were missing for deployed-server. Before I continue to investigate, I want to make sure you have the right parameters.
Specially regarding network, Marios pointed out that os-collect-config wasn't able to reach Swift endpoint.
Created attachment 1338181 [details]
just had another look marius I see the same issues as yesterday - os-collect-config can't connect. Attaching the debug info/notes for now thanks
After removing the deprecated parameters from deployed-server-roles-data.yaml(https://review.openstack.org/#/c/512343/) I was able to move forward with the upgrade.
Moving assignment to slagle since he posted the review we are tracking here and moving to DF with upgrades secondary thanks
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.