Bug 1501237
| Summary: | OSP11 -> OSP12 upgrade: split stack upgrade times out after 4h while running major-upgrade-composable-steps-docker.yaml | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | ||||||||
| Component: | openstack-tripleo-heat-templates | Assignee: | James Slagle <jslagle> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> | ||||||||
| Severity: | urgent | Docs Contact: | |||||||||
| Priority: | urgent | ||||||||||
| Version: | 12.0 (Pike) | CC: | aschultz, dbecker, emacchi, jslagle, mandreou, mburns, mcornea, morazi, rhel-osp-director-maint, yprokule | ||||||||
| Target Milestone: | rc | Keywords: | Triaged | ||||||||
| Target Release: | 12.0 (Pike) | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | openstack-tripleo-heat-templates-7.0.3-0.20171024200823.el7ost | Doc Type: | If docs needed, set a value | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2017-12-13 22:13:59 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Marius Cornea
2017-10-12 09:10:00 UTC
Created attachment 1337688 [details]
openstack stack failures list overcloud --long
Created attachment 1337692 [details]
overcloud deploy and upgrade CLI args
just attached the overcloud stack fail list and the deploy/upgrade commands and assigning to myself for triage. I'd like to understand more about what is happening on that compute node. AFAICS:
* upgrade happened at Oct 11 ~ 20:08 (taking time from the boxes). I see ansible-pacemaker beign installed on the controller:
Oct 11 20:42:05 controller-0 yum[304143]: Installed: ansible-pacemaker-1.0.3-0.20170929170820.1279294.el7ost.noarch
* I don't see ansible-pacemaker being installed on the compute node, leading me to believe the upgrade init simply didn't happen there.
* I don't see any evidence of any ansible upgrade_tasks being executed anywhere even on the controller.
Most interesting issue I saw was problem reaching the undercloud VIP by os-collect-config which might point to some connectivity issue causing the hang we see ?
Oct 11 20:01:23 compute-0 os-collect-config: HTTPSConnectionPool(host='192.168.0.2', port=13808): Max retries exceeded with url: /v1/AUTH_6924b95b44284bb4b7a161c7136f5e6b/ov-edServer-vsvwpsxtavoz-deployed-server-u2vd6ruzdzpb/b957fc16-69d5-4251-9513-da1419ceeafc?temp_url_sig=ae9084afc12a9dea800aeda87e5afdf319754596&temp_url_expires=2147483586 (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x38ed850>, 'Connection to 192.168.0.2 timed out. (connect timeout=10.0)'))
...
Oct 11 20:08:33 compute-0 os-collect-config: ('Connection aborted.', BadStatusLine("''",))
Oct 11 20:08:33 compute-0 os-collect-config: Source [request] Unavailable.
...
Oct 11 20:09:33 compute-0 os-collect-config: 401 Client Error: Unauthorized for url: https://192.168.0.2:13808/v1/AUTH_6924b95b44284bb4b7a161c7136f5e6b/ov-edServer-vsvwpsxtavoz-deployed-server-u2vd6ruzdzpb/b957fc16-69d5-4251-9513-da1419ceeafc?temp_url_sig=ae9084afc12a9dea800aeda87e5afdf319754596&temp_url_expires=2147483586
Also on the controller
Oct 11 20:07:44 controller-0 os-collect-config: HTTPSConnectionPool(host='192.168.0.2', port=13808): Max retries exceeded with url: /v1/AUTH_6924b95b44284bb4b7a161c7136f5e6b/ov-edServer-q2xl3ve4y2ae-deployed-server-wi5mbmf6cqso/9187ee04-15df-4570-9f23-11d1f8e5946f?temp_url_sig=b352fa2d882de874462a84e83f82b91db2ff4753&temp_url_expires=2147483586 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x2a937d0>: Failed to establish a new connection: [Errno 113] No route to host',))
Oct 11 20:08:14 controller-0 os-collect-config: ('Connection aborted.', BadStatusLine("''",))
Oct 11 20:08:14 controller-0 os-collect-config: Source [request] Unavailable.
It seems the controller recovered from this however since the ansible-pacemaker was installed after that error so still not fully clear what the problem is.
We discussed this on upgrades scrum today too. Marius will try and see if it recreates easily. Given the time for OSP12 we thought we should also reach out to DFG:DF to see if anyone there has any ideas. In particular the os-c-c issue in comment 5 - could this be related to the fact that this is split-stack. Adding TC of DFG:DF for assignment thanks (Emilien mojo says its you sorry if it isn't ;) ) Marius, could you try again with https://review.openstack.org/#/c/511523/ ? It seems like some data were missing for deployed-server. Before I continue to investigate, I want to make sure you have the right parameters. Specially regarding network, Marios pointed out that os-collect-config wasn't able to reach Swift endpoint. Thanks Created attachment 1338181 [details]
marios_debug_13thOct17
just had another look marius I see the same issues as yesterday - os-collect-config can't connect. Attaching the debug info/notes for now thanks
After removing the deprecated parameters from deployed-server-roles-data.yaml(https://review.openstack.org/#/c/512343/) I was able to move forward with the upgrade. Moving assignment to slagle since he posted the review we are tracking here and moving to DF with upgrades secondary thanks Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |