Bug 1383839

Summary: rhel-osp-director: OC update fails right after starting.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED WORKSFORME QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: dbecker, jcoufal, jslagle, lbezdick, mandreou, mburns, morazi, rhel-osp-director-maint, sasha
Target Milestone: gaKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-25 15:34:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
heat logs from the undercloud none

Description Alexander Chuzhoy 2016-10-11 23:33:24 UTC 
rhel-osp-director:   OC update fails right after starting.
Environment:
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161003064637.d636e3a.1.1.el7ost.noarch
instack-undercloud-5.0.0-0.20160930175750.9d2a655.el7ost.noarch

Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log


2. Update the undercloud to latest.
3. Run "openstack undercloud upgrade"
4. Apply the necessary patches.
5. Make sure the latest repos are available on OC nodes.
6. Run:openstack overcloud update stack  -i overcloud


Result:

[stack@undercloud-0 ~]$ openstack overcloud update stack  -i overcloud                                                                                                                                               starting package update on stack overcloud
FAILED
update finished with status FAILED
Stack update failed.







[stack@undercloud-0 ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                         | resource_type                                                                                                       | resource_status | updated_time         | stack_name                                                                                                             |
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| ComputeServiceChain                       | 2ef89af8-8101-4fda-b452-c4aca9ac0491         | OS::TripleO::Services                                                                                               | UPDATE_FAILED   | 2016-10-11T23:10:39Z | overcloud                                                                                                              |
| CephStorageServiceChain                   | e40c397f-6bf4-455b-abe7-0e362efe81d2         | OS::TripleO::Services                                                                                               | UPDATE_FAILED   | 2016-10-11T23:10:44Z | overcloud                                                                                                              |
| ServiceChain                              | b3083ed9-fbe2-44cc-876d-d56fb49fff94         | OS::Heat::ResourceChain                                                                                             | UPDATE_FAILED   | 2016-10-11T23:10:44Z | overcloud-ComputeServiceChain-nzer7i4vw2yt                                                                             |
| ControllerServiceChain                    | f7d7ff41-71eb-430b-b080-5c1182682366         | OS::TripleO::Services                                                                                               | UPDATE_FAILED   | 2016-10-11T23:10:49Z | overcloud                                                                                                              |
| ServiceChain                              | bc716ddf-67c3-4943-bb25-5e12f5fecdff         | OS::Heat::ResourceChain                                                                                             | UPDATE_FAILED   | 2016-10-11T23:10:58Z | overcloud-ControllerServiceChain-nvr2r5npv2id                                                                          |
| 21                                        | ada7498d-3f2c-49af-b7e9-e61e6a66ad96         | OS::TripleO::Services::NeutronApi                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:15Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 26                                        | c731dd1d-4a14-47c5-961a-520d89c52dff         | OS::TripleO::Services::Keepalived                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:15Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 36                                        | 2534768c-28ee-4a64-bc20-3936235512f2         | OS::TripleO::Services::NovaVncProxy                                                                                 | UPDATE_FAILED   | 2016-10-11T23:11:16Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 49                                        | b794e869-1de2-43da-a73d-8113427f6e44         | OS::TripleO::Services::GnocchiApi                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:31Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 39                                        | 7852fd88-2500-4099-aca5-6745285f7d82         | OS::TripleO::Services::SwiftStorage                                                                                 | UPDATE_FAILED   | 2016-10-11T23:11:32Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 51                                        | 0b44c3f1-fb35-4b5a-b4cb-85a7f511f263         | OS::TripleO::Services::GnocchiStatsd                                                                                | UPDATE_FAILED   | 2016-10-11T23:11:32Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 13                                        | 0ab864db-4093-4fe9-8d5e-8f667ebc716f         | OS::TripleO::Services::HeatApi                                                                                      | UPDATE_FAILED   | 2016-10-11T23:11:33Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 58                                        | 16b9cada-7ca9-44dd-ab78-980c41f6cc2d         | OS::TripleO::Services::AodhApi                                                                                      | UPDATE_FAILED   | 2016-10-11T23:11:34Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 14                                        | a78fb874-9cc3-4779-a0f1-ee1a797b4942         | OS::TripleO::Services::HeatApiCfn                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:35Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 44                                        | 02a783a6-5dec-47c1-ba0d-2e6bb3ab777c         | OS::TripleO::Services::CeilometerCollector                                                                          | UPDATE_FAILED   | 2016-10-11T23:11:35Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 40                                        | e9d32646-cf9b-4fff-b1a2-eedc38f653a3         | OS::TripleO::Services::SwiftRingBuilder                                                                             | UPDATE_FAILED   | 2016-10-11T23:11:42Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 37                                        | 5f1362e3-1890-4613-8739-47994d1cb029         | OS::TripleO::Services::Ntp                                                                                          | UPDATE_FAILED   | 2016-10-11T23:11:43Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 61                                        | 3c5a4316-a6bf-42a0-b869-15abc11aa63b         | OS::TripleO::Services::AodhListener                                                                                 | UPDATE_FAILED   | 2016-10-11T23:11:43Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 20                                        | 56efe84d-bf56-4749-a925-906754882070         | OS::TripleO::Services::NeutronMetadataAgent                                                                         | UPDATE_FAILED   | 2016-10-11T23:11:44Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
Comment 1 Alexander Chuzhoy 2016-10-11 23:37:25 UTC 
Created attachment 1209368 [details]
heat logs from the undercloud
Comment 2 Alexander Chuzhoy 2016-10-11 23:38:17 UTC 
These patches were applied before starting the OC update:
https://review.openstack.org/#/c/375540/
https://review.openstack.org/#/c/383708/
https://review.openstack.org/#/c/379547/
https://review.openstack.org/#/c/381899/
Comment 3 Lukas Bezdicka 2016-10-12 10:51:15 UTC 
Sounds like issue on undercloud:
Stack UPDATE FAILED (overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6): RemoteError: resources[20]: Remote error: DBConnectionError (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
Comment 4 James Slagle 2016-10-18 19:50:48 UTC 
(In reply to Lukas Bezdicka from comment #3)
> Sounds like issue on undercloud:
> Stack UPDATE FAILED
> (overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6):
> RemoteError: resources[20]: Remote error: DBConnectionError
> (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server
> during query')

do you have any update, or an upstream patch available?
Comment 5 Lukas Bezdicka 2016-10-19 11:15:04 UTC 
I can't reproduce this and only thing I've seen from logs is db error on undercloud. I'd close this bug.
Comment 6 Marios Andreou 2016-10-20 06:15:28 UTC 
Sasha is this still a thing or can we close it as per comment 5?
Comment 7 Alexander Chuzhoy 2016-10-25 15:34:09 UTC 
This issue doesn't reproduce now.