1383839 – rhel-osp-director: OC update fails right after starting.

Bug 1383839 - rhel-osp-director: OC update fails right after starting.

Summary: rhel-osp-director: OC update fails right after starting.

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	rhosp-director
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	ga
Target Release:	10.0 (Newton)
Assignee:	Lukas Bezdicka
QA Contact:	Omri Hochman
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-10-11 23:33 UTC by Alexander Chuzhoy
Modified:	2016-12-29 17:02 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-10-25 15:34:09 UTC
Target Upstream Version:

Attachments	(Terms of Use)
heat logs from the undercloud (240.19 KB, application/x-gzip) 2016-10-11 23:37 UTC, Alexander Chuzhoy	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Alexander Chuzhoy 2016-10-11 23:33:24 UTC

rhel-osp-director:   OC update fails right after starting.
Environment:
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161003064637.d636e3a.1.1.el7ost.noarch
instack-undercloud-5.0.0-0.20160930175750.9d2a655.el7ost.noarch

Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log


2. Update the undercloud to latest.
3. Run "openstack undercloud upgrade"
4. Apply the necessary patches.
5. Make sure the latest repos are available on OC nodes.
6. Run:openstack overcloud update stack  -i overcloud


Result:

[stack@undercloud-0 ~]$ openstack overcloud update stack  -i overcloud                                                                                                                                               starting package update on stack overcloud
FAILED
update finished with status FAILED
Stack update failed.







[stack@undercloud-0 ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                         | resource_type                                                                                                       | resource_status | updated_time         | stack_name                                                                                                             |
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| ComputeServiceChain                       | 2ef89af8-8101-4fda-b452-c4aca9ac0491         | OS::TripleO::Services                                                                                               | UPDATE_FAILED   | 2016-10-11T23:10:39Z | overcloud                                                                                                              |
| CephStorageServiceChain                   | e40c397f-6bf4-455b-abe7-0e362efe81d2         | OS::TripleO::Services                                                                                               | UPDATE_FAILED   | 2016-10-11T23:10:44Z | overcloud                                                                                                              |
| ServiceChain                              | b3083ed9-fbe2-44cc-876d-d56fb49fff94         | OS::Heat::ResourceChain                                                                                             | UPDATE_FAILED   | 2016-10-11T23:10:44Z | overcloud-ComputeServiceChain-nzer7i4vw2yt                                                                             |
| ControllerServiceChain                    | f7d7ff41-71eb-430b-b080-5c1182682366         | OS::TripleO::Services                                                                                               | UPDATE_FAILED   | 2016-10-11T23:10:49Z | overcloud                                                                                                              |
| ServiceChain                              | bc716ddf-67c3-4943-bb25-5e12f5fecdff         | OS::Heat::ResourceChain                                                                                             | UPDATE_FAILED   | 2016-10-11T23:10:58Z | overcloud-ControllerServiceChain-nvr2r5npv2id                                                                          |
| 21                                        | ada7498d-3f2c-49af-b7e9-e61e6a66ad96         | OS::TripleO::Services::NeutronApi                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:15Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 26                                        | c731dd1d-4a14-47c5-961a-520d89c52dff         | OS::TripleO::Services::Keepalived                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:15Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 36                                        | 2534768c-28ee-4a64-bc20-3936235512f2         | OS::TripleO::Services::NovaVncProxy                                                                                 | UPDATE_FAILED   | 2016-10-11T23:11:16Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 49                                        | b794e869-1de2-43da-a73d-8113427f6e44         | OS::TripleO::Services::GnocchiApi                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:31Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 39                                        | 7852fd88-2500-4099-aca5-6745285f7d82         | OS::TripleO::Services::SwiftStorage                                                                                 | UPDATE_FAILED   | 2016-10-11T23:11:32Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 51                                        | 0b44c3f1-fb35-4b5a-b4cb-85a7f511f263         | OS::TripleO::Services::GnocchiStatsd                                                                                | UPDATE_FAILED   | 2016-10-11T23:11:32Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 13                                        | 0ab864db-4093-4fe9-8d5e-8f667ebc716f         | OS::TripleO::Services::HeatApi                                                                                      | UPDATE_FAILED   | 2016-10-11T23:11:33Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 58                                        | 16b9cada-7ca9-44dd-ab78-980c41f6cc2d         | OS::TripleO::Services::AodhApi                                                                                      | UPDATE_FAILED   | 2016-10-11T23:11:34Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 14                                        | a78fb874-9cc3-4779-a0f1-ee1a797b4942         | OS::TripleO::Services::HeatApiCfn                                                                                   | UPDATE_FAILED   | 2016-10-11T23:11:35Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 44                                        | 02a783a6-5dec-47c1-ba0d-2e6bb3ab777c         | OS::TripleO::Services::CeilometerCollector                                                                          | UPDATE_FAILED   | 2016-10-11T23:11:35Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 40                                        | e9d32646-cf9b-4fff-b1a2-eedc38f653a3         | OS::TripleO::Services::SwiftRingBuilder                                                                             | UPDATE_FAILED   | 2016-10-11T23:11:42Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 37                                        | 5f1362e3-1890-4613-8739-47994d1cb029         | OS::TripleO::Services::Ntp                                                                                          | UPDATE_FAILED   | 2016-10-11T23:11:43Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 61                                        | 3c5a4316-a6bf-42a0-b869-15abc11aa63b         | OS::TripleO::Services::AodhListener                                                                                 | UPDATE_FAILED   | 2016-10-11T23:11:43Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
| 20                                        | 56efe84d-bf56-4749-a925-906754882070         | OS::TripleO::Services::NeutronMetadataAgent                                                                         | UPDATE_FAILED   | 2016-10-11T23:11:44Z | overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6                                                |
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+

Comment 1 Alexander Chuzhoy 2016-10-11 23:37:25 UTC

Created attachment 1209368 [details]
heat logs from the undercloud

Comment 2 Alexander Chuzhoy 2016-10-11 23:38:17 UTC

These patches were applied before starting the OC update:
https://review.openstack.org/#/c/375540/
https://review.openstack.org/#/c/383708/
https://review.openstack.org/#/c/379547/
https://review.openstack.org/#/c/381899/

Comment 3 Lukas Bezdicka 2016-10-12 10:51:15 UTC

Sounds like issue on undercloud:
Stack UPDATE FAILED (overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6): RemoteError: resources[20]: Remote error: DBConnectionError (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')

Comment 4 James Slagle 2016-10-18 19:50:48 UTC

(In reply to Lukas Bezdicka from comment #3)
> Sounds like issue on undercloud:
> Stack UPDATE FAILED
> (overcloud-ControllerServiceChain-nvr2r5npv2id-ServiceChain-2awdifsterm6):
> RemoteError: resources[20]: Remote error: DBConnectionError
> (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server
> during query')

do you have any update, or an upstream patch available?

Comment 5 Lukas Bezdicka 2016-10-19 11:15:04 UTC

I can't reproduce this and only thing I've seen from logs is db error on undercloud. I'd close this bug.

Comment 6 Marios Andreou 2016-10-20 06:15:28 UTC

Sasha is this still a thing or can we close it as per comment 5?

Comment 7 Alexander Chuzhoy 2016-10-25 15:34:09 UTC

This issue doesn't reproduce now.

Note You need to log in before you can comment on or make changes to this bug.