Bug 1639255

Summary: Upgrade converge failed: TemplateResource BlockStorageServiceChain Timed out
Product: Red Hat OpenStack Reporter: Dariusz Wojewódzki <dwojewod>
Component: openstack-heatAssignee: Zane Bitter <zbitter>
Status: CLOSED ERRATA QA Contact: Ronnie Rasouli <rrasouli>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: emacchi, jjoyce, jschluet, jstransk, mburns, ramishra, sbaker, shardy, slinaber, tvignaud
Target Milestone: z5Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-heat-10.0.2-4.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-14 13:50:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dariusz Wojewódzki 2018-10-15 12:00:17 UTC
Description of problem:
-----------------------
Overcloud upgrade converge failed:

openstack overcloud upgrade converge --templates /home/stack/templates/dynamic-cloud_origin \
-e ./firstboot.yaml \
-e ./network-isolation.yaml \
-e ./network-environment.yaml \
-e ./scheduler-hints_env.yaml \
-e ./puppet-ceph-external.yaml \
-e ./extraconfig.yaml \
-e ./predeploy.yaml \
-e ./dynamicCloud.yaml \
-e ./enable-tls.yaml \
-e ./inject-trust-anchor-hiera.yaml \
-e ./keystone_domain_specific_ldap_backend.yaml \
-e ./tls-endpoints-public-dns.yaml \
-e ./cloudname.yaml \
-e ./netapp.yaml \
-e ./rhel-registration/environment-rhel-registration.yaml \
-e ./rhel-registration/rhel-registration-resource-registry.yaml \
-r ./roles_data-no.network.yaml \
-e ./node-count-flavor.yaml \
-e ./overcloud_images.yaml \
-e ./ntp-server.yaml \
-e ./postconfig.yaml \
--libvirt-type kvm


fails with the following error (only last lines shown, next I will attach updated sosreport files ):

[.......]

2018-10-12 14:44:54Z [overcloud-ControllerServiceChain-3sajdwh4eqr7-ServiceChain-36xd4p573nvw]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-10-12 14:44:55Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.ServiceChain]: UPDATE_COMPLETE  state changed
2018-10-12 14:45:00Z [overcloud-NetworkerServiceChain-43o4lcig2rez]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-10-12 14:45:00Z [NetworkerServiceChain]: UPDATE_COMPLETE  state changed
2018-10-12 14:45:02Z [NetworkerServiceChainRoleData]: UPDATE_IN_PROGRESS  state changed
2018-10-12 14:45:02Z [NetworkerServiceChainRoleData]: UPDATE_COMPLETE  state changed
2018-10-12 14:45:24Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.PuppetConfig]: UPDATE_IN_PROGRE
2018-10-12 14:45:25Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.PuppetConfig]: UPDATE_COMPLETE 
2018-10-12 14:45:26Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.UpgradeTasks]: UPDATE_IN_PROGRE
2018-10-12 14:45:27Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.UpgradeTasks]: UPDATE_COMPLETE 
2018-10-12 14:45:30Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.UpdateTasks]: UPDATE_IN_PROGRES
2018-10-12 14:45:30Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.DockerConfig]: UPDATE_IN_PROGRE
2018-10-12 14:45:31Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.DockerConfig]: UPDATE_COMPLETE 
2018-10-12 14:45:32Z [overcloud-ControllerServiceChain-3sajdwh4eqr7.UpdateTasks]: UPDATE_COMPLETE  
2018-10-12 14:45:32Z [overcloud-ControllerServiceChain-3sajdwh4eqr7]: UPDATE_COMPLETE  Stack UPDATE
2018-10-12 14:45:32Z [ControllerServiceChain]: UPDATE_COMPLETE  state changed
2018-10-12 14:45:34Z [ControllerServiceChainRoleData]: UPDATE_IN_PROGRESS  state changed
2018-10-12 14:45:34Z [ControllerServiceChainRoleData]: UPDATE_COMPLETE  state changed


2018-10-12 18:34:44Z [BlockStorageServiceChain]: UPDATE_FAILED  UPDATE aborted (Task update from Te-5a36-4403-be4d-36fb1c19c8e7] Stack "overcloud" [a74eef8a-65ea-47c3-9c67-79a8dfc7bfb5] Timed out)
2018-10-12 18:34:44Z [overcloud-BlockStorageServiceChain-e66gp5qijhsp]: UPDATE_FAILED  Stack UPDATE
2018-10-12 18:34:45Z [overcloud]: UPDATE_FAILED  Timed out

 Stack overcloud UPDATE_FAILED 

overcloud.BlockStorageServiceChain:
  resource_type: OS::TripleO::Services
  physical_resource_id: 87859b8d-5a36-4403-be4d-36fb1c19c8e7
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted (Task update from TemplateResource "BlockStorageServiceChain" [87859b8d-5a36-440ea-47c3-9c67-79a8dfc7bfb5] Timed out)
Heat Stack update failed.
Heat Stack update failed.
Script done, file is openstack_overcloud_upgrade_converge_2018.10.12_erster_anlauf.txt


Version-Release number of selected component (if applicable):
-------------------------------------------------------------



openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
openstack-heat-api-10.0.1-2.el7ost.noarch                   
openstack-heat-api-cfn-10.0.1-2.el7ost.noarch               
openstack-heat-common-10.0.1-2.el7ost.noarch                
openstack-heat-engine-10.0.1-2.el7ost.noarch                
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch        
openstack-tripleo-common-8.6.3-13.el7ost.noarch             
openstack-tripleo-common-containers-8.6.3-13.el7ost.noarch  
openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch     
openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch      
openstack-tripleo-puppet-elements-8.0.1-1.el7ost.noarch     
openstack-tripleo-ui-8.3.2-1.el7ost.noarch                  
openstack-tripleo-validations-8.4.2-1.el7ost.noarch         
puppet-cinder-12.4.1-0.20180628102250.641e036.el7ost.noarch 
puppet-heat-12.4.1-0.20180416203421.90e3fb0.el7ost.noarch   
puppet-tripleo-8.3.4-5.el7ost.noarch                       
python-heat-agent-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-heat-agent-ansible-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-heat-agent-apply-config-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-heat-agent-docker-cmd-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-heat-agent-hiera-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-heat-agent-json-file-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-heat-agent-puppet-1.5.4-0.20180308153305.ecf43c7.el7ost.noarch 
python-tripleoclient-9.2.3-4.el7ost.noarch                  
python2-cinderclient-3.5.0-1.el7ost.noarch                  
python2-heatclient-1.14.0-1.el7ost.noarch                   


Actual results:
---------------
Upgrade fails on converge step

Comment 2 Dariusz Wojewódzki 2018-10-15 15:00:04 UTC
Customer rebooted the director and both controllers. By retrying the same command again it went through successfully.

Comment 13 errata-xmlrpc 2019-03-14 13:50:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0563