Bug 1324142

Summary: [RFE] Rollback of major upgrades in containerized fashion
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: RFEsAssignee: RHOS Maint <rhos-maint>
Status: CLOSED WONTFIX QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: augol, ccamacho, dbecker, jcoufal, jzaher, lbezdick, mandreou, markmc, mbracho, mbultel, mburns, mcornea, morazi, rhel-osp-director-maint, sasha, sathlang, srevivo
Target Milestone: ---Keywords: FutureFeature, Triaged, ZStream
Target Release: ---Flags: augol: needinfo? (mbracho)
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 11:36:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1384647    
Bug Blocks:    

Description Omri Hochman 2016-04-05 15:38:33 UTC 
[rhel-ospd][upgrade]: There is no rollback mechanism in case overcloud fails in upgrade from 7.3 to 8.0. 

Environment:
------------
instack-undercloud-2.2.7-2.el7ost.noarch
instack-0.0.8-2.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch


Description: 
------------
the upgrade procedure is very sensitive and can take hours, in case there's a failure due to any reason (environment, networking/power outage), there is no rollback mechanism for the overcloud. 

In order to attempt to overcome the upgrade failure, it will be required to debug each case individually, reach a solution before repeat the last upgrade step that resulted with failure.   

In case of failure the stack status will switch to: Stack overcloud UPDATE_FAILED.
Comment 2 Jaromir Coufal 2016-04-05 15:42:46 UTC 
In the meantime until this is addressed, we should start troubleshooting section in our docs on how get out of various upgrades failures and improve it over time.
Comment 3 Mike Burns 2016-04-05 15:50:11 UTC 
per comment 2, moving this to OSP 10.  We'll develop and maintain a troubleshooting section in product docs and/or kbase.
Comment 7 Red Hat Bugzilla Rules Engine 2017-03-13 05:21:57 UTC 
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 8 Red Hat Bugzilla Rules Engine 2017-03-13 05:22:24 UTC 
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 11 jzaher 2017-11-08 22:45:19 UTC 
Would this process be applicable to minor updates within a major version as well?  For example, would a user be able to use this RFE to rollback an upgrade to OSP13z2 to OSP13z1?
Comment 14 Lukas Bezdicka 2019-05-14 11:36:24 UTC 
Once the database is touched during major upgrade there is no other way than restoring from snapshot. We could refactor Director to use by default LVM and do snapshots/restores fast or there is already existing procedure for backup and restore.