Bug 1570042

Summary: OSP13 docs please add recommendation for full "openstack overcloud deploy" after converge.
Product: Red Hat OpenStack Reporter: Marios Andreou <mandreou>
Component: documentationAssignee: Dan Macpherson <dmacpher>
Status: CLOSED NOTABUG QA Contact: RHOS Documentation Team <rhos-docs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: ccamacho, dmacpher, mandreou, mcornea, srevivo, yprokule
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-13 06:50:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marios Andreou 2018-04-20 13:41:41 UTC
o/ Hi, 

for OSP13 the major upgrade converge step (traditionally this has always been the last step in the major upgrade workflow) is no longer running a heat stack update. All configuration has already been applied across all nodes during the upgrade, and we _prepare_ the heat stack to make it suitable for future stack updates, but no longer actually invoke heat during the upgrade converge.

For clarity:

    openstack overcloud upgrade converge --$ARGS
    
will not run a heat stack update. This BZ is to track a docs "recommendation" that operators *should* follow a successful upgrade converge with a full "overcloud deploy", and include all the environment files that the operator has already assembled and used for the earlier "openstack overcloud upgrade prepare --$ARGS" and also for the "openstack overcloud upgrade converge --$ARGS step). See [1][2] for more info.

Also worth noting that for 13, the 3 upgrades related clis:

    openstack overcloud upgrade converge --$ARGS
    openstack overcloud update converge --$ARGS
    openstack overcloud ffwd-upgrade converge --$ARGS
    
are consistent in *not* running a heat stack update so this recommendation applies to all (not sure you want cloned bugs for that though).  

thank you    

[1] https://gitlab.cee.redhat.com/mandreou/OSP12-OSP13-Upgrade/commit/9978d21b1b205316ab22ada2bce2b781d70dc65b
[2] https://gitlab.cee.redhat.com/mandreou/OSP12-OSP13-Upgrade/blob/master/README.md

Comment 1 Carlos Camacho 2018-04-20 13:56:27 UTC
I think this can be tracked in a single BZ, with changes over those 3 CLIs.

Dan what do you think about this?

Comment 2 Dan Macpherson 2018-04-20 15:18:33 UTC
Agreed. This can be one BZ.

Comment 3 Dan Macpherson 2018-04-20 16:41:54 UTC
Also, if I understand the upgrade process correctly, wouldn't this be less of a recommendation and more of mandatory step?

Comment 4 Marios Andreou 2018-04-23 07:54:12 UTC
o/ Dan. Well yes and no. It is no longer mandatory, in the sense that since OSP12 when you run the upgrade on a particular Role (e.g. upgrade your controllers, or your computes) the given upgrade step (for OSP12, this was like openstack overcloud deploy -e environments/major-upgrade-composable-steps-docker.yaml   for controllers and then upgrade-non-controller.sh for computes) will fully upgrade then apply config for the target version (13 in this case).

One of the reasons the converge was so important in the past is it was the first step at which we were applying the target version configuration across all nodes. We are now doing this at each step. The final converge is a nice 'sanity check' that the upgrade was successful, since it should be a 'noop' ... you've already applied all the config so it should complete successfully. 

Another nuance is, in the OSP13 upgrade workflow, there is only one stack update, during the first step, openstack overcloud upgrade prepare. The actual upgrade and deployment config is all handled with ansible playbooks. So it would be good to assert that you can still use Heat to successfully apply config across your deployment, with a final "openstack overcloud deploy -e " (... CRUCIAL that the -e are included in all commands for 13 as an aside). Any subsequent stack updates, e.g. for scaling up/down or for replacing a node etc, will still be using Heat.

We had a specific reason (ceph) to remove the stack update from the upgrade converge (see https://review.openstack.org/#/c/562205/ for some discussion if interested/useful for docs) and otherwise I think we would have kept it, imo at least. Basically running this final stack update will guard you against any unexpected issues (related to the upgrade, but which have gone unnoticed for $reasons) trying to run a heat stack update against the overcloud.

hope it helps, thanks!

Comment 5 Dan Macpherson 2018-04-24 17:30:14 UTC
Thanks for the detailed answer, Marios! Based on your reasoning, we'll treat it as an actual part of the FFU procedure but more as a final verification step. How does that sound?

Comment 6 Marios Andreou 2018-04-25 06:06:17 UTC
> Thanks for the detailed answer, Marios! Based on your reasoning, we'll treat
> it as an actual part of the FFU procedure but more as a final verification
> step. How does that sound?

Yes that sounds fine to me. I filed this BZ with the major upgrade in mind but indeed it applies to all three OSP13 clis 'upgrade' 'update' and 'ffwd-upgrade'. Please note that for FFU specifically (since you called it out here) it may be a little more complicated (we should discuss this with PM and members of the Upgrades DFG) because of maintenance window requirements.. I guess it will be easier/more efficient to discuss that once you have something posted for us to review.

Comment 7 Marios Andreou 2018-05-02 13:12:59 UTC
Dan please hold on this as it looks like we are re-adding the stack updates on converge for all of update/upgrade/ffwd-upgrade - needinfo just to make sure you see it and don't spend too much time doing this hopefully. I'll update in a sec once I file a BZ explaining why fwiw. thanks

Comment 9 Dan Macpherson 2018-05-03 01:41:01 UTC
ACK, thanks for letting me know

Comment 12 Dan Macpherson 2018-08-10 06:19:08 UTC
Marios, since the stack update is included in the converge commands, can we close this BZ?

Comment 13 Marios Andreou 2018-08-13 06:50:49 UTC
yes thanks for checking Dan.. as per the discussion in the comments above, the final OSP 13 upgrade/update/ffwd-upgrade workflow includes a stack update so this bug is indeed invalid closing