for OSP13 the major upgrade converge step (traditionally this has always been the last step in the major upgrade workflow) is no longer running a heat stack update. All configuration has already been applied across all nodes during the upgrade, and we _prepare_ the heat stack to make it suitable for future stack updates, but no longer actually invoke heat during the upgrade converge.
openstack overcloud upgrade converge --$ARGS
will not run a heat stack update. This BZ is to track a docs "recommendation" that operators *should* follow a successful upgrade converge with a full "overcloud deploy", and include all the environment files that the operator has already assembled and used for the earlier "openstack overcloud upgrade prepare --$ARGS" and also for the "openstack overcloud upgrade converge --$ARGS step). See  for more info.
Also worth noting that for 13, the 3 upgrades related clis:
openstack overcloud upgrade converge --$ARGS
openstack overcloud update converge --$ARGS
openstack overcloud ffwd-upgrade converge --$ARGS
are consistent in *not* running a heat stack update so this recommendation applies to all (not sure you want cloned bugs for that though).
I think this can be tracked in a single BZ, with changes over those 3 CLIs.
Dan what do you think about this?
Agreed. This can be one BZ.
Also, if I understand the upgrade process correctly, wouldn't this be less of a recommendation and more of mandatory step?
o/ Dan. Well yes and no. It is no longer mandatory, in the sense that since OSP12 when you run the upgrade on a particular Role (e.g. upgrade your controllers, or your computes) the given upgrade step (for OSP12, this was like openstack overcloud deploy -e environments/major-upgrade-composable-steps-docker.yaml for controllers and then upgrade-non-controller.sh for computes) will fully upgrade then apply config for the target version (13 in this case).
One of the reasons the converge was so important in the past is it was the first step at which we were applying the target version configuration across all nodes. We are now doing this at each step. The final converge is a nice 'sanity check' that the upgrade was successful, since it should be a 'noop' ... you've already applied all the config so it should complete successfully.
Another nuance is, in the OSP13 upgrade workflow, there is only one stack update, during the first step, openstack overcloud upgrade prepare. The actual upgrade and deployment config is all handled with ansible playbooks. So it would be good to assert that you can still use Heat to successfully apply config across your deployment, with a final "openstack overcloud deploy -e " (... CRUCIAL that the -e are included in all commands for 13 as an aside). Any subsequent stack updates, e.g. for scaling up/down or for replacing a node etc, will still be using Heat.
We had a specific reason (ceph) to remove the stack update from the upgrade converge (see https://review.openstack.org/#/c/562205/ for some discussion if interested/useful for docs) and otherwise I think we would have kept it, imo at least. Basically running this final stack update will guard you against any unexpected issues (related to the upgrade, but which have gone unnoticed for $reasons) trying to run a heat stack update against the overcloud.
hope it helps, thanks!
Thanks for the detailed answer, Marios! Based on your reasoning, we'll treat it as an actual part of the FFU procedure but more as a final verification step. How does that sound?
> Thanks for the detailed answer, Marios! Based on your reasoning, we'll treat
> it as an actual part of the FFU procedure but more as a final verification
> step. How does that sound?
Yes that sounds fine to me. I filed this BZ with the major upgrade in mind but indeed it applies to all three OSP13 clis 'upgrade' 'update' and 'ffwd-upgrade'. Please note that for FFU specifically (since you called it out here) it may be a little more complicated (we should discuss this with PM and members of the Upgrades DFG) because of maintenance window requirements.. I guess it will be easier/more efficient to discuss that once you have something posted for us to review.
Dan please hold on this as it looks like we are re-adding the stack updates on converge for all of update/upgrade/ffwd-upgrade - needinfo just to make sure you see it and don't spend too much time doing this hopefully. I'll update in a sec once I file a BZ explaining why fwiw. thanks
ACK, thanks for letting me know
Marios, since the stack update is included in the converge commands, can we close this BZ?
yes thanks for checking Dan.. as per the discussion in the comments above, the final OSP 13 upgrade/update/ffwd-upgrade workflow includes a stack update so this bug is indeed invalid closing