Bug 1413199

We cannot specify required RHEL version because RHEL release is not synced with OpenStack releases. Our docs are pointing to OpenStack release and we have official support statements for what OSP versions are supported on what RHEL releases.

For concrete example OSP8 is supported on RHEL 7.2 as well as RHEL 7.3. So we have customers running on RHEL 7.2 since they deployed in the time when 7.2 was out and we have customer who are running on RHEL 7.3 since they updated to latest 8.

The instructions in the docs should state to update repos as needed but I don't think we should say to update to RHEL 7.2 or 7.3 repos since this changes over time. As long as we say the latest this should be fine.

We have repo requirements listed, what specifically is missing there?

Hi.

The issue that I ran into was that I was pulling packages from Satellite, not CDN. Using a frozen RHEL repo (7.2) and OSP9 (latest) we ran into dependency issues that we could not resolve without updating RHEL repo to latest as well.

So while there may not be a hard dependency between RHEL 7 and OSP 8/9, these two repos are somewhat linked as OSP repos do have requirements from rhel repos.

Reading through the documentation, it was not immediately clear at the time that the documentation intended us to update to the latest CDN repos, not just to the latest in our local satellite channels. 

All this could have been more clear, and might save others from the issues that we faced initially.

That being said. I do not know exactly how we would communicate this in the prereqs section of the doc. 

Does this help at all?

Hi Chris,

I think it helps.

The problem here are the moving parts.  While there are not hard
dependency *at the beginning* of the process, you end up by having
some given some time.

So for instance, osp9 *used to be* fine with rhel 7.2 but now it seems
it has some hard dependencies on rhel 7.3.

The idea here is that it should be all covered by OSP *minor update*.
So for instance, you have your OSP8 up and running on rhel 7.2.  You
do the *OSP* minor update which should bring you OSP8 on rhel 7.3
(which is a rhel major upgrade...).  Then you make you OSP major
upgrade, which brings you OSP9 on rhel7.3.  It has the added benefit
of diminishing the downtime of floating ip during the OSP major
upgrade

I think we should include Mike Burns to tinker how we could capture
this.  Maybe include a matrix of the expected version of rhel repo
before doing an upgrade, which should be updated each time a new rhel
major release is done ?

Before upgrading OSP, which rhel version you should have (brought by OSP minor update): 
|          | osp9 | osp10 | osp11 |
| rhel 7.2 |      |       |       |
| rhel 7.3 | X    | X     | X     |

Initial installation:
|          | osp8 | osp9 | osp10 | osp11 |
| rhel 7.2 | X    | X    |       |       |
| rhel 7.3 | X    | X    | X     | X     |

What do you think ?

(In reply to Sofer Athlan-Guyot from comment #3)
> Hi Chris,
> 
> I think it helps.
> 
> The problem here are the moving parts.  While there are not hard
> dependency *at the beginning* of the process, you end up by having
> some given some time.
> 
> So for instance, osp9 *used to be* fine with rhel 7.2 but now it seems
> it has some hard dependencies on rhel 7.3.
> 
> The idea here is that it should be all covered by OSP *minor update*.
> So for instance, you have your OSP8 up and running on rhel 7.2.  You
> do the *OSP* minor update which should bring you OSP8 on rhel 7.3
> (which is a rhel major upgrade...).  Then you make you OSP major
> upgrade, which brings you OSP9 on rhel7.3.  It has the added benefit
> of diminishing the downtime of floating ip during the OSP major
> upgrade
> 
> I think we should include Mike Burns to tinker how we could capture
> this.  Maybe include a matrix of the expected version of rhel repo
> before doing an upgrade, which should be updated each time a new rhel
> major release is done ?
> 
> Before upgrading OSP, which rhel version you should have (brought by OSP
> minor update): 
> |          | osp9 | osp10 | osp11 |
> | rhel 7.2 |      |       |       |
> | rhel 7.3 | X    | X     | X     |
> 
> Initial installation:
> |          | osp8 | osp9 | osp10 | osp11 |
> | rhel 7.2 | X    | X    |       |       |
> | rhel 7.3 | X    | X    | X     | X     |
> 
> What do you think ?

I think that's over-complicating it. The RHEL content for *all* RHOSP releases comes from rhel-7-server-rpms (and friends), when you sync the RHOSP content you need to sync the latest RHEL content *at that point in time* or you will have dependency issues. If you want to sync them/freeze them then you need to do that at a point in time, not piecemeal, because both channels receive updates over time and those updates are tested together.

This applies across the board. FWIW the document linked in the description of this bug *does* include the correct repository rhel-7-server-rpms, it does not include EUS and it does not say or imply it's safe to use an out of date copy of this repository (this seems like a generally unsafe assumption to be making and the opposite of what I would assume). 

So I think the documentation update is really related to point (1) in the description - ensure that the upgrade procedure for all releases includes the pre-requisite step apply *all* RHOSP and RHEL minor release updates.

Ok thanks for the clarification here.  

So we should add a big warning at the beginning of the upgrade documentation that says:

 "Make sure you've followed the update procedure to bring RHOS to the latest minor version available before upgrading.  If that's bring a new kernel version (RHEL major upgrade) or an new major Openvswitch version (2.4->2.5 or the like) then you'll have to reboot all your nodes at some point before doing the upgrade"

Or something along those lines.

This is true for all version of RHOS.

WDYT ?

(In reply to Sofer Athlan-Guyot from comment #6)
> Ok thanks for the clarification here.  
> 
> So we should add a big warning at the beginning of the upgrade documentation
> that says:
> 
>  "Make sure you've followed the update procedure to bring RHOS to the latest
> minor version available before upgrading.  If that's bring a new kernel
> version (RHEL major upgrade) or an new major Openvswitch version (2.4->2.5
> or the like) then you'll have to reboot all your nodes at some point before
> doing the upgrade"
> 
> Or something along those lines.
> 
> This is true for all version of RHOS.
> 
> WDYT ?

Does the end user need to update to the latest minor version before upgrading major versions, or do they just need to make sure that they are performing the upgrade with the latest OSP and RHEL repos? 

I do not want the user to think that they need to perform a minor update and reboot all nodes (and move or shutdown all workloads) before performing a
major update (which also requires all workloads moved or shutdown). Unless this is actually what we require them to do?

Hi,

(In reply to Chris Paquin from comment #7)

> Does the end user need to update to the latest minor version before
> upgrading major versions, or do they just need to make sure that they are
> performing the upgrade with the latest OSP and RHEL repos?

They need to do the minor update before doing the major upgrade.

> I do not want the user to think that they need to perform a minor update and
> reboot all nodes (and move or shutdown all workloads) before performing a
> major update (which also requires all workloads moved or shutdown). Unless
> this is actually what we require them to do?

This is required.  For instance, for an osp9->10 upgrade, it is expected that the rhel version on all nodes is 7.3 before starting the upgrade (that's what the QA has tested).

The "reboot node during minor update" part is only mandatory if a new major kernel version is fetched during the update (or openvswitch major version jump).  This is not a requirement of the update but of the normal rhel lifecycle.  So if your minor update doesn't bring a new kernel (ie doesn't change the rhel version) or a new openvswitch major version then you don't have to reboot the nodes.

So we've got a note in the OSP10 upgrade doc:

"Red Hat OpenStack Platform 10 uses some new kernel parameters now available in Red Hat Enterprise Linux 7.3. Make sure that you have upgraded your undercloud and overcloud to Red Hat Enterprise Linux 7.3 and Open vSwitch 2.5. See Chapter 2, Director-Based Environments: Performing Updates to Minor Versions for instruction on performing a package update to your undercloud and overcloud. When you have updated the kernel to the latest version, perform a reboot so that the new kernel parameters take effect."

I'll add similar note (i.e. perform a minor update before the major upgrade) to the OSP9 docs.

Adding a note that reads:

"Make sure you have upgraded your undercloud and overcloud to the latest minor release of Red Hat OpenStack Platform 8 and Red Hat Enterprise Linux 7 before attempting a major upgrade to Red Hat OpenStack Platform 9. See <<sect-Updating_the_Environment>> for instruction on performing a package update to your undercloud and overcloud. If the kernel updates to the latest version, perform a reboot so that new kernel parameters take effect."

Note added:

https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Important_Pre-Upgrade_Notes

The note reads:

"Make sure that you have upgraded your undercloud and overcloud to the latest minor release of Red Hat OpenStack Platform 8 and Red Hat Enterprise Linux 7 before attempting a major upgrade to Red Hat OpenStack Platform 9. See Chapter 2, Director-Based Environments: Performing Updates to Minor Versions for instructions on performing a package update to your undercloud and overcloud. If the kernel updates to the latest version, perform a reboot so that new kernel parameters take effect."

Sofer, any further updates required for this BZ?

Hi,

(In reply to Dan Macpherson from comment #12)

> Sofer, any further updates required for this BZ?

Nope it's fine.  We need this for all versions though.  A bz has been open for osp10 there: https://bugzilla.redhat.com/show_bug.cgi?id=1419664

Hi Dan,

I think we can close this one, can't we ?

Thanks,

I think so. As a recent development, we've given the note more prominence as a big warning at the beginning of the chapter. So I think it's pretty clear that the minor version update is required now. Thanks, Sofer!