(In reply to Daniel Berrange from comment #0) 
> It should be possible to override the default compatibility level in OSP
> director at time of initial deployment. For example, if the default level of
> a particular OSP release is rhel-7.3.0, but it is known that the deployment
> will only ever have RHEL-7.5 compute nodes, then it should be possible for
> the admin to set the machine type to the rhel-7.5.0 version straightaway.

We probably also need to scope how this will fit into the director upgrade flow, the first question would be whether director has knowledge (or can get it) of exactly which version of RHEL is on all the hypervisors. Then using this knowledge it can keep the lowest common denominator machine type (e.g. rhel-7.3.0) for all nodes as the upgrade is applied. Once all machines are upgraded though there will be a need to push the new machine type setting (e.g. rhel-7.5.0) out, ideally without triggering a simultaneous stack update for all nodes.

(In reply to Stephen Gordon from comment #1)
> (In reply to Daniel Berrange from comment #0) 
> > It should be possible to override the default compatibility level in OSP
> > director at time of initial deployment. For example, if the default level of
> > a particular OSP release is rhel-7.3.0, but it is known that the deployment
> > will only ever have RHEL-7.5 compute nodes, then it should be possible for
> > the admin to set the machine type to the rhel-7.5.0 version straightaway.
> 
> We probably also need to scope how this will fit into the director upgrade
> flow, the first question would be whether director has knowledge (or can get
> it) of exactly which version of RHEL is on all the hypervisors. Then using
> this knowledge it can keep the lowest common denominator machine type (e.g.
> rhel-7.3.0) for all nodes as the upgrade is applied. Once all machines are
> upgraded though there will be a need to push the new machine type setting
> (e.g. rhel-7.5.0) out, ideally without triggering a simultaneous stack
> update for all nodes.

Note you may have further constraints as well;
  a) You may not want to change the machine type for existing VMs since this produces a guest visible change that may need driver updates or trigger a licensing issue.
  b) You say 'lowest common denominator machine type' - although you shouldn't have more than 2 versions in your cluster since you normally only ask us to ensure migration works from 7.n->7.n-1

(In reply to Dr. David Alan Gilbert from comment #2)
> (In reply to Stephen Gordon from comment #1)
> > (In reply to Daniel Berrange from comment #0) 
> > > It should be possible to override the default compatibility level in OSP
> > > director at time of initial deployment. For example, if the default level of
> > > a particular OSP release is rhel-7.3.0, but it is known that the deployment
> > > will only ever have RHEL-7.5 compute nodes, then it should be possible for
> > > the admin to set the machine type to the rhel-7.5.0 version straightaway.
> > 
> > We probably also need to scope how this will fit into the director upgrade
> > flow, the first question would be whether director has knowledge (or can get
> > it) of exactly which version of RHEL is on all the hypervisors. Then using
> > this knowledge it can keep the lowest common denominator machine type (e.g.
> > rhel-7.3.0) for all nodes as the upgrade is applied. Once all machines are
> > upgraded though there will be a need to push the new machine type setting
> > (e.g. rhel-7.5.0) out, ideally without triggering a simultaneous stack
> > update for all nodes.
> 
> Note you may have further constraints as well;
>   a) You may not want to change the machine type for existing VMs since this
> produces a guest visible change that may need driver updates or trigger a
> licensing issue.

Yes but...

>   b) You say 'lowest common denominator machine type' - although you
> shouldn't have more than 2 versions in your cluster since you normally only
> ask us to ensure migration works from 7.n->7.n-1

The lowest common denominator of two possible values is still the lowest common denominator ;). We don't have more than two RHEL releases in the cluster on the *hosts* (and we only have two while the upgrade is being rolled out) but when it comes to guest machine types things are a little more complicated and I believe that has been reflected in our transitive live migration requests in the past E.g:

"""
This bug is intended to track the need to be able to live migrate virtual machine instances to RHEL 7.2 hosts from:

- RHEL 6.7 - rhel-6.5.0 and rhel-6-6-0 and rhel-6-7-0 (if introduced*) machine type.
- RHEL 7.0 - rhel-7-0-0 machine type.
- RHEL 7.1 - rhel-7-0-0 and rhel-7-1-0 (if introduced*) machine type.
"""

From: https://bugzilla.redhat.com/show_bug.cgi?id=1191307

As it stands today if someone has carried e.g. rhel-6.5.0 VMs since early RHOS 4/5 days and managed to avoid rebuilding them via live migration (which is feasible) there is no process in RHOSP currently that would bring the machine type up to a newer one AFAIK (at least without triggering a rebuild anyway...). There's also no reason we believe this would cause a problem because, as above, we've ensured we request transitive forward live migration is supported for these guests. 

In the previous comment I was talking only about ensuring we move forward the machine type setting on the *hosts* (what this bug asks to introduce) for new guests as part of our upgrade process - not suggesting we would be going back and changing that of existing guests.

From my POV moving this forward in an automated fashion serves two purposes:

1) Ensuring new guests get access to whatever new features have been introduced in later machine types.

2) Limiting the scope of the damage if/when we ever do need to break the transitive live migration requirements for some as yet unforeseen reason in the future.

(In reply to Stephen Gordon from comment #3)
> ... managed to avoid rebuilding them via live migration (which
> is feasible)

Just to avoid possible confusion, rebuild and live migration are separate actions. What I was intending to highlight that using live migration they can move guests around the cluster through multiple rolling upgrades without ever having to rebuild the guest.

The patch to facilitate this from a puppet perspective is done, in 12 we need to do the "hard" parts:

1) Set our values for fresh installs, and maintain them going forward. Although we currently only support x86 to save ourselves problems in the future I would follow Dan's lead and cover all bases:

hw_machine_type=x86_64=pc-i440fx-rhel7.3.0,aarch64=virt-rhel7.3.0,ppc64=pseries-rhel7.3.0,ppc64le=pseries-rhel7.3.0

2) The more problematic case is upgrades, this will require some thought but I believe we can leave the *non*-x86 cases alone while for x86_64 we will need to provide a more exhaustive list OOTB on the upgrade *and* documentation for the admin on how they can determine whether they can whittle it down. This exhaustive list would need to cover all the combinations going back to whatever first rhel-6.*.* machine type we supported with RHOSP (somewhere around 6.1/6.2 I suspect?).

This bugzilla has been removed from the release and needs to be reviewed for targeting another release.

With https://review.openstack.org/#/c/396323 puppet-nova now supports setting the nova::compute::libvirt::libvirt_hw_machine_type parameter. That can be used already now (OSP11) with ExtraConfig in an environment file, for example:

ComputeExtraConfig:
    nova::compute::libvirt::libvirt_hw_machine_type: XXX

We can expose that in THT as well, not sure if that is useful though?

That should be:

NovaComputeExtraConfig:
    nova::compute::libvirt::libvirt_hw_machine_type: XXX

(In reply to Sven Anderson from comment #7)
> With https://review.openstack.org/#/c/396323 puppet-nova now supports
> setting the nova::compute::libvirt::libvirt_hw_machine_type parameter. That
> can be used already now (OSP11) with ExtraConfig in an environment file, for
> example:
> 
> ComputeExtraConfig:
>     nova::compute::libvirt::libvirt_hw_machine_type: XXX
> 
> We can expose that in THT as well, not sure if that is useful though?

It's a start, the key now is for us as a team to work out what to do with it particularly in the context of upgrades. The RHV approach would be something like:

1) I install RHV with 7.3 hosts, the machine type is fixed to rhel-7-3-0.
2) I start upgrading my hosts to 7.4, because I keep my machine type fixed I can still migrate guests both from old->new and new->old.
3) When I am finished upgrading my hosts I move my machine type to rhel-7-4-0 (this is hidden behind something like updating the cluster compat version in RHV).

In OSP the way this works is:

1) I install OSP with 7.3 hosts, my machine type goes to whatever is latest (rhel-7-3-0).
2) I start upgrading my hosts to 7.4, because my machine type goes to whatever is latest I now get a mix of rhel-7-3-0 and rhel-7-4-0 machine types depending on which host an instance lands on. I can only migrate old->new.

I believe now that this is parameterized we need to think about how we would implement something close to the first approach where we fix on a machine type using this setting and automagically move it forward when we have completed a RHEL upgrade of the entire cluster (but not before).

(In reply to Stephen Gordon from comment #9)
> (In reply to Sven Anderson from comment #7)
> > With https://review.openstack.org/#/c/396323 puppet-nova now supports
> > setting the nova::compute::libvirt::libvirt_hw_machine_type parameter. That
> > can be used already now (OSP11) with ExtraConfig in an environment file, for
> > example:
> > 
> > ComputeExtraConfig:
> >     nova::compute::libvirt::libvirt_hw_machine_type: XXX
> > 
> > We can expose that in THT as well, not sure if that is useful though?
> 
> It's a start, the key now is for us as a team to work out what to do with it
> particularly in the context of upgrades. The RHV approach would be something
> like:
> 
> 1) I install RHV with 7.3 hosts, the machine type is fixed to rhel-7-3-0.
> 2) I start upgrading my hosts to 7.4, because I keep my machine type fixed I
> can still migrate guests both from old->new and new->old.
> 3) When I am finished upgrading my hosts I move my machine type to
> rhel-7-4-0 (this is hidden behind something like updating the cluster compat
> version in RHV).
> 
> In OSP the way this works is:
> 
> 1) I install OSP with 7.3 hosts, my machine type goes to whatever is latest
> (rhel-7-3-0).
> 2) I start upgrading my hosts to 7.4, because my machine type goes to
> whatever is latest I now get a mix of rhel-7-3-0 and rhel-7-4-0 machine
> types depending on which host an instance lands on. I can only migrate
> old->new.
> 
> I believe now that this is parameterized we need to think about how we would
> implement something close to the first approach where we fix on a machine
> type using this setting and automagically move it forward when we have
> completed a RHEL upgrade of the entire cluster (but not before).

There's two migration scenarios.

 a. OSP version is unchanged, but RHEL update version is increased
 b. OSP is being upgraded to new OSP, RHEL may or may not be the same version

IIUC, we only need to support backwards migration in scenario (a). Ie we'll never support live migration from a host running OSP 10, to a host running OSP 9.

Given this, I would suggest that we keep things simple. Just fix the machine type for any given OSP version, at the RHEL version that co-incides with the initial GA for that OSP version. ie if OSP 11 is released on RHEL-7.4, then configure OSP 11 to use 'rhel-7.4.0' as the machine type out of the box. When users upgrade to 7.5, OSP 11 will still use rhel-7.4.0... when they upgrade to RHEL 7.6, OSP 11 will still use rhel-7.4.0... etc.

*if* the user really badly wants to use some feature only available in rhel-7.5.0 machine type, then let them manually change   nova::compute::libvirt::libvirt_hw_machine_type:  after they've upgraded all hosts to 7.5 (or partitioned their compute hosts to avoid backwards migration).

I don't think we should try todo anything clever to automaticallly update machine type within an OSP deployment.  IOW, most customers would just wait for OSP 12 deployment which will use the newer rhel-7.5.0 machine type out of the box.

I think we could also cover the upgrade scenario, as we already have to do something similar to the nova upgrade level pinning.

I.e the OSP12 upgrade initially sets the machine type to the OSP11 machine type (rhel-7.4.0 in the example above) and the nova upgrade level to ocata. The computes are gradually upgraded to OSP12... then finally the upgrade level pin is removed and the machine type is set to the OSP12 machine type (rhel 7.5.0).

(In reply to Ollie Walsh from comment #11)
> I think we could also cover the upgrade scenario, as we already have to do
> something similar to the nova upgrade level pinning.
> 
> I.e the OSP12 upgrade initially sets the machine type to the OSP11 machine
> type (rhel-7.4.0 in the example above) and the nova upgrade level to ocata.
> The computes are gradually upgraded to OSP12... then finally the upgrade
> level pin is removed and the machine type is set to the OSP12 machine type
> (rhel 7.5.0).

There's no reason for OSP12 deployment to ever use the OSP11 machine type, because you never live migrate from OSP12 down to OSP11. Any OSP12 compute nodes should always just start with their right default. Any running VMs that are migrated in from OSP11 will stay on the older machine type for as long as they live, but that's fine.

(In reply to Daniel Berrange from comment #12)
> (In reply to Ollie Walsh from comment #11)
> > I think we could also cover the upgrade scenario, as we already have to do
> > something similar to the nova upgrade level pinning.
> > 
> > I.e the OSP12 upgrade initially sets the machine type to the OSP11 machine
> > type (rhel-7.4.0 in the example above) and the nova upgrade level to ocata.
> > The computes are gradually upgraded to OSP12... then finally the upgrade
> > level pin is removed and the machine type is set to the OSP12 machine type
> > (rhel 7.5.0).
> 
> There's no reason for OSP12 deployment to ever use the OSP11 machine type,
> because you never live migrate from OSP12 down to OSP11. Any OSP12 compute
> nodes should always just start with their right default. Any running VMs
> that are migrated in from OSP11 will stay on the older machine type for as
> long as they live, but that's fine.

The point is that these nodes are not really OSP12 yet. They are OSP12 with the version pinned to OSP11, so from an external perspective at least they are still OSP11 computes. IIUC this state can persist for weeks while the operators gradually upgrade the computes.

(In reply to Ollie Walsh from comment #13)
> (In reply to Daniel Berrange from comment #12)
> > There's no reason for OSP12 deployment to ever use the OSP11 machine type,
> > because you never live migrate from OSP12 down to OSP11. Any OSP12 compute
> > nodes should always just start with their right default. Any running VMs
> > that are migrated in from OSP11 will stay on the older machine type for as
> > long as they live, but that's fine.
> 
> The point is that these nodes are not really OSP12 yet. They are OSP12 with
> the version pinned to OSP11, so from an external perspective at least they
> are still OSP11 computes. IIUC this state can persist for weeks while the
> operators gradually upgrade the computes.

I understand what you're getting at here, but from the POV of the goals of the "machine type" concept I think it is irrelevant. The "machine type" is a mechanism to encode an hardware ABI, to allow running VM state to be moved between hosts running different versions. The only reason to hold machine type back to an old version, is if you need to be able to relocate a running VM to an older version of the platform.  Even when OSP is in this hybrid state, with OSP12 compute nodes pinned to an OSP11 Nova RPC version, I don't believe we support migrating from the OSP12 node back to an OSP11 node, so I don't see a reason to hold back the machine type on the OSP12 nodes.

(In reply to Daniel Berrange from comment #14)
> I don't believe we support migrating from the OSP12 node back to an OSP11 node, so I
> don't see a reason to hold back the machine type on the OSP12 nodes.

Shouldn't we support this, given the epic "secure, seamless workload mobility"? Unless of course there is a technical reason not to permit it.

(In reply to Daniel Berrange from comment #14)
> (In reply to Ollie Walsh from comment #13)
> > (In reply to Daniel Berrange from comment #12)
> > > There's no reason for OSP12 deployment to ever use the OSP11 machine type,
> > > because you never live migrate from OSP12 down to OSP11. Any OSP12 compute
> > > nodes should always just start with their right default. Any running VMs
> > > that are migrated in from OSP11 will stay on the older machine type for as
> > > long as they live, but that's fine.
> > 
> > The point is that these nodes are not really OSP12 yet. They are OSP12 with
> > the version pinned to OSP11, so from an external perspective at least they
> > are still OSP11 computes. IIUC this state can persist for weeks while the
> > operators gradually upgrade the computes.
> 
> I understand what you're getting at here, but from the POV of the goals of
> the "machine type" concept I think it is irrelevant. The "machine type" is a
> mechanism to encode an hardware ABI, to allow running VM state to be moved
> between hosts running different versions. The only reason to hold machine
> type back to an old version, is if you need to be able to relocate a running
> VM to an older version of the platform.  Even when OSP is in this hybrid
> state, with OSP12 compute nodes pinned to an OSP11 Nova RPC version, I don't
> believe we support migrating from the OSP12 node back to an OSP11 node, so I
> don't see a reason to hold back the machine type on the OSP12 nodes.

The only reason we don't support it right now as I understand it is *because* we don't do any handling of the machine type, RHEL Virt are telling me they do test this scenario for RHV where they can do migrations both ways while the cluster is still being upgraded.

(In reply to Stephen Gordon from comment #16)
> The only reason we don't support it right now as I understand it is
> *because* we don't do any handling of the machine type, RHEL Virt are
> telling me they do test this scenario for RHV where they can do migrations
> both ways while the cluster is still being upgraded.

There's two different scenarios here.

1. Updating the RHEL version, while leaving OSP version unchanged. If we do the machine type handling as described above, that'll support migration in both directions.

2. Updating the OSP version, irrespective of whether RHEL version changes or not.  This is nothing todo with machine type or platform virt testing. It is a question of whether Nova supports such a scenario or not. My understanding is that we don't support live migration from OSP 12 compute nodes, over to OSP 11 compute nodes - only the forwards direction in order to move VMs off legacy compute nodes as part of the OSP upgrade process.

(In reply to Daniel Berrange from comment #17)
> (In reply to Stephen Gordon from comment #16)
> > The only reason we don't support it right now as I understand it is
> > *because* we don't do any handling of the machine type, RHEL Virt are
> > telling me they do test this scenario for RHV where they can do migrations
> > both ways while the cluster is still being upgraded.
> 
> There's two different scenarios here.

Yes but they are not completely independent scenarios, I first encounter (1) and then eventually I need to do (2) which will in part involve moving up to the machine type defaulted to by the given OSP release (e.g. we can envisage that for 11 if we had done this it would be rhel-7-3-0, for 12 it would be rhel-7-4-0).

> 1. Updating the RHEL version, while leaving OSP version unchanged. If we do
> the machine type handling as described above, that'll support migration in
> both directions.
>
> 2. Updating the OSP version, irrespective of whether RHEL version changes or
> not.  This is nothing todo with machine type or platform virt testing. It is
> a question of whether Nova supports such a scenario or not. My understanding
> is that we don't support live migration from OSP 12 compute nodes, over to
> OSP 11 compute nodes - only the forwards direction in order to move VMs off
> legacy compute nodes as part of the OSP upgrade process.

What specific aspect of Nova prevents us supporting live migration from e.g. 12->11 if the RPC versioning is still pinned to the lower one? Where is this documented? If there is no issue from a configuration of the virt platform POV then why *don't* we support this?

The problem I have with the above, and perhaps why we seem to talking past each other, is there seems to be an assumption that we can discount (2) because we would never want to migrate a guest back when there *is* in fact a desire to support such as part of a wider goal to ultimately facilitate rolling back part or all of an upgrade.

> 1. Updating the RHEL version, while leaving OSP version unchanged. If we do
> the machine type handling as described above, that'll support migration in
> both directions.
>

The documented update procedure[*] is:
1. Update undercloud packages (delivers new overcloud images but does not upload to glance).
2. Upload the overcloud images in glance (only apply to new nodes).
3. Update the existing overcloud nodes.

So between steps 2 and 3 is the only window where new compute nodes should get a newer RHEL version than the existing compute nodes.
I expect we would avoid the issue by swapping steps 2 and 3 and warning not to scale-up or replace compute nodes until the images are up to date.

* https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html/upgrading_red_hat_openstack_platform/keeping_openstack_platform_updated

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811