Bug 1772955

This happens if z8 content is used with z9 THT. Please make sure that the systems have been properly updated (packages/overcloud images/containers).

It may not technically be a bug, but it is a hard-to-catch mistake and one that could be avoided through a simple compatibility check.

I just hit this same issue using RHOSP 13.0-20190827.1.el7ost. Please consider how a check on z8 content versus z9 could be added

We can try and add some additional logic in, however it won't land until at least z10. This highlights a break down in best practices for applying updates and ensuring environments have the correct content which seems like an end user problem that isn't necessarily solved by more checks in director.

It also highlights an implicit dependency that end users have no way of identifying. Adding checks to Director may not be ideal, but Director code (or rather its developers) is in a better position to understand its dependencies. 

In my case, I pinned the RHOSP Director image version to one that was working for me at the time. I believe it is quite common for deployments to not always take the latest and greatest - perhaps some guidance on how to pick a particular version and the consequences of that choice ( e.g. if you pick RHOSP overcloud image 13.0-20190827.1.el7ost, also pick xyz ) could help users avoid this particular mistake

Thanks for reopening

Pinning to an image is not a good idea. The image must match the current version or scale outs/deployments will fail. When you update the overcloud, the overcloud-full image must match the software being updated to.  This is just one instance where this is, but there is always an assumption of the correct base when performing deployment actions. You can't use an OSP10 overcloud image with an OSP13 THT, it just doesn't work.  You'll hit problems all over the place for example minor RHEL updates doing that as well when things like pacemaker or libvirt end up being mix-matched.  This would be fine if you use the THT that went with that overcloud image, however because a newer THT is used the expectations were out of date. Container versions are have this similar expectations.

Specifically this is documented as part of upgrading the undercloud: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/keeping_red_hat_openstack_platform_updated/assembly-upgrading_the_undercloud

We expect that if you update the undercloud (which would imply a newer THT) that you upload the current overcloud-full images.

There were some compatibility issues with the newer overcloud image, as it was based on RHEL 7.6 versus 7.5 and our custom installed packages weren't ready for the migration. I am not saying you are wrong, but users may also have valid reasons for staying with a particular image version.

And sometimes it may still work with an older disk image. In order to avoid "bug reports" like this one, I think it would benefit everyone involved if there were some explicit version checks added.

The link you referenced says "only use OpenStack Platform 13 images with the OpenStack Platform 13 Heat templates". But apparently (in this case) the compatibility requirements are more strict than that, i.e. only use OSP13 z9 images with OSP13 z9 Heat templates (and OSP13 z9 python-tripleo client, etc. )

In my view, the key moment to check is during a fresh install - i.e. during 'overcloud deploy', as one of the first things to check when running Ansible scripts on the individual node. I am curious as to where exactly things break down, and I wonder if it wouldn't be possible to design things to be 'self contained' - i.e. Director would run an install script that is part of the overcloud image itself, and therefore (by definition) compatible.

I think version control has always been an important part of OSP deployments (and software in general), and it goes beyond simple major version equality.

Right but if you're locking to a version, it needs to be consistent. Unfortunately the deployment framework doesn't have a concept of these minor version so it assumes that is handled externally via satellite or some other method.  If you have a need to lock to a specific zstream, it must be a complete lock otherwise you will end up in this case.  We try to make them as backwards compatible but OSP ends up also being at the mercy of RHEL minor releases as well.  When we test we're testing a complete zstream to another complete zstream as that is the recommended way to keep your environment.  

Anyway I've got a proposed patch to make this specific functionality ignore the error since it should be fine if it fails as it's just for speed improvements. If you hit this error, you may have speed related deployment issues in environments with large number of network interfaces.

Ah, the wonderful world of software package versions and their dependencies...

Anyway, agreed - this specific issue can be a simple matter of ignoring the error, making the code more compatible with previous releases.
We can solve world hunger next time...

Turns out the fix referenced here was also to address Bug 1772201 (same). Feel free to follow that bz for updates on when the fix will be shipped.

*** This bug has been marked as a duplicate of bug 1772201 ***