Bug 1366392

This Bz blocks the path for Upgrade from:  7.3  ->  8.0 ->  9.0 

this came after applying the workaround for Bz#1364583 that suggest to disable NetworkManager on the overcloud nodes - in order to successfully upgrade from 7.3 to 8.0.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1364583#c14

The error to take in account is:

Could not find declared class ::aodh

Note: don't worry about "Error: NetworkManager is not running" this is not critical.

So if TripleO can't find ::aodh, this is because puppet-aodh is not installed. You need to make sure OPM was upgraded during the process and make sure the aodh is here.

[heat-admin@overcloud-controller-0 ~]$ rpm -q puppet
puppet-3.6.2-2.el7.noarch
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.168.0.11 closed.
[stack@undercloud72 ~]$ rpm -qa|grep puppet
openstack-tripleo-puppet-elements-2.0.0-4.el7ost.noarch
puppet-3.6.2-4.el7sat.noarch
openstack-puppet-modules-8.1.8-1.el7ost.noarch

The setup was successfully upgraded from 7 to 8.

When I cam to upgrade it to 9, the reported issue occured during the aodh migration step.

I'm pretty sure OPM is not updated on the overcloud at the stage of Aodh migration.

Again: the error is OPM is not updated. Please ignore the NetworkManager thing.

Debugged the deployment. Moving the BZ to openstack-puppet-modules component and listing a workaround below.

Emilien is right that the root cause is missing aodh module. It's not in fact completely missing, it's present in /usr/share/openstack-puppet/modules, but it's not symlinked from /etc/puppet/modules, where Puppet looks for modules.

The symlinking is currently (probably incorrectly?) part of the overcloud image building process rather than the openstack-puppet-module RPM itself. The effect is that whenever we add a new module during RPM update of openstack-puppet-modules, it cannot be found by Puppet, because the symlink in /etc/puppet/modules is not created.

We should probably move the symlinking onto the RPM level. Re-running the DIB elements on overcloud to create the symlinks is probably not a realistic solution.


The workaround could to run this on every overcloud node:

ln -f -s /usr/share/openstack-puppet/modules/* /etc/puppet/modules/

prior to triggering the AODH migration. (The script line is taken from what the DIB element does when building an image [1].)


[1] https://github.com/openstack/tripleo-puppet-elements/blob/627b949430f9124181d4470abd908e25a9bfa760/elements/puppet-modules/install.d/puppet-modules-package-install/75-puppet-modules-package#L7

(In reply to Jiri Stransky from comment #10)
> Debugged the deployment. Moving the BZ to openstack-puppet-modules component
> and listing a workaround below.
> 
> Emilien is right that the root cause is missing aodh module. It's not in
> fact completely missing, it's present in
> /usr/share/openstack-puppet/modules, but it's not symlinked from
> /etc/puppet/modules, where Puppet looks for modules.
> 
> The symlinking is currently (probably incorrectly?) part of the overcloud
> image building process rather than the openstack-puppet-module RPM itself.
> The effect is that whenever we add a new module during RPM update of
> openstack-puppet-modules, it cannot be found by Puppet, because the symlink
> in /etc/puppet/modules is not created.
> 
> We should probably move the symlinking onto the RPM level. Re-running the
> DIB elements on overcloud to create the symlinks is probably not a realistic
> solution.
> 
> 
> The workaround could to run this on every overcloud node:
> 
> ln -f -s /usr/share/openstack-puppet/modules/* /etc/puppet/modules/
> 
> prior to triggering the AODH migration. (The script line is taken from what
> the DIB element does when building an image [1].)
> 
> 
> [1]
> https://github.com/openstack/tripleo-puppet-elements/blob/
> 627b949430f9124181d4470abd908e25a9bfa760/elements/puppet-modules/install.d/
> puppet-modules-package-install/75-puppet-modules-package#L7

Given the move that DIB makes, I'm not sure that it makes sense to have the RPM do this automatically.  I think I'd rather see the upgrade do this symlink.  

In theory, we're only going to add new modules between releases.  (There is a rare case, possibly, where we add one within a release in which case I'd say do this on update and upgrade).  

Doing this in the rpm could have significant issues for people who use OPM outside of director or packstack.  The might install them on a foreman server and import them for other hosts, but not want them in /etc/puppet on that machine.  

We also have to consider the packaging changes around OPM (separate rpms per module).

(In reply to Mike Burns from comment #11)
> Doing this in the rpm could have significant issues for people who use OPM
> outside of director or packstack.  The might install them on a foreman
> server and import them for other hosts, but not want them in /etc/puppet on
> that machine.  

Ok that's fair.

The solution on upgrade side will probably be a bit hacky because we'll need to put it into the AODH migration (as there's nothing prior that running on the cloud during upgrade) to fix up existing OSP 8 which are already in wrong state. 

The above would be just a mitaka-specific fix. We'd deal with this properly upstream in a slightly different way -- we should probably trigger the symlinking both on minor updates and major upgrades to keep consistent state at all times, though the chance of breakage during minor update is low.

Moving back to t-h-t then :)

Submitted a Mitaka-only patch to fixup existing Liberty deployments before doing AODH migration:

https://review.openstack.org/#/c/355446

And also a patch that should prevent getting the deployment into a bad state in the future:

https://review.openstack.org/#/c/356028

I'd still like to investigate if we can move away from the symlinks altogether, but such solution has some minor conflict potential (e.g. custom roles in TripleO, and probably unclean backport to mitaka), so putting the fix into the updates/upgrades first could still be a way to go.

Submitted another patch for TripleO to not depend on the /etc/puppet/modules symlinks.

https://review.openstack.org/#/c/356457/


The mitaka-only fixup already got merged and solves our immediate problem. The other two are more forward-looking to prevent such issues in the future, but we don't necessarily need them for OSP 7->8->9 upgrade.

I suggest backporting only the mitaka-only fixup for now, as it should fix the problem during AODH migration. I'll also change the BZ title accordingly.

Short term fix went into Mitaka and a long term solution to prevent similar issues from happening has been merged to Newton. Closing this BZ as it is predominantly about the fixed Mitaka issue. Adding the Newton patch too into the external trackers.