Bug 1365691

The root cause i think is that at the environment already contains OSP 9 openstack-puppet-modules (openstack-puppet-modules-8.x) when at this stage it should still contain OSP 8 ones (openstack-puppet-modules-7.x).

[root@overcloud-controller-0 ~]# rpm -q openstack-puppet-modules
openstack-puppet-modules-8.1.7-2.el7ost.noarch

Is it possible that the environment was deployed with OSP 9 image rather than OSP 8 image?


There might also be one other environment misconfig that i noticed -- the OSP 8 repos aren't enabled on the controllers:

[root@overcloud-controller-0 ~]# ls /etc/yum.repos.d/
redhat.repo

Having OSP 8 repos enabled is a necessity for the AODH migration step, which should be done before the Keystone migration. So this environment is generally in a peculiar state :)

Correction, it probably wasn't OSP 9 image, but something updated OPM specifically. Here's yum log from controller 0:

Aug 09 21:13:17 Installed: rhn-org-trusted-ssl-cert-1.0-1.noarch
Aug 09 21:23:07 Erased: 1:openstack-ceilometer-alarm-5.0.2-2.el7ost.noarch
Aug 09 21:24:43 Updated: 2:python-oslo-config-3.9.0-1.el7ost.noarch
Aug 09 21:24:44 Installed: python-gnocchiclient-2.2.0-1.el7ost.noarch
Aug 09 21:24:44 Updated: python-wsme-0.8.0-1.el7ost.noarch
Aug 09 21:24:44 Installed: python-pika-0.10.0-3.el7ost.noarch
Aug 09 21:24:44 Installed: python-pika_pool-0.1.3-3.el7ost.noarch
Aug 09 21:24:45 Updated: python-oslo-messaging-4.5.0-2.el7ost.noarch
Aug 09 21:24:45 Installed: python-aodh-2.0.3-2.el7ost.noarch
Aug 09 21:24:45 Installed: openstack-aodh-common-2.0.3-2.el7ost.noarch
Aug 09 21:24:46 Installed: openstack-aodh-evaluator-2.0.3-2.el7ost.noarch
Aug 09 21:24:52 Installed: openstack-aodh-notifier-2.0.3-2.el7ost.noarch
Aug 09 21:24:58 Installed: openstack-aodh-listener-2.0.3-2.el7ost.noarch
Aug 09 21:25:05 Installed: openstack-aodh-api-2.0.3-2.el7ost.noarch
Aug 09 21:34:39 Updated: 1:openstack-puppet-modules-8.1.7-2.el7ost.noarch

By the timestamp it doesn't look like the OPM update was pulled in by the AODH migration though, and AODH migration should be the step in upgrade workflow right before Keystone migration.

Another piece of info -- the openstack-puppet-modules upgrade wasn't triggered by Heat, it happened at a time when os-collect-config was idling:

https://paste.fedoraproject.org/405501/8372281/raw/

Talked to @sasha and this hasn't reproduced.  Given it appears that it was impacted by something updating OPM outside of the upgrade process, let's call this NOTABUG until it reproduces.

Hi I am also experiencing this error. I can confirm the puppet modules have been updated as per the upgrade documentation (section 3.4.3): https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/paged/upgrading-red-hat-openstack-platform/chapter-3-director-based-environments-performing-upgrades-to-major-versions

This sounds like a mistake in the documentation?

I can confirm downgrading the OPM to 7.x

 for i in $(nova list|grep ctlplane|awk -F' ' '{ print $12 }'|awk -F'=' '{ print $2 }'); do ssh -o StrictHostKeyChecking=no heat-admin@$i "sudo yum -y downgrade openstack-puppet-modules-7.0.19-1.el7ost.noarch" ; done

and running 'pcs resource cleanup' on the controllers before redeploying appears to have resolved the issue.

Hello, could you please post what was the version of the openstack-puppet-modules package before downgrading to 7.0.19-1? Was the overcloud already receiving OSP 9 packages on overcloud?

At the point of "3.4.3. Upgrading Keystone" in the docs, only the undercloud (the Director node) should be on OSP 9 repositories. The overcloud servers should still be on OSP 8 repositories, so updating the openstack-puppet-modules on overcloud should result in getting the latest OPM in 7.x series, but not 8.x yet.

The overcloud should switch to OSP 9 repositories during "3.4.4. Installing the Upgrade Scripts".

Is there a chance that the repository switching commands in "3.2. Upgrading the Director" were run on the overcloud too perhaps?

This is very likely as I explicitly enabled the OSP9 repos on the controllers as presumably they are required to install the aodh packages in 3.4.2? Without them being enabled then the puppet update fails as it cannot satisfy the package requirements. It could be I'm missing something though?

The AODH packages are in OSP 8 repos too, so the replacement of ceilometer alarm with AODH should succeed while the controllers are on OSP 8 repos.

Okay this must be my error, apologies for the confusion. Thanks Charlie