The full 'fix' here consists of two reviews - they were originally tracked as independent fixes for BZ 1364540 (the original ovs upgrade workaround) and then BZ 1388675 (added --replacepkgs incase ovs was already upgraded and fix ceph upgrade script syntax nit). The fixes are https://review.openstack.org/#/c/389753/ (needs to be cherrypicked to mitaka and liberty, already in newton) and https://review.openstack.org/#/c/390792/ (needs cherrypick to newton mitaka and liberty but master still in review. )
This landed into stable/mitaka today so moving to POST (and linking the stable/mitaka reviews above)
*** Bug 1396313 has been marked as a duplicate of this bug. ***
This issue is blocking life-cycle's OSP10 mandatory scenario , which is Update + Upgrade 9.0GA -> 9.0Async -> 10.0 latest https://bugzilla.redhat.com/show_bug.cgi?id=1396313 raising severity to be fixed before OSP10 release
o/ mburns hello, I know how much you like bz spam, so here have some more: I have an AI after Lifecycle scrum to ping you about this bug. Can we please have a build for OSP9 tripleo-heat-templates that includes this fix? The reviews linked have both landed to mitaka for a while now. As Omri notes it is blocking us for 10 because we want update to latest 9 then upgrade, and sasha is hitting BZ 1396313 (which is this issue) in his tests.
so discussed on today's scrum, sasha will verify and then we can move to onqa... @sasha please check mburns added a Fixed In Version: openstack-tripleo-heat-templates-2.0.0-39.el7ost → openstack-tripleo-heat-templates-2.0.0-40.el7ost so make sure you grab the right version - looks like mburns landed this directly downstream over this weekend (it landed into mitaka a couple of weeks ago)
Version: openstack-tripleo-heat-templates-2.0.0-40.el7ost.noarch So what happens now is that the controllers remained reachable and the non-controllers became unreachable during minor update. The minor update failed. 01:42:17 IN_PROGRESS 01:42:17 IN_PROGRESS 01:42:17 IN_PROGRESS 01:42:17 IN_PROGRESS 01:42:17 IN_PROGRESS 01:42:17 IN_PROGRESS 01:42:17 ERROR: Authentication failed: Authentication required 01:42:17 There was an error running Run the OC update command. Exiting.... This is from a controller's yum.log: [root@overcloud-controller-0 ~]# grep -i openvswitch /var/log/yum.log Nov 21 21:52:09 Updated: python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch Nov 21 21:53:31 Updated: 1:openstack-neutron-openvswitch-8.1.2-12.el7ost.noarch
(In reply to Alexander Chuzhoy from comment #14) > Version: > openstack-tripleo-heat-templates-2.0.0-40.el7ost.noarch > > > So what happens now is that the controllers remained reachable and the > non-controllers became unreachable during minor update. > The minor update failed. > > 01:42:17 IN_PROGRESS > 01:42:17 IN_PROGRESS > 01:42:17 IN_PROGRESS > 01:42:17 IN_PROGRESS > 01:42:17 IN_PROGRESS > 01:42:17 IN_PROGRESS > 01:42:17 ERROR: Authentication failed: Authentication required > 01:42:17 There was an error running Run the OC update command. Exiting.... > > This is from a controller's yum.log: > [root@overcloud-controller-0 ~]# grep -i openvswitch /var/log/yum.log > Nov 21 21:52:09 Updated: > python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch > Nov 21 21:53:31 Updated: > 1:openstack-neutron-openvswitch-8.1.2-12.el7ost.noarch thanks Sasha, so this shows that openvswitch was updated ^^^ for controller-0 but there isn't much more to go on here. "The minor update failed" do we have any logs? Is this environment still available? We may need to hand over to networking team if there is still a problem with the ovs upgrade on non controllers
Failed QA for good reason https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/yum_update.sh#L81 we need to handle the OVS case also on noncontrollers before exit0
(In reply to Lukas Bezdicka from comment #16) > Failed QA for good reason > https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/ > tasks/yum_update.sh#L81 > > we need to handle the OVS case also on noncontrollers before exit0 thanks Lukas and Sasha - looks like a legit issue and a good catch @Sasha . I don't think we need to hand-over then to the networking team. This is something we can resolve ourselves. The workaround just needs to move up by a few lines so it gets executed the same on controllers/non.
removing master and adding the mitaka cherrypick for the latest fix we need here @ https://review.openstack.org/#/c/401365/
https://review.openstack.org/#/c/401365/ mitaka merged moving POST
Verified: Environment: openstack-tripleo-heat-templates-2.0.0-41.el7ost.noarch The controller is reachable and has IPs. [root@overcloud-controller-0 ~]# grep openvswitch-2 /var/log/yum.log Nov 29 21:27:44 Updated: python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch
(In reply to Alexander Chuzhoy from comment #26) > Verified: > > Environment: > openstack-tripleo-heat-templates-2.0.0-41.el7ost.noarch > > > The controller is reachable and has IPs. > [root@overcloud-controller-0 ~]# grep openvswitch-2 /var/log/yum.log > Nov 29 21:27:44 Updated: > python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch @Sasha, just to be clear, it should be 'openvswitch-2.5.x...' you are pointing at python-openvswitch here ^^^ though they carry the same release version so I think they are updated together anyway (i.e. you very likely did get openvswitch 2.5 on that env) e.g. [root@overcloud-controller-0 ~]# rpm -qa | grep openvswitch openstack-neutron-openvswitch-9.1.0-7.el7ost.noarch openvswitch-2.5.0-14.git20160727.el7fdp.x86_64 python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch [root@overcloud-controller-0 ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2983.html