Bug 1388543

The full 'fix' here consists of two reviews - they were originally tracked as independent fixes for BZ 1364540 (the original ovs upgrade workaround) and then BZ 1388675 (added --replacepkgs incase ovs was already upgraded and fix ceph upgrade script syntax nit).  

The fixes are https://review.openstack.org/#/c/389753/ (needs to be cherrypicked to mitaka and liberty, already in newton) and https://review.openstack.org/#/c/390792/ (needs cherrypick to newton mitaka and liberty but master still in review. )

This landed into stable/mitaka today so moving to POST (and linking the stable/mitaka reviews above)

*** Bug 1396313 has been marked as a duplicate of this bug. ***

This issue is blocking life-cycle's OSP10 mandatory scenario , which is  Update + Upgrade  9.0GA -> 9.0Async -> 10.0 latest

https://bugzilla.redhat.com/show_bug.cgi?id=1396313 

raising severity to be fixed before OSP10 release

o/ mburns hello, I know how much you like bz spam, so here have some more:

I have an AI after Lifecycle scrum to ping you about this bug. Can we please have a build for OSP9 tripleo-heat-templates that includes this fix? The reviews linked have both landed to mitaka for a while now.

As Omri notes it is blocking us for 10 because we want update to latest 9 then upgrade, and sasha is hitting BZ 1396313 (which is this issue) in his tests.

so discussed on today's scrum, sasha will verify and then we can move to onqa... @sasha please check mburns added a Fixed In Version: openstack-tripleo-heat-templates-2.0.0-39.el7ost → openstack-tripleo-heat-templates-2.0.0-40.el7ost

so make sure you grab the right version - looks like mburns landed this directly downstream over this weekend (it landed into mitaka a couple of weeks ago)

Version:
openstack-tripleo-heat-templates-2.0.0-40.el7ost.noarch


So what happens now is that the controllers remained reachable and the non-controllers became unreachable during minor update.
The minor update failed.

01:42:17 IN_PROGRESS
01:42:17 IN_PROGRESS
01:42:17 IN_PROGRESS
01:42:17 IN_PROGRESS
01:42:17 IN_PROGRESS
01:42:17 IN_PROGRESS
01:42:17 ERROR: Authentication failed: Authentication required
01:42:17 There was an error running Run the OC update command. Exiting....

This is from a controller's yum.log:
[root@overcloud-controller-0 ~]# grep -i openvswitch /var/log/yum.log
Nov 21 21:52:09 Updated: python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch
Nov 21 21:53:31 Updated: 1:openstack-neutron-openvswitch-8.1.2-12.el7ost.noarch

(In reply to Alexander Chuzhoy from comment #14)
> Version:
> openstack-tripleo-heat-templates-2.0.0-40.el7ost.noarch
> 
> 
> So what happens now is that the controllers remained reachable and the
> non-controllers became unreachable during minor update.
> The minor update failed.
> 
> 01:42:17 IN_PROGRESS
> 01:42:17 IN_PROGRESS
> 01:42:17 IN_PROGRESS
> 01:42:17 IN_PROGRESS
> 01:42:17 IN_PROGRESS
> 01:42:17 IN_PROGRESS
> 01:42:17 ERROR: Authentication failed: Authentication required
> 01:42:17 There was an error running Run the OC update command. Exiting....
> 
> This is from a controller's yum.log:
> [root@overcloud-controller-0 ~]# grep -i openvswitch /var/log/yum.log
> Nov 21 21:52:09 Updated:
> python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch
> Nov 21 21:53:31 Updated:
> 1:openstack-neutron-openvswitch-8.1.2-12.el7ost.noarch

thanks Sasha, so this shows that openvswitch was updated ^^^ for controller-0 but there isn't much more to go on here. "The minor update failed" do we have any logs? Is this environment still available? We may need to hand over to networking team if there is still a problem with the ovs upgrade on non controllers

Failed QA for good reason https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/yum_update.sh#L81

we need to handle the OVS case also on noncontrollers before exit0

(In reply to Lukas Bezdicka from comment #16)
> Failed QA for good reason
> https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/
> tasks/yum_update.sh#L81
> 
> we need to handle the OVS case also on noncontrollers before exit0

thanks Lukas and Sasha - looks like a legit issue and a good catch @Sasha . I don't think we need to hand-over then to the networking team. This is something we can resolve ourselves. The workaround just needs to move up by a few lines so it gets executed the same on controllers/non.

removing master and adding the mitaka cherrypick for the latest fix we need here @ https://review.openstack.org/#/c/401365/

https://review.openstack.org/#/c/401365/ mitaka merged moving POST

Verified:

Environment:
openstack-tripleo-heat-templates-2.0.0-41.el7ost.noarch


The controller is reachable and has IPs.
[root@overcloud-controller-0 ~]# grep openvswitch-2 /var/log/yum.log
Nov 29 21:27:44 Updated: python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch

(In reply to Alexander Chuzhoy from comment #26)
> Verified:
> 
> Environment:
> openstack-tripleo-heat-templates-2.0.0-41.el7ost.noarch
> 
> 
> The controller is reachable and has IPs.
> [root@overcloud-controller-0 ~]# grep openvswitch-2 /var/log/yum.log
> Nov 29 21:27:44 Updated:
> python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch

@Sasha, just to be clear, it should be 'openvswitch-2.5.x...' you are pointing at python-openvswitch here ^^^ though they carry the same release version so I think they are updated together anyway (i.e. you very likely did get openvswitch 2.5 on that env)

e.g. 

[root@overcloud-controller-0 ~]# rpm -qa | grep openvswitch 
openstack-neutron-openvswitch-9.1.0-7.el7ost.noarch
openvswitch-2.5.0-14.git20160727.el7fdp.x86_64
python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch
[root@overcloud-controller-0 ~]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2983.html