Bug 1419751 - Assure external routing to VMs through out update/upgrade
Summary: Assure external routing to VMs through out update/upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Marios Andreou
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks: 1437322
TreeView+ depends on / blocked
 
Reported: 2017-02-06 23:38 UTC by Jaromir Coufal
Modified: 2017-05-17 19:45 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-heat-templates-6.0.0-7.el7ost
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 19:45:13 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC
RDO 5951 None None None 2017-03-24 16:24:41 UTC
OpenStack gerrit 445494 None None None 2017-03-15 15:48:05 UTC
Launchpad 1671504 None None None 2017-03-09 14:44:40 UTC

Description Jaromir Coufal 2017-02-06 23:38:44 UTC
As a tenant user, I want to be able to access my VMs during update/upgrade procedures so that I can continue providing my business critical deliverables.

Comment 1 Steven Hardy 2017-03-09 14:46:35 UTC
https://bugs.launchpad.net/tripleo/+bug/1671504 raised - I wonder if we can simply use upgrade_batch_tasks for this, but we'll need to investigate the package dependencies.

Comment 3 Marios Andreou 2017-03-24 16:24:42 UTC
Hi, update on progress (tl;dr we know what breaks the pingtest, but still blocked on ovs/related issue )  I reached out to jlibosva from the network team for help and he immediately responded (copy/paste my email at [0] for context). 

So Jakub quickly confirmed it was openvswitch which is causing the neutron-openvswitch agent to be started (even though it is in a stopped, by us, state). He found an issue in the neutron-openvswitch-agent service file and posted https://review.rdoproject.org/r/#/c/5951/ to fix it. The idea is if we upgrade to this version of openstack-neutron packages (with Jakub fix) then the subsequent openvswitch upgrade should no longer cause the neutron-openvswitch-agent to try and start (prematurely, see [0] for more info on why this is a problem).

Unfortunately in my testing upgrading this way, that is, first upgrade openstack-neutron packages to the ones with jakub fix (he made a repo which has builds with the fix, which i enabled as part of my upgrade-init.yaml environment file) then upgrading openvswitch/all the things. As soon as openvswitch is upgraded to ovs 2.6 i lose all node connectivity/all 3 controllers. I tried doing this both via 'yum update' (for openvswitch i mean) and also including https://review.openstack.org/#/c/434346/ (i.e. the 'special case upgrade with flags' discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1424945#c16 ) and had same result both times.

I think we still need the fix at https://review.rdoproject.org/r/#/c/5951/ (adding to the trackers) but not sure what is going on with openvswitch ... we can pick this up next week and/or someone has more ideas.

thanks, marios


[0] (via email marios->jlibosva):
"
https://review.openstack.org/#/c/445494/ is the review (the 'rolling one node at a time' mechanism is already in place, we are just using it and adding to the l3 agent service here). It does what it's meant to - that code is executed on one node at a time so only one l3 agent is down. Lose like 1/2 ping. Great.

However, upgrade continues and at this point all services are down ( cluster, neutron-* except l3, and all the things). Then _something_ starts the neutron-openvswitch-agent - I am fairly confident it is openvswitch itself (am going from openvswitch-2.5.0-14 to 2.6 so there is an openvswitch restart?). Someone suggested it may even be python-openvswitch but not sure at this point. In other words as these packages are updated as part of the workflow, the neutron-openvswitch-agent is started 

Problem is neutron-openvswitch-agent cannot start at this point because rabbit is still down. And the fact that n-ovs-a starts/tries to start kills the ping and it stays down (even though l3 agents are running) until puppet reconfigures and starts all the things again.   
"

Comment 4 Sofer Athlan-Guyot 2017-03-24 16:52:01 UTC
Hi, Marios,

for what is worth, just updating:


       1:python-openvswitch-2.6.1-8.git20161206.el7fdb.noarch
       1:openstack-neutron-openvswitch-10.0.0-6.el7ost.noarch

achieves to have openvswitch issue:

       2017-03-24T15:02:55.075Z|03596|rconn|WARN|br-tun<->tcp:127.0.0.1:6633:
       connection failed (Connection refused)

       2017-03-24T15:02:55.075Z|03599|rconn|WARN|br-infra<->tcp:127.0.0.1:6633:
       connection failed (Connection refused)

(on compute node during testing)

Just FYI.

Comment 7 Marios Andreou 2017-03-31 16:13:15 UTC
Adding a note on request here...

Current status is still as summarized in comment #3 - we need  aversion of neutron/openvswitch packages that do not cause the stopped neutron-openvswitch-agent to start when the openvswitch is upgraded. I am not sure if puddle versions for neutron have been updated but I include them at the bottom if it is significant to why it may be working better today.

I tested today with https://review.openstack.org/#/c/448602/12 https://review.openstack.org/#/c/434346/25 and https://review.openstack.org/#/c/445494/9 .

The first two reviews in particular have had multiple changes today but some combination of those things, plus adding SkipUpgradeConfigTags: [validation] into my upgrade env file (for reasons commented at /#/c/445494/9)) so the upgrade could actually proceed seems to allow the ping to continue through the upgrade.

We need to revalidate of course next week, but fyi. The versions of neutron/openvswitch are like:

[root@overcloud-controller-1 ~]# rpm -qa | grep "neutron\|openvswitch"
openstack-neutron-common-10.0.0-11.el7ost.noarch
python-neutron-tests-10.0.0-11.el7ost.noarch
openstack-neutron-openvswitch-10.0.0-11.el7ost.noarch
python-neutron-lib-1.1.0-1.el7ost.noarch
openstack-neutron-ml2-10.0.0-11.el7ost.noarch
openstack-neutron-metering-agent-10.0.0-11.el7ost.noarch
python-neutronclient-6.1.0-1.el7ost.noarch
openstack-neutron-sriov-nic-agent-10.0.0-11.el7ost.noarch
python-neutron-10.0.0-11.el7ost.noarch
openvswitch-2.6.1-10.git20161206.el7fdp.x86_64
python-neutron-lbaas-10.0.0-3.el7ost.noarch
openstack-neutron-bigswitch-agent-10.0.3-1.el7ost.noarch
openstack-neutron-bigswitch-lldp-10.0.3-1.el7ost.noarch
puppet-neutron-10.3.0-2.el7ost.noarch
python-openvswitch-2.6.1-10.git20161206.el7fdp.noarch
openstack-neutron-10.0.0-11.el7ost.noarch
openstack-neutron-lbaas-10.0.0-3.el7ost.noarch

Comment 8 Marios Andreou 2017-04-03 11:34:50 UTC
After a call with ajo today I think the premise behind the current upstream effort is wrong. We apparently don't even need the l3 agents for the tenant vm IPs. Sure, if all l3 agents are down you won't be able to _create_ new IPs for example but the existing ones should be reachable OK.

If they aren't then there is a bug (and there *were*/*are* bugs) - like the one discovered testing https://review.openstack.org/#/c/445494/9 that neutron-openvswitch-agent is being started when openvswitch is being updated during the upgrade package update. Another is an issue with 'ryu' and is likely the one I hit on Friday see review comments@ /#/c/445494/). Apparently there are newer package builds from Friday afternoon for neutron-* that might solve some of these.

So, plan today is to test without this change and see where the ping fails using those latest packages, so we can be clearer about any outstanding bugs.

Comment 9 Miguel Angel Ajo 2017-04-04 13:51:00 UTC
(In reply to marios from comment #8)
> After a call with ajo today I think the premise behind the current upstream
> effort is wrong. We apparently don't even need the l3 agents for the tenant
> vm IPs. Sure, if all l3 agents are down you won't be able to _create_ new
> IPs for example but the existing ones should be reachable OK.
> 
> If they aren't then there is a bug (and there *were*/*are* bugs) - like the
> one discovered testing https://review.openstack.org/#/c/445494/9 that
> neutron-openvswitch-agent is being started when openvswitch is being updated
> during the upgrade package update. Another is an issue with 'ryu' and is
> likely the one I hit on Friday see review comments@ /#/c/445494/).
> Apparently there are newer package builds from Friday afternoon for
> neutron-* that might solve some of these.
> 
> So, plan today is to test without this change and see where the ping fails
> using those latest packages, so we can be clearer about any outstanding bugs.

That is correct.

The automatic startup of neutron-openvswitch-agent on restart of openvswitch may not be disrupting the dataplane. If we can get that state reproduced I'd like to look at the logs or log into the system and explore the failure. I've tried reproducing it locally (with a simplified deployment) with no success.

If we want to stop the neutron-openvswitch-agent to automatically start on openvswitch restart we need to merge this patch: https://review.rdoproject.org/r/#/c/5951/ but apparently that breaks tripleo somewhere else.

Set the needinfo to wait on results of the playback without the patches you said on comment 8.

Comment 10 Marios Andreou 2017-04-04 16:38:38 UTC
ack ajo, thanks for following up. FTR I've been *trying* to get that ^^^ by running a tenant pingtest through the upgrades today/yesterday but we are having unrelated problems ([1] fwiw/if interested) which causes the stack to fail before hitting the relevant part. I will update asap.

[1] https://bugs.launchpad.net/tripleo/+bug/1678101 & /+bug/1679486

Comment 11 Marios Andreou 2017-04-05 11:09:19 UTC
Hey, so thanks for waiting - I ran an upgrade this morning with a pingtest going and it completed OK. AFAICS there was no problem with the ping this time - package versions & details below.

The environment had the 'manual' openvswitch upgrade patch [0] applied, with the rpm --flags openvswitch upgrade from ovs2.5->2.6 delivered as ansible steps (during the main controller upgrade step).

I deployed vanila OSP10 3control/1compute and start a tenant vm for pinging (I use [1]). Through the controller upgrade I was checking the ping with [2] and this time there was no loss, even when at some point during the upgrade all neutron-* were down including l3-agent.

I also saw that after openvswitch was upgraded to 2.6 there was no start to neutron-openvswitch-agent as discussed in comment #3. I am not sure if that is because of [0] and/or it is something that landed into packaging? Package versions are like [3]

I noticed that the tenant router was hosted on the same controller before/after the upgrade (neutron l3-agent-list-hosting-router) indeed there was no failover as I didn't attempt to upgrade l3 agents one at a time. So to be clear and to set expectations correctly, as we discussed this means that during the controller upgrade operator shouldn't expect to *manage* floating IPs, but, it looks like they should be able to reach them during the upgrade (and that is a huge improvement),

thanks, marios

[0] https://review.openstack.org/#/c/434346

[1] if ! [[ -d ~/tripleo-ci ]]; then git clone http://github.com/openstack-infra/tripleo-ci.git ~/tripleo-ci ;fi; ~/tripleo-ci/scripts/tripleo.sh --overcloud-pingtest --skip-pingtest-cleanup # pingtest no cleanup.

[2] LOGFILE=PINGTEST_CONTROLLER_UPGRADE ; IP=192.0.2.53; while [ true ]; do if ping -c 1 $IP; then echo ""; echo "OK `date`"; sleep 3; else echo "************* NOPE ! ******************* `date`"; sleep 2; fi ; done 2>&1 | tee -a $LOGFILE

[3] [root@overcloud-controller-1 ~]# rpm -qa | grep "neutron\|openvswitch"
puppet-neutron-10.3.0-2.el7ost.noarch
python-neutron-lib-1.1.0-1.el7ost.noarch
openstack-neutron-metering-agent-10.0.0-13.el7ost.noarch
openstack-neutron-ml2-10.0.0-13.el7ost.noarch
python-neutron-tests-10.0.0-13.el7ost.noarch
python-neutronclient-6.1.0-1.el7ost.noarch
openstack-neutron-10.0.0-13.el7ost.noarch
openvswitch-2.6.1-10.git20161206.el7fdp.x86_64
openstack-neutron-bigswitch-agent-10.0.3-2.el7ost.noarch
python-neutron-lbaas-10.0.0-5.el7ost.noarch
openstack-neutron-lbaas-10.0.0-5.el7ost.noarch
python-neutron-10.0.0-13.el7ost.noarch
openstack-neutron-sriov-nic-agent-10.0.0-13.el7ost.noarch
openstack-neutron-bigswitch-lldp-10.0.3-2.el7ost.noarch
python-openvswitch-2.6.1-10.git20161206.el7fdp.noarch
openstack-neutron-openvswitch-10.0.0-13.el7ost.noarch
openstack-neutron-common-10.0.0-13.el7ost.noarch

Comment 12 Ian Pilcher 2017-04-05 13:14:19 UTC
(In reply to marios from comment #8)
> After a call with ajo today I think the premise behind the current upstream
> effort is wrong. We apparently don't even need the l3 agents for the tenant
> vm IPs. Sure, if all l3 agents are down you won't be able to _create_ new
> IPs for example but the existing ones should be reachable OK.

I'm 99% sure that this is not correct.  An L3 agent is required for almost all "north/south" traffic -- i.e. a VM communicating with anything not on its own network.

The only exception of which I know is a VM with an IPv4 floating IP assigned when DVR is used.  In this case, the floating IP "lives" on the compute node.  In all other cases (IPv6, no floating IP assigned, routing to a non-external network), the VM will lose connectivity when it doesn't have an L3 agent.

Comment 13 Miguel Angel Ajo 2017-04-05 14:06:38 UTC
(In reply to Ian Pilcher from comment #12)
> (In reply to marios from comment #8)
> > After a call with ajo today I think the premise behind the current upstream
> > effort is wrong. We apparently don't even need the l3 agents for the tenant
> > vm IPs. Sure, if all l3 agents are down you won't be able to _create_ new
> > IPs for example but the existing ones should be reachable OK.
> 
> I'm 99% sure that this is not correct.  An L3 agent is required for almost
> all "north/south" traffic -- i.e. a VM communicating with anything not on
> its own network.

That is wrong and a common missunderstanding :-), I explained it to marios 2 days ago, but let me summarise here for others: once the neutron-l3-agent and neutron-openvswitch-agent have configured the dataplane (which consist of network namespaces, processes, like keepalived running inside those namespaces, and openflow rules, iptables rules, etc..) even if you switch off the agents, the dataplane connectivity *should not be disrupted*, because we keep all those resources in the linux kernel and system, that is *by design*.

If you ever find such behaviour, please fill a bug to neutron.

What will fail when you have set the agents down, is the updating of routers (floating ips changing,new FIPS, new ports...) that won't be handled until the agents are up again and able to resync the system dataplane configuration to the intended state.


(In reply to marios from comment #11)
> Hey, so thanks for waiting - I ran an upgrade this morning with a pingtest
> going and it completed OK. AFAICS there was no problem with the ping this
> time - package versions & details below.

Nice!!

> 
> The environment had the 'manual' openvswitch upgrade patch [0] applied, with
> the rpm --flags openvswitch upgrade from ovs2.5->2.6 delivered as ansible
> steps (during the main controller upgrade step).
> 
> I deployed vanila OSP10 3control/1compute and start a tenant vm for pinging
> (I use [1]). Through the controller upgrade I was checking the ping with [2]
> and this time there was no loss, even when at some point during the upgrade
> all neutron-* were down including l3-agent.
> 
> I also saw that after openvswitch was upgraded to 2.6 there was no start to
> neutron-openvswitch-agent as discussed in comment #3. I am not sure if that
> is because of [0] and/or it is something that landed into packaging? Package
> versions are like [3]

Regarding that, nothing was added to packaging, that is pending on an RDO review here: https://review.rdoproject.org/r/#/c/5951/

It's likely to have happen because you're disabling the rpm upgrade scriptlets when you install the new openvswitch, so restart won't happen.

> 
> I noticed that the tenant router was hosted on the same controller
> before/after the upgrade (neutron l3-agent-list-hosting-router) indeed there
> was no failover as I didn't attempt to upgrade l3 agents one at a time. 

Perfect!

> So
> to be clear and to set expectations correctly, as we discussed this means
> that during the controller upgrade operator shouldn't expect to *manage*
> floating IPs, but, it looks like they should be able to reach them during
> the upgrade (and that is a huge improvement),

That is correct!


> 
> thanks, marios

thank *you*! :D

> 
> [0] https://review.openstack.org/#/c/434346
> 
> [1] if ! [[ -d ~/tripleo-ci ]]; then git clone
> http://github.com/openstack-infra/tripleo-ci.git ~/tripleo-ci ;fi;
> ~/tripleo-ci/scripts/tripleo.sh --overcloud-pingtest --skip-pingtest-cleanup
> # pingtest no cleanup.
> 
> [2] LOGFILE=PINGTEST_CONTROLLER_UPGRADE ; IP=192.0.2.53; while [ true ]; do
> if ping -c 1 $IP; then echo ""; echo "OK `date`"; sleep 3; else echo
> "************* NOPE ! ******************* `date`"; sleep 2; fi ; done 2>&1 |
> tee -a $LOGFILE
> 
> [3] [root@overcloud-controller-1 ~]# rpm -qa | grep "neutron\|openvswitch"
> puppet-neutron-10.3.0-2.el7ost.noarch
> python-neutron-lib-1.1.0-1.el7ost.noarch
> openstack-neutron-metering-agent-10.0.0-13.el7ost.noarch
> openstack-neutron-ml2-10.0.0-13.el7ost.noarch
> python-neutron-tests-10.0.0-13.el7ost.noarch
> python-neutronclient-6.1.0-1.el7ost.noarch
> openstack-neutron-10.0.0-13.el7ost.noarch
> openvswitch-2.6.1-10.git20161206.el7fdp.x86_64
> openstack-neutron-bigswitch-agent-10.0.3-2.el7ost.noarch
> python-neutron-lbaas-10.0.0-5.el7ost.noarch
> openstack-neutron-lbaas-10.0.0-5.el7ost.noarch
> python-neutron-10.0.0-13.el7ost.noarch
> openstack-neutron-sriov-nic-agent-10.0.0-13.el7ost.noarch
> openstack-neutron-bigswitch-lldp-10.0.3-2.el7ost.noarch
> python-openvswitch-2.6.1-10.git20161206.el7fdp.noarch
> openstack-neutron-openvswitch-10.0.0-13.el7ost.noarch
> openstack-neutron-common-10.0.0-13.el7ost.noarch

Comment 14 Marios Andreou 2017-04-05 14:29:05 UTC
(In reply to Ian Pilcher from comment #12)
> (In reply to marios from comment #8)
> > After a call with ajo today I think the premise behind the current upstream
> > effort is wrong. We apparently don't even need the l3 agents for the tenant
> > vm IPs. Sure, if all l3 agents are down you won't be able to _create_ new
> > IPs for example but the existing ones should be reachable OK.
> 
> I'm 99% sure that this is not correct.  An L3 agent is required for almost
> all "north/south" traffic -- i.e. a VM communicating with anything not on
> its own network.
> 
> The only exception of which I know is a VM with an IPv4 floating IP assigned
> when DVR is used.  In this case, the floating IP "lives" on the compute
> node.  In all other cases (IPv6, no floating IP assigned, routing to a
> non-external network), the VM will lose connectivity when it doesn't have an
> L3 agent.

thanks Ian, so it was our original understanding when filing this and the upstream bug that we should preserve the l3 agents by stopping only one at a a time (we were trying to do that with the gerrit review in the trackers above). I cc ajo for confirmation/more info please about this requirement in order to preserve tenant floating IP - as I said, my understanding about this changed to 'we don't need the l3 agent' after our conversation. 

My env is indeed IPv4 (vanilla osp10 single-nic-vlan - I didn't htink we had dvr enabled by default) and after what you said I went looking and found a vxlan_sys_4789 interface on my compute node which I think must be the 'floating IP' (I don't see the IP but tcpdump tells me it is getting the icmps).

Comment 15 Marios Andreou 2017-04-05 14:37:14 UTC
(In reply to marios from comment #14)
> (In reply to Ian Pilcher from comment #12)
> > (In reply to marios from comment #8)
> 
> thanks Ian, so it was our original understanding when filing this and the
> upstream bug that we should preserve the l3 agents by stopping only one at a
> a time (we were trying to do that with the gerrit review in the trackers
> above). I cc ajo for confirmation/more info please about this requirement in
> order to preserve tenant floating IP - as I said, my understanding about
> this changed to 'we don't need the l3 agent' after our conversation. 
> 

should have refreshed before posting! thanks to ajo for responding already above comment #13

Comment 16 Jakub Libosvar 2017-04-06 15:25:06 UTC
Marios, Miguel, is there any work needed in scope of this bugzilla? As per comments above it seems north-south traffic is not disrupted during upgrades, yay. Shall we close this bug?

Also Marios I have a question re. [1] - is there any workaround in tripleo now that avoids upgrading openvswitch ([2] ?)? Are we good to go with [1] or that would still lose connectivity to the nodes? The [1] is required to fix a blocker bug 1436021.

Thanks!

[1] https://review.rdoproject.org/r/#/c/6170/
[2] https://review.openstack.org/#/c/434346

Comment 17 Marios Andreou 2017-04-06 15:36:59 UTC
(In reply to Jakub Libosvar from comment #16)
> Marios, Miguel, is there any work needed in scope of this bugzilla? As per
> comments above it seems north-south traffic is not disrupted during
> upgrades, yay. Shall we close this bug?

yeah not close but let's let it go through the usual process and the QE folks can have at it for more testing.

> 
> Also Marios I have a question re. [1] - is there any workaround in tripleo
> now that avoids upgrading openvswitch ([2] ?)? Are we good to go with [1] or
> that would still lose connectivity to the nodes? The [1] is required to fix
> a blocker bug 1436021.

so there *is* a workaround but not one that *avoids* upgrading openvswitch but rather that upgrades it manually using rpm --flags (see details in [2]). At this point I can't comment on [1] (its about haproxy version constraint?)... unless you meant https://review.rdoproject.org/r/#/c/5951/? - but again I am not sure if we need that one at this point since n-ovs-agent wasn't started when ovs was upgraded using [2] as commented earlier. I don't see how it would hurt though? wdyt?



> 
> Thanks!
> 
> [1] https://review.rdoproject.org/r/#/c/6170/
> [2] https://review.openstack.org/#/c/434346

Comment 18 Jakub Libosvar 2017-04-06 15:50:13 UTC
(In reply to marios from comment #17)
> (In reply to Jakub Libosvar from comment #16)
> > Marios, Miguel, is there any work needed in scope of this bugzilla? As per
> > comments above it seems north-south traffic is not disrupted during
> > upgrades, yay. Shall we close this bug?
> 
> yeah not close but let's let it go through the usual process and the QE
> folks can have at it for more testing.
> 
> > 
> > Also Marios I have a question re. [1] - is there any workaround in tripleo
> > now that avoids upgrading openvswitch ([2] ?)? Are we good to go with [1] or
> > that would still lose connectivity to the nodes? The [1] is required to fix
> > a blocker bug 1436021.
> 
> so there *is* a workaround but not one that *avoids* upgrading openvswitch
> but rather that upgrades it manually using rpm --flags (see details in [2]).
> At this point I can't comment on [1] (its about haproxy version
> constraint?)... unless you meant https://review.rdoproject.org/r/#/c/5951/?
Oops, yeah, sorry for wrong link :) ofc I meant 5951.

> - but again I am not sure if we need that one at this point since
> n-ovs-agent wasn't started when ovs was upgraded using [2] as commented
> earlier. I don't see how it would hurt though? wdyt?

I just don't want to break tripleo upgrades by merging that before we understand why the connectivity gets lost. If it doesn't influence your upgrades, we can merge and backport it to OSP11.

Comment 24 Assaf Muller 2017-04-26 11:01:34 UTC
Just to clarify was there any code change as part of this RHBZ or is this test-only?

Comment 25 Marios Andreou 2017-04-26 11:19:50 UTC
(In reply to Assaf Muller from comment #24)
> Just to clarify was there any code change as part of this RHBZ or is this
> test-only?

no in the end it was test-only... well, there were 'some package fixes' if I understood correctly so there may have been some neutron -spec/packages changes but wrt Director the review linked in trackers was unnecessary and abandoned

Comment 26 Marius Cornea 2017-04-26 17:10:18 UTC
During major-upgrade-composable-steps.yaml, ping floating IP attached to an instance running on the overcloud from the undercloud:

stdout: 3190 packets transmitted, 3190 received, 0% packet loss, time 3190641ms
rtt min/avg/max/mdev = 0.453/0.893/7.504/0.273 ms

During major-upgrade-converge.yaml:

stdout: 1255 packets transmitted, 1255 received, 0% packet loss, time 1254706ms
rtt min/avg/max/mdev = 0.504/0.869/6.682/0.315 ms

Comment 27 errata-xmlrpc 2017-05-17 19:45:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.