Bug 1595230

I tried to reproduce this yesterday by pinging the floating IP while restarting the l3-agent on the controller hosting the master router.  I didn't see any packet loss so will go through the logs to see if I can piece something together.

I couldn't find l3-agent logs in the sosreport for all the controllers, but looking on controller-2, which is not the current master I do see this after it's restarted:

2018-06-26 11:20:57.586 13204 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-4bedd2a6-1c8f-4876-a9d5-411768c75d53', 'sysctl', '-w', 'net.ipv6.conf.all.forwarding=0']

2018-06-26 11:20:57.725 13204 INFO neutron.agent.l3.ha [-] Router 4bedd2a6-1c8f-4876-a9d5-411768c75d53 transitioned to master

2018-06-26 11:20:57.886 13204 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-4bedd2a6-1c8f-4876-a9d5-411768c75d53', 'sysctl', '-w', 'net.ipv6.conf.qg-3913e1a3-8f.forwarding=1']

2018-06-26 11:21:09.472 13204 INFO neutron.agent.l3.ha [-] Router 4bedd2a6-1c8f-4876-a9d5-411768c75d53 transitioned to backup

2018-06-26 11:21:09.501 13204 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-4bedd2a6-1c8f-4876-a9d5-411768c75d53', 'sysctl', '-w', 'net.ipv6.conf.qg-3913e1a3-8f.forwarding=0']

So it bounced the forwarding knob on and off, which was the issue that caused the failure originally.

But the router on controller-0 had already declared itself the master less than a minute before:

2018-06-26 11:19:44.463 12652 INFO neutron.agent.l3.ha [-] Router 4bedd2a6-1c8f-4876-a9d5-411768c75d53 transitioned to master

So the packet loss is most likely a result of the router on controller-2 transitioning from master to backup, since even that would cause an MLDv2 to get sent, causing a possible interruption for ~1 minutes.

Yurii recreated this condition again today, but looking on the three controllers they do not have the version of OSP 10 that has the fix:

openstack-neutron.noarch         1:9.4.1-19.el7ost

1:9.4.1-25.el7ost or later is required.  I know yesterday I did see -25 installed so maybe a minor update is required?  And then we can re-check things?

Also, one other thing that would be useful is to test a 10 -> 11 upgrade and see if this failure occurs again.  Thanks.

So I have a thought on what could have happened, although I think this would have affected other upgrades as well.

From the logs I can see that net.ipv6.conf.*.forwarding is getting changes, even on the standby routers:

2018-07-12 06:16:21.292 27186 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-d8abaffc-f1ed-42ff-97c0-a05db06756f8', 'sysctl', '-w', 'net.ipv6.conf.all.forwarding=1'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:106
2018-07-12 07:26:49.823 17470 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-d8abaffc-f1ed-42ff-97c0-a05db06756f8', 'sysctl', '-w', 'net.ipv6.conf.all.forwarding=0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2018-07-12 07:26:50.132 17470 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-d8abaffc-f1ed-42ff-97c0-a05db06756f8', 'sysctl', '-w', 'net.ipv6.conf.qg-1a2c96d7-d5.forwarding=0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2018-07-12 09:09:33.280 73628 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-d8abaffc-f1ed-42ff-97c0-a05db06756f8', 'sysctl', '-w', 'net.ipv6.conf.all.forwarding=0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2018-07-12 09:09:33.913 73628 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-d8abaffc-f1ed-42ff-97c0-a05db06756f8', 'sysctl', '-w', 'net.ipv6.conf.qg-1a2c96d7-d5.forwarding=0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105

Even that switch from 1 to 0 at 7:26:49.823 could have caused a disruption, although I'm not sure if that's the time where the packets were lost, or if it was at 9:09 ?

Just for testing I did flip the all.forwarding on one of the standby routers and saw packet loss.  The good thing is that this shouldn't happen going forward, since all the backups have these at 0 now, but I would have figured an upgrade from 10 to 13 would have seen this too, unless the 10 setup hadn't applied this patch yet.

This is what I was trying to do with https://review.openstack.org/#/c/569458/ - don't ever change that value from 1 to 0 if possible.  I stopped working on that once the other patch seemed to fix the issue, but can pick it up again since it might fix this edge case.

I actually think this was fixed in a later neutron patch, just never updated this but.  Added that additional information.

According to our records, this should be resolved by openstack-neutron-9.4.1-48.el7ost.  This build is available now.

According to DFG decision, the priority of this bug is low.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4298