Bug 1570394 - [3.9] openshift-sdn partially deletes openflow rules leading to "no route to host" for service
Summary: [3.9] openshift-sdn partially deletes openflow rules leading to "no route to ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.9.z
Assignee: Dan Winship
QA Contact: Hongan Li
URL:
Whiteboard:
Depends On: 1538220
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-22 14:46 UTC by Dan Winship
Modified: 2018-05-17 06:44 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Incorrect management of OVS flows Consequence: If two nodes rebooted and swapped IP addresses when they came back up, then other nodes might become unable to send traffic to pods on one or both of those nodes. Fix: The code that manages OVS flows is now more careful to make the correct changes in cases of node IP reassignment. Result: Pod-to-pod traffic should continue to work correctly even after nodes swap IP addresses.
Clone Of: 1538220
: 1570395 1570396 (view as bug list)
Environment:
Last Closed: 2018-05-17 06:43:40 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ose/pull/1215 0 None None None 2020-05-22 14:05:33 UTC
Red Hat Product Errata RHBA-2018:1566 0 None None None 2018-05-17 06:44:00 UTC

Comment 1 Dan Winship 2018-04-22 14:49:17 UTC 
https://github.com/openshift/ose/pull/1215
Comment 5 Hongan Li 2018-05-03 10:03:14 UTC 
cannot reproduce the problem in old version, and verified in atomic-openshift-3.9.27-1.git.0.964617d that the ovs flow has been updated as below:

 cookie=0xe216df8b, duration=30407.563s, table=10, n_packets=96649, n_bytes=19580171, priority=100,tun_src=10.240.0.10 actions=goto_table:30
 cookie=0x6d3e6ac, duration=12016.153s, table=10, n_packets=6434, n_bytes=4920683, priority=100,tun_src=10.240.0.11 actions=goto_table:30
 cookie=0xe216df8b, duration=30407.554s, table=50, n_packets=1237, n_bytes=51954, priority=100,arp,arp_tpa=10.130.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.10->tun_dst,output:1
 cookie=0x6d3e6ac, duration=12016.135s, table=50, n_packets=163, n_bytes=6846, priority=100,arp,arp_tpa=10.129.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.11->tun_dst,output:1
 cookie=0xe216df8b, duration=30407.548s, table=90, n_packets=97799, n_bytes=8442205, priority=100,ip,nw_dst=10.130.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.10->tun_dst,output:1
 cookie=0x6d3e6ac, duration=12016.110s, table=90, n_packets=6851, n_bytes=893681, priority=100,ip,nw_dst=10.129.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.11->tun_dst,output:1

Each ovs flow for other node's IP has a random cookie now. Did some related regression test such as delete/add/reboot nodes and no issues found.


OS: Red Hat Enterprise Linux Server release 7.5 (Maipo)
Kernel: 3.10.0-862.el7.x86_64
Comment 6 Dan Winship 2018-05-03 11:27:37 UTC 
> cannot reproduce the problem in old version

Yeah, it's a race condition, and it might only happen when the nodes are slow/overloaded.

FWIW, I did verify that the new unit test TestHostSubnetReassignment() *does* fail when run against the old ovscontroller.go code.
Comment 9 errata-xmlrpc 2018-05-17 06:43:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566
Note You need to log in before you can comment on or make changes to this bug.