Bug 1570394
| Summary: | [3.9] openshift-sdn partially deletes openflow rules leading to "no route to host" for service | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Dan Winship <danw> | |
| Component: | Networking | Assignee: | Dan Winship <danw> | |
| Status: | CLOSED ERRATA | QA Contact: | Hongan Li <hongli> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 3.9.0 | CC: | aos-bugs, bbennett, hongli | |
| Target Milestone: | --- | |||
| Target Release: | 3.9.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: Incorrect management of OVS flows
Consequence: If two nodes rebooted and swapped IP addresses when they came back up, then other nodes might become unable to send traffic to pods on one or both of those nodes.
Fix: The code that manages OVS flows is now more careful to make the correct changes in cases of node IP reassignment.
Result: Pod-to-pod traffic should continue to work correctly even after nodes swap IP addresses.
|
Story Points: | --- | |
| Clone Of: | 1538220 | |||
| : | 1570395 1570396 (view as bug list) | Environment: | ||
| Last Closed: | 2018-05-17 06:43:40 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1538220 | |||
| Bug Blocks: | ||||
|
Comment 1
Dan Winship
2018-04-22 14:49:17 UTC
cannot reproduce the problem in old version, and verified in atomic-openshift-3.9.27-1.git.0.964617d that the ovs flow has been updated as below: cookie=0xe216df8b, duration=30407.563s, table=10, n_packets=96649, n_bytes=19580171, priority=100,tun_src=10.240.0.10 actions=goto_table:30 cookie=0x6d3e6ac, duration=12016.153s, table=10, n_packets=6434, n_bytes=4920683, priority=100,tun_src=10.240.0.11 actions=goto_table:30 cookie=0xe216df8b, duration=30407.554s, table=50, n_packets=1237, n_bytes=51954, priority=100,arp,arp_tpa=10.130.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.10->tun_dst,output:1 cookie=0x6d3e6ac, duration=12016.135s, table=50, n_packets=163, n_bytes=6846, priority=100,arp,arp_tpa=10.129.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.11->tun_dst,output:1 cookie=0xe216df8b, duration=30407.548s, table=90, n_packets=97799, n_bytes=8442205, priority=100,ip,nw_dst=10.130.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.10->tun_dst,output:1 cookie=0x6d3e6ac, duration=12016.110s, table=90, n_packets=6851, n_bytes=893681, priority=100,ip,nw_dst=10.129.0.0/16 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.240.0.11->tun_dst,output:1 Each ovs flow for other node's IP has a random cookie now. Did some related regression test such as delete/add/reboot nodes and no issues found. OS: Red Hat Enterprise Linux Server release 7.5 (Maipo) Kernel: 3.10.0-862.el7.x86_64 > cannot reproduce the problem in old version
Yeah, it's a race condition, and it might only happen when the nodes are slow/overloaded.
FWIW, I did verify that the new unit test TestHostSubnetReassignment() *does* fail when run against the old ovscontroller.go code.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1566 |