Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1807638 - [4.5]Minimize disruption of new and existing connections while OVS is being upgraded [NEEDINFO]
Summary: [4.5]Minimize disruption of new and existing connections while OVS is being u...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.5.0
Assignee: Aniket Bhat
QA Contact: zhaozhanqi
URL:
Whiteboard:
: 1803188 (view as bug list)
Depends On:
Blocks: 1807648
TreeView+ depends on / blocked
 
Reported: 2020-02-26 19:53 UTC by Clayton Coleman
Modified: 2020-07-29 16:05 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1807648 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:21:42 UTC
Target Upstream Version:
igreen: needinfo? (ccoleman)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-network-operator pull 476 0 None closed Bug 1807638: Minimize disruption to pod network while OVS is being upgraded 2020-07-29 15:44:51 UTC
Github openshift cluster-network-operator pull 524 0 None closed Bug 1807638: Fixes to reliably save/restore flows. 2020-07-29 15:44:51 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:22:11 UTC

Description Clayton Coleman 2020-02-26 19:53:40 UTC
During OVS shutdown and startup new and reused connections are being impacted (in upgrade).

We should attempt to preserve all existing flows in the kernel while the OVS daemon is offline, and during startup avoid clearing those flows.

While this does not completely mitigate dropped / failed connections while OVS is upgrading, it dramatically reduces the amount of time user applications are impacted both by upgrades or by unexpected disruption (an OOM kill or OVS crash). Future changes will build on this to try to make OVS upgrade completely transparent to end user applications.

Known gaps:

ARP flows are potentially still being lost for new connections (workarounds being investigated with OVS team).

After testing in 4.5, we will consider back porting this to all active releases.

Comment 3 zhaozhanqi 2020-03-06 08:46:24 UTC
run this on 4.5.0-0.nightly-2020-03-05-190442

seems this not work well using the current version, see following:

#oc rsh -n openshift-sdn ovs-c9mmk
sh-4.2#/usr/share/openvswitch/scripts/ovs-save save-flows br0
ovs-ofctl add-tlv-map br0 '2020-03-06T08:38:41Z|00001|vconn|WARN|unix:/var/run/openvswitch/br0.mgmt: version negotiation failed (we support version 0x01, peer supports version 0x04)
ovs-ofctl: br0: failed to connect to socket (Broken pipe)
'
ovs-ofctl -O OpenFlow13 add-groups br0               "/tmp/ovs-save.Nk93bxOLHn/br0.groups.dump" 
ovs-ofctl -O OpenFlow13 replace-flows br0               "/tmp/ovs-save.Nk93bxOLHn/br0.flows.dump" 
rm -rf "/tmp/ovs-save.Nk93bxOLHn"

Comment 6 zhaozhanqi 2020-04-01 07:37:24 UTC
verified this bug on 4.5.0-0.nightly-2020-04-01-015139

Comment 7 Tomas Smetana 2020-05-26 13:29:25 UTC
*** Bug 1803188 has been marked as a duplicate of this bug. ***

Comment 9 errata-xmlrpc 2020-07-13 17:21:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.