Bug 2089416
| Summary: | data plane downtime during the first flow installation. | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Siggy Sigwald <ssigwald> | |
| Component: | ovn-2021 | Assignee: | Mark Michelson <mmichels> | |
| Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | FDP 22.D | CC: | apevec, astupnik, ctrautma, dalvarez, dhill, froyo, fwissing, jiji, jlibosva, jpretori, jveiraca, kurathod, lhh, ltamagno, majopela, matteo.panella, mmichels, sathlang, scohen, shtiwari, twilson, ykarel | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | ovn-2021-21.12.0-93.el8fdp | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2127166 (view as bug list) | Environment: | ||
| Last Closed: | 2022-11-21 18:33:59 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2094265, 2111871, 2141873 | |||
When upgrading from 16.1.6 to 16.2.latest is a hotfix described here necessary to avoid data plane interruption, or is this going into errata within the short term? Hi all, this bug will be listed in the 22.K errata for OVN. I'm making the errata currently. Tested with script in https://bugzilla.redhat.com/show_bug.cgi?id=2139425#c3, reproduced on ovn-2021-21.12.0-82.el8: + ip netns exec vm1 ping 172.16.0.102 -c 1 PING 172.16.0.102 (172.16.0.102) 56(84) bytes of data. 64 bytes from 172.16.0.102: icmp_seq=1 ttl=62 time=22.9 ms --- 172.16.0.102 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 22.900/22.900/22.900/0.000 ms + ip netns exec vm1 ping 172.16.0.100 -c 1 PING 172.16.0.100 (172.16.0.100) 56(84) bytes of data. 64 bytes from 172.16.0.100: icmp_seq=1 ttl=63 time=3.43 ms --- 172.16.0.100 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 3.426/3.426/3.426/0.000 ms + ovs-vsctl set open . external_ids:ovn-ofctrl-wait-before-clear=7000 + systemctl restart ovn-controller + ip netns exec vm1 ping 172.16.0.102 -c 300 -i 0.1 + wait + tail ping.log 64 bytes from 172.16.0.102: icmp_seq=295 ttl=62 time=0.022 ms 64 bytes from 172.16.0.102: icmp_seq=296 ttl=62 time=0.023 ms 64 bytes from 172.16.0.102: icmp_seq=297 ttl=62 time=0.024 ms 64 bytes from 172.16.0.102: icmp_seq=298 ttl=62 time=0.023 ms 64 bytes from 172.16.0.102: icmp_seq=299 ttl=62 time=0.022 ms 64 bytes from 172.16.0.102: icmp_seq=300 ttl=62 time=0.022 ms --- 172.16.0.102 ping statistics --- 300 packets transmitted, 180 received, 40% packet loss, time 31095ms rtt min/avg/max/mdev = 0.022/0.047/1.448/0.118 ms 64 bytes from 172.16.0.102: icmp_seq=31 ttl=62 time=0.060 ms 64 bytes from 172.16.0.102: icmp_seq=32 ttl=62 time=0.078 ms 64 bytes from 172.16.0.102: icmp_seq=153 ttl=62 time=1.45 ms 64 bytes from 172.16.0.102: icmp_seq=154 ttl=62 time=0.289 ms [root@dell-per750-18 bz2139425]# rpm -qa | grep -E "openvswitch2.15|ovn-2021" ovn-2021-21.12.0-82.el8fdp.x86_64 openvswitch2.15-2.15.0-127.el8fdp.x86_64 ovn-2021-host-21.12.0-82.el8fdp.x86_64 ovn-2021-central-21.12.0-82.el8fdp.x86_64 Verified on ovn-2021-21.12.0-94.el8: + ip netns exec vm1 ping 172.16.0.102 -c 1 PING 172.16.0.102 (172.16.0.102) 56(84) bytes of data. 64 bytes from 172.16.0.102: icmp_seq=1 ttl=62 time=17.2 ms --- 172.16.0.102 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 17.237/17.237/17.237/0.000 ms + ip netns exec vm1 ping 172.16.0.100 -c 1 PING 172.16.0.100 (172.16.0.100) 56(84) bytes of data. 64 bytes from 172.16.0.100: icmp_seq=1 ttl=63 time=3.99 ms --- 172.16.0.100 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 3.992/3.992/3.992/0.000 ms + ovs-vsctl set open . external_ids:ovn-ofctrl-wait-before-clear=7000 + systemctl restart ovn-controller + ip netns exec vm1 ping 172.16.0.102 -c 300 -i 0.1 + wait + tail ping.log 64 bytes from 172.16.0.102: icmp_seq=295 ttl=62 time=0.022 ms 64 bytes from 172.16.0.102: icmp_seq=296 ttl=62 time=0.023 ms 64 bytes from 172.16.0.102: icmp_seq=297 ttl=62 time=0.022 ms 64 bytes from 172.16.0.102: icmp_seq=298 ttl=62 time=0.023 ms 64 bytes from 172.16.0.102: icmp_seq=299 ttl=62 time=0.022 ms 64 bytes from 172.16.0.102: icmp_seq=300 ttl=62 time=0.025 ms --- 172.16.0.102 ping statistics --- 300 packets transmitted, 300 received, 0% packet loss, time 31094ms rtt min/avg/max/mdev = 0.013/0.053/1.399/0.093 ms [root@dell-per750-18 bz2139425]# rpm -qa | grep -E "openvswitch2.15|ovn-2021" ovn-2021-21.12.0-94.el8fdp.x86_64 openvswitch2.15-2.15.0-127.el8fdp.x86_64 ovn-2021-central-21.12.0-94.el8fdp.x86_64 ovn-2021-host-21.12.0-94.el8fdp.x86_64 Hi Mark, Does ovn22.06 and ovn22.09 support this feature? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn-2021 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8568 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
Description of problem: During our last OpenStack update from 16.1 to 16.2, we encountered a network dataplane outage on instances at step 3.3 from the documentation [2]. It was detected using a ping on multiple instances and lasted 1 or 2 minutes. We found two OVN commits that seems relevant to this behaviour : https://github.com/ovn-org/ovn/commit/896adfd2d8b3369110e9618bd190d190105372a9 https://github.com/ovn-org/ovn/commit/d53c599ed05ea3c708a045a9434875458effa21e We hope these patches will be soon backported into RHOSP OVN to avoid this issue for the next upgrades. This outage had a big impact for some of our clients, especially those using Kubernetes clusters as nodes were failing and pods were massively re-scheduled which also led to high CPU usage on compute nodes. [2] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/keeping_red_hat_openstack_platform_updated/index#proc_updating-ovn-controller-container_updating-overcloud