In an scenario with VIPs that are balanced at application level (by for example, keepalived) there could be issues at such level which leads to the VIP flapping very frequently. This will make the app to send the gARPs announcing the location of the new IP address which will trigger updates of MAC_Binding entries very frequently if the ports flap at high rate. The result is that ovn-northd will stay at 100% CPU and ovsdb-server for SB database will be very loaded. Just like normal switches have, it would be great if OVN could implement some protection mechanism that when such condition is detected, the MAC address is banned from the switch in one or more ports to prevent this situation from happening.
COPP upstream series posted: http://patchwork.ozlabs.org/project/ovn/list/?series=241400&state=%2A&archive=both
upstream series: http://patchwork.ozlabs.org/project/ovn/cover/cover.1627052896.git.lorenzo.bianconi@redhat.com/
upstream support: http://patchwork.ozlabs.org/project/ovn/cover/cover.1627052896.git.lorenzo.bianconi@redhat.com/
IIRC this support landed in OVN 21.09 mid 2021.
as comment8 said, it is included in https://errata.devel.redhat.com/advisory/96200, it's for ovn-2021, I tried on version ovn-2021-21.12.0-73.el8fdp.x86_64 the command for copp is not the same as that in ovn22.03. I want to make sure I should verify this bug in ovn22.03 or ovn-2021?
(In reply to ying xu from comment #9) > > as comment8 said, it is included in > https://errata.devel.redhat.com/advisory/96200, it's for ovn-2021, > I tried on version ovn-2021-21.12.0-73.el8fdp.x86_64 > the command for copp is not the same as that in ovn22.03. > I want to make sure I should verify this bug in ovn22.03 or ovn-2021? I would suggest to test both of them, it is just a tiny change I guess.
(In reply to lorenzo bianconi from comment #10) > (In reply to ying xu from comment #9) > > > > as comment8 said, it is included in > > https://errata.devel.redhat.com/advisory/96200, it's for ovn-2021, > > I tried on version ovn-2021-21.12.0-73.el8fdp.x86_64 > > the command for copp is not the same as that in ovn22.03. > > I want to make sure I should verify this bug in ovn22.03 or ovn-2021? > > I would suggest to test both of them, it is just a tiny change I guess. but why the commands are not the same for these two versions? I think it is not friendly and not compatiable.
verified on version: ovn-2021-21.12.0-73.el8fdp.x86_64 ovn22.03-22.03.0-52.el8fdp.x86_64 although there are some differences between the two versions, the test procedure is the same. :: [ 06:17:13 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w reject1.pcap& ' :: [ 06:17:13 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w reject1.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:17:15 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' WARNING: Interface lo: no address assigned :: [ 06:17:16 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' (Expected 0, got 0) 28 packets captured 29 packets received by filter 0 packets dropped by kernel reading from file reject.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:17:20 ] :: [ PASS ] :: reject packets count:1 should be 1 (Assert: '1' should equal '1') SYNC_NC: sync_set client ovn_copp SYNC_NC: sent "ovn_copp" to dell-per740-54.rhts.eng.pek2.redhat.com SYNC_NC: sync_wait client ovn_copp SYNC_NC: waiting "dell-per740-54.rhts.eng.pek2.redhat.com" SYNC_NC: got "ovn_copp" from dell-per740-54.rhts.eng.pek2.redhat.com :: [ 06:17:31 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w reject10.pcap& ' :: [ 06:17:31 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w reject10.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:17:33 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' WARNING: Interface lo: no address assigned :: [ 06:17:34 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' (Expected 0, got 0) 32 packets captured 32 packets received by filter 0 packets dropped by kernel reading from file reject10.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:17:38 ] :: [ PASS ] :: reject packets count:10 should be 10 (Assert: '10' should equal '10') SYNC_NC: sync_set client ovn_copp SYNC_NC: sent "ovn_copp" to dell-per740-54.rhts.eng.pek2.redhat.com SYNC_NC: sync_wait client ovn_copp SYNC_NC: waiting "dell-per740-54.rhts.eng.pek2.redhat.com" SYNC_NC: got "ovn_copp" from dell-per740-54.rhts.eng.pek2.redhat.com :: [ 06:17:49 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv copp-del copp0 reject' :: [ 06:17:49 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv copp-del copp0 reject' (Expected 0, got 0) :: [ 06:17:49 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w rejectno.pcap& ' :: [ 06:17:49 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w rejectno.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:17:51 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' WARNING: Interface lo: no address assigned :: [ 06:17:52 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' (Expected 0, got 0) 42 packets captured 42 packets received by filter 0 packets dropped by kernel reading from file rejectno.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:17:56 ] :: [ PASS ] :: reject packets count:20 should be 20 (Assert: '20' should equal '20') :: [ 06:18:22 ] :: [ BEGIN ] :: Running 'ovn-nbctl meter-add icmp-meter drop 1 pktps 0' :: [ 06:18:22 ] :: [ PASS ] :: Command 'ovn-nbctl meter-add icmp-meter drop 1 pktps 0' (Expected 0, got 0) :: [ 06:18:22 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv copp-add copp2 icmp4-error icmp-meter' :: [ 06:18:22 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv copp-add copp2 icmp4-error icmp-meter' (Expected 0, got 0) :: [ 06:18:22 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv lr-copp-add copp2 lr1' :: [ 06:18:22 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv lr-copp-add copp2 lr1' (Expected 0, got 0) :: [ 06:18:22 ] :: [ BEGIN ] :: Running 'ovn-nbctl copp-list copp2 |grep icmp4-error' icmp4-error: icmp-meter :: [ 06:18:22 ] :: [ PASS ] :: Command 'ovn-nbctl copp-list copp2 |grep icmp4-error' (Expected 0, got 0) :: [ 06:18:22 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w icmperror.pcap& ' :: [ 06:18:22 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w icmperror.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:18:24 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py' WARNING: Interface lo: no address assigned :: [ 06:18:25 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py' (Expected 0, got 0) 23 packets captured 23 packets received by filter 0 packets dropped by kernel reading from file icmperror.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:18:29 ] :: [ PASS ] :: ICMP packets count:1 should be 1 (Assert: '1' should equal '1') :: [ 06:18:29 ] :: [ BEGIN ] :: Running 'tcpdump -r icmperror.pcap -nnle' reading from file icmperror.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump 06:18:25.309472 Out 00:00:01:01:01:01 ethertype ARP (0x0806), length 44: Request who-has 192.168.1.254 tell 192.168.1.2, length 28 06:18:25.309619 In 00:00:01:0f:01:02 ethertype ARP (0x0806), length 44: Reply 192.168.1.254 is-at 00:00:01:0f:01:02, length 28 06:18:25.324666 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.327465 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.330200 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.331406 In 00:00:01:0f:01:02 ethertype IPv4 (0x0800), length 148: 192.168.1.254 > 192.168.1.2: ICMP time exceeded in-transit, length 112 06:18:25.332928 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.335530 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.336832 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.338115 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.339397 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.340687 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.341983 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.343278 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.344337 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.345497 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.346488 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.347482 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.348478 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.349473 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.350462 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.351455 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP 06:18:25.352461 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP :: [ 06:18:29 ] :: [ PASS ] :: Command 'tcpdump -r icmperror.pcap -nnle' (Expected 0, got 0) :: [ 06:18:29 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv copp-del copp2 icmp4-error' :: [ 06:18:29 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv copp-del copp2 icmp4-error' (Expected 0, got 0) :: [ 06:18:29 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w icmperror.pcap& ' :: [ 06:18:29 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w icmperror.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:18:31 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py' WARNING: Interface lo: no address assigned :: [ 06:18:32 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py' (Expected 0, got 0) 42 packets captured 42 packets received by filter 0 packets dropped by kernel reading from file icmperror.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:18:36 ] :: [ PASS ] :: ICMP packets count:20 should be 20 (Assert: '20' should equal '20') :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl meter-add sctp-meter drop 1 pktps 0' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl meter-add sctp-meter drop 1 pktps 0' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv copp-add copp3 reject sctp-meter' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv copp-add copp3 reject sctp-meter' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv ls-copp-add copp3 ls1' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv ls-copp-add copp3 ls1' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl lb-add lb 30.0.0.1:8000 ""' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl lb-add lb 30.0.0.1:8000 ""' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d protocol=sctp' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d protocol=sctp' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl set logical_switch ls1 load_balancer=3c69661c-ad6c-4c19-9226-cc863445996d' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl set logical_switch ls1 load_balancer=3c69661c-ad6c-4c19-9226-cc863445996d' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d options:reject=true' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d options:reject=true' (Expected 0, got 0) :: [ 06:18:36 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w sctp.pcap& ' :: [ 06:18:36 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w sctp.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:18:38 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py' WARNING: Interface lo: no address assigned :: [ 06:18:39 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py' (Expected 0, got 0) 23 packets captured 23 packets received by filter 0 packets dropped by kernel reading from file sctp.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:18:43 ] :: [ PASS ] :: SCTP ABORT packets count:1 should be 1 (Assert: '1' should equal '1') :: [ 06:18:43 ] :: [ BEGIN ] :: Running 'ovn-nbctl --wait=hv copp-del copp3 reject' :: [ 06:18:43 ] :: [ PASS ] :: Command 'ovn-nbctl --wait=hv copp-del copp3 reject' (Expected 0, got 0) :: [ 06:18:43 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any -w sctp.pcap& ' :: [ 06:18:43 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any -w sctp.pcap& ' (Expected 0, got 0) dropped privs to tcpdump tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes :: [ 06:18:45 ] :: [ BEGIN ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py' WARNING: Interface lo: no address assigned :: [ 06:18:46 ] :: [ PASS ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py' (Expected 0, got 0) 42 packets captured 42 packets received by filter 0 packets dropped by kernel reading from file sctp.pcap, link-type LINUX_SLL (Linux cooked v1) dropped privs to tcpdump :: [ 06:18:50 ] :: [ PASS ] :: SCTP ABORT packets count:20 should be 20 (Assert: '20' should equal '20')
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:5446
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days