Bug 1858191
| Summary: | the connection to load balancer VIP breaks when ACL is added | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Jianlin Shi <jishi> |
| Component: | ovn2.13 | Assignee: | Numan Siddique <nusiddiq> |
| Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | FDP 20.E | CC: | ctrautma, jishi, ltomasbo, mdemaced, nusiddiq, ralongi |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-08-18 11:23:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1857865 | ||
Verified on ovn2.13-20.06.1-4.el8fdp.x86_64:
+ ip netns exec sw0p2 ping -c3 20.0.0.3
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=1.35 ms
64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1061 ms
64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.076 ms
--- 20.0.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 63ms
rtt min/avg/max/mdev = 0.076/354.211/1061.205/499.920 ms, pipe 2
+ ip netns exec sw0p1 nc -vz 20.0.0.4 80
+ ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Listening on 20.0.0.4:80
Ncat: Connected to 20.0.0.4:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:45528.
NCAT DEBUG: EOF on stdin
NCAT DEBUG: Closing fd 4.
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:50484.
NCAT DEBUG: Closing fd 4.
+ ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:50486.
NCAT DEBUG: Closing fd 4.
+ ovn-nbctl clear logical_switch sw1 acls
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:50488.
NCAT DEBUG: Closing fd 4.
[root@dell-per740-12 bz1858191]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
openvswitch2.13-2.13.0-48.el8fdp.x86_64
ovn2.13-host-20.06.1-4.el8fdp.x86_64
ovn2.13-20.06.1-4.el8fdp.x86_64
ovn2.13-central-20.06.1-4.el8fdp.x86_64
also Verified on ovn2.13-20.06.1-4.el7fdp.x86_64:
+ ip netns exec sw0p2 ping -c3 20.0.0.3
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=3.22 ms
64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1003 ms
64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.069 ms
--- 20.0.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.069/335.482/1003.153/472.116 ms, pipe 2
+ ip netns exec sw0p1 nc -vz 20.0.0.4 80
+ ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Listening on 20.0.0.4:80
Ncat: Connected to 20.0.0.4:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:58956.
NCAT DEBUG: EOF on stdin
NCAT DEBUG: Closing fd 4.
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:51278.
NCAT DEBUG: Closing fd 4.
+ ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:51280.
NCAT DEBUG: Closing fd 4.
+ ovn-nbctl clear logical_switch sw1 acls
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:51282.
NCAT DEBUG: Closing fd 4.
[root@dell-per740-42 bz1858191]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-39.el7fdp.x86_64
ovn2.13-20.06.1-4.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch
ovn2.13-central-20.06.1-4.el7fdp.x86_64
ovn2.13-host-20.06.1-4.el7fdp.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3488 |
Description of problem: the connection to load balancer VIP breaks when ACL is added Version-Release number of selected component (if applicable): ovn2.13-20.06.1-2.el8fdp.x86_64 How reproducible: Always Steps to Reproduce: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.111.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.111.25 systemctl restart ovn-controller ovn-nbctl ls-add sw0 ovn-nbctl lsp-add sw0 sw0-port1 ovn-nbctl lsp-set-addresses sw0-port1 "10:54:00:00:00:03 10.0.0.3" ovn-nbctl lsp-add sw0 sw0-port2 ovn-nbctl lsp-set-addresses sw0-port2 "10:54:00:00:00:04 10.0.0.4" ovn-nbctl ls-add sw1 ovn-nbctl lsp-add sw1 sw1-port1 sw0-port1 10 ovn-nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03 20.0.0.3" ovn-nbctl lsp-add sw1 sw1-port2 sw0-port2 20 ovn-nbctl lsp-set-addresses sw1-port2 "40:54:00:00:00:04 20.0.0.4" ovn-nbctl lr-add lr0 ovn-nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24 ovn-nbctl lsp-add sw1 sw1-lr0 ovn-nbctl lsp-set-type sw1-lr0 router ovn-nbctl lsp-set-addresses sw1-lr0 router ovn-nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1 ovn-nbctl ls-add sw2 ovn-nbctl lsp-add sw2 sw2-port1 ovn-nbctl lsp-set-addresses sw2-port1 "50:54:00:00:00:03 30.0.0.3" ovn-nbctl lrp-add lr0 lr0-sw2 00:00:00:00:ff:03 30.0.0.1/24 ovn-nbctl lsp-add sw2 sw2-lr0 ovn-nbctl lsp-set-type sw2-lr0 router ovn-nbctl lsp-set-addresses sw2-lr0 router ovn-nbctl lsp-set-options sw2-lr0 router-port=lr0-sw2 ovn-nbctl lb-add lb0 "30.0.0.10:80" "20.0.0.4:80" ovn-nbctl ls-lb-add sw1 lb0 ovn-nbctl ls-lb-add sw2 lb0 ovn-nbctl lr-lb-add lr0 lb0 ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal ip netns add sw0p1 ip link set sw0p1 netns sw0p1 ip netns exec sw0p1 ip link set lo up ip netns exec sw0p1 ip link set sw0p1 up ip netns exec sw0p1 ip link set sw0p1 address 10:54:00:00:00:03 ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1 ip netns exec sw0p1 ip route add default via 10.0.0.1 dev sw0p1 ovs-vsctl set Interface sw0p1 external_ids:iface-id=sw0-port1 # Create the interface for lport sw1-port1 ip netns exec sw0p1 ip link add link sw0p1 name sw1p1 type vlan id 10 ip netns exec sw0p1 ip link set sw1p1 address 40:54:00:00:00:03 ip netns exec sw0p1 ip link set sw1p1 up ip netns exec sw0p1 ip addr add 20.0.0.3/24 dev sw1p1 ip netns exec sw0p1 ip route delete default via 10.0.0.1 dev sw0p1 ip netns exec sw0p1 ip route add default via 20.0.0.1 dev sw1p1 ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal ip netns add sw0p2 ip link set sw0p2 netns sw0p2 ip netns exec sw0p2 ip link set lo up ip netns exec sw0p2 ip link set sw0p2 up ip netns exec sw0p2 ip link set sw0p2 address 10:54:00:00:00:04 ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2 ip netns exec sw0p2 ip route add default via 10.0.0.1 dev sw0p2 ovs-vsctl set Interface sw0p2 external_ids:iface-id=sw0-port2 # Create the interface for lport sw1-port2 ip netns exec sw0p2 ip link add link sw0p2 name sw1p2 type vlan id 20 ip netns exec sw0p2 ip link set sw1p2 address 40:54:00:00:00:04 ip netns exec sw0p2 ip link set sw1p2 up ip netns exec sw0p2 ip addr add 20.0.0.4/24 dev sw1p2 ip netns exec sw0p2 ip route delete default via 10.0.0.1 dev sw0p2 ip netns exec sw0p2 ip route add default via 20.0.0.1 dev sw1p2 ip netns exec sw0p2 ping -c3 20.0.0.3 # Start nc server on sw1p2 (sw0p2 is the parent) ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv & # connect to sw1p2 (which is the backend of LB VIP 30.0.0.10) from sw1p1 using nc client. It works. ip netns exec sw0p1 nc -vz 20.0.0.4 80 # connect to LB VIP from sw1p1 using nc client. It works. ip netns exec sw0p1 nc -vz 30.0.0.10 80 # Add the below ACL on sw1 ovn-nbctl acl-add sw1 to-lport 2002 "ip" allow-related # connect to LB VIP from sw1p1 using nc client. It doesn't work now. ip netns exec sw0p1 nc -vz 30.0.0.10 80 # Clear the ACL and it works again ovn-nbctl clear logical_switch sw1 acls ip netns exec sw0p1 nc -vz 30.0.0.10 80 Actual results: + ip netns exec sw0p2 ping -c3 20.0.0.3 PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data. 64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1.44 ms 64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=0.085 ms 64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.069 ms --- 20.0.0.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 41ms rtt min/avg/max/mdev = 0.069/0.530/1.438/0.642 ms + ip netns exec sw0p1 nc -vz 20.0.0.4 80 + ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Listening on 20.0.0.4:80 Ncat: Connected to 20.0.0.4:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:38964. NCAT DEBUG: EOF on stdin NCAT DEBUG: Closing fd 4. + ip netns exec sw0p1 nc -vz 30.0.0.10 80 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 30.0.0.10:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:35544. NCAT DEBUG: Closing fd 4. + ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related + ip netns exec sw0p1 nc -vz 30.0.0.10 80 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connection timed out. <=== times out here + ovn-nbctl clear logical_switch sw1 acls + ip netns exec sw0p1 nc -vz 30.0.0.10 80 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 30.0.0.10:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:35548. NCAT DEBUG: Closing fd 4. [root@hp-dl380pg8-12 test]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-41.el8fdb.x86_64 ovn2.13-central-20.06.1-2.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch ovn2.13-host-20.06.1-2.el8fdp.x86_64 ovn2.13-20.06.1-2.el8fdp.x86_64 [root@hp-dl380pg8-12 test]# grep failed /var/log/openvswitch/ovs-vswitchd.log 2020-07-17T07:41:35.462Z|00001|dpif(handler1)|WARN|system@ovs-system: execute ct(commit,label=0/0x1),push_vlan(vid=20,pcp=0),3 failed (Invalid argument) on packet tcp,vlan_tci=0x0000,dl_src=00:00:00:00:ff:02,dl_dst=40:54:00:00:00:04,nw_src=20.0.0.3,nw_dst=20.0.0.4,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=35556,tp_dst=80,tcp_flags=syn tcp_csum:400c <=== failed in ovs-vswitchd.log Expected results: nc should pass Additional info: the issue doesn't exist on ovn2.13.0-39: + ip netns exec sw0p2 ping -c3 20.0.0.3 PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data. 64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1.44 ms 64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=0.091 ms 64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.068 ms --- 20.0.0.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 64ms rtt min/avg/max/mdev = 0.068/0.532/1.438/0.640 ms + ip netns exec sw0p1 nc -vz 20.0.0.4 80 + ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv Ncat: Ncat: Version 7.70 ( https://nmap.org/ncat ) Version 7.70 ( https://nmap.org/ncat ) Ncat: Listening on 20.0.0.4:80 Ncat: Connected to 20.0.0.4:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:38954. NCAT DEBUG: EOF on stdin NCAT DEBUG: Closing fd 4. + ip netns exec sw0p1 nc -vz 30.0.0.10 80 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 30.0.0.10:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:35534. NCAT DEBUG: Closing fd 4. + ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related + ip netns exec sw0p1 nc -vz 30.0.0.10 80 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 30.0.0.10:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:35536. NCAT DEBUG: Closing fd 4. <=== passed here + ovn-nbctl clear logical_switch sw1 acls + ip netns exec sw0p1 nc -vz 30.0.0.10 80 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 30.0.0.10:80. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Ncat: Connection from 20.0.0.3. Ncat: Connection from 20.0.0.3:35538. NCAT DEBUG: Closing fd 4. [root@hp-dl380pg8-12 test]# rpm -qa | grep -E "openvswitch|ovn" ovn2.13-host-2.13.0-39.el8fdp.x86_64 openvswitch2.13-2.13.0-41.el8fdb.x86_64 openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch ovn2.13-central-2.13.0-39.el8fdp.x86_64 ovn2.13-2.13.0-39.el8fdp.x86_64