Bug 1858191 - the connection to load balancer VIP breaks when ACL is added
Summary: the connection to load balancer VIP breaks when ACL is added
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: ovn2.13
Version: FDP 20.E
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Numan Siddique
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On:
Blocks: 1857865
TreeView+ depends on / blocked
 
Reported: 2020-07-17 07:42 UTC by Jianlin Shi
Modified: 2020-08-18 11:24 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-18 11:23:51 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github ovn-org/ovn/commit/925632ee6032225ea399e3671353c04297204eae None None None 2020-09-29 21:30:36 UTC
Red Hat Product Errata RHBA-2020:3488 None None None 2020-08-18 11:24:15 UTC

Description Jianlin Shi 2020-07-17 07:42:35 UTC
Description of problem:
the connection to load balancer VIP breaks when ACL is added

Version-Release number of selected component (if applicable):
ovn2.13-20.06.1-2.el8fdp.x86_64

How reproducible:
Always

Steps to Reproduce:
systemctl start openvswitch                                                                           
systemctl start ovn-northd                                                                            
ovn-nbctl set-connection ptcp:6641                                                                    
ovn-sbctl set-connection ptcp:6642                                                                    
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.111.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.111.25
systemctl restart ovn-controller 
                                         
    ovn-nbctl ls-add sw0                                                                              
    ovn-nbctl lsp-add sw0 sw0-port1                      
    ovn-nbctl lsp-set-addresses sw0-port1 "10:54:00:00:00:03 10.0.0.3"                                
                                         
    ovn-nbctl lsp-add sw0 sw0-port2                                                                   
    ovn-nbctl lsp-set-addresses sw0-port2 "10:54:00:00:00:04 10.0.0.4"                                
                                                                                                      
    ovn-nbctl ls-add sw1                                                                              
    ovn-nbctl lsp-add sw1 sw1-port1 sw0-port1 10                                                      
    ovn-nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03 20.0.0.3"
                                                                                                      
    ovn-nbctl lsp-add sw1 sw1-port2 sw0-port2 20                                                      
    ovn-nbctl lsp-set-addresses sw1-port2 "40:54:00:00:00:04 20.0.0.4"                                
                                                                                                      
                                                                                                      
    ovn-nbctl lr-add lr0                                                                              
    ovn-nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24                                       
    ovn-nbctl lsp-add sw1 sw1-lr0                                  
    ovn-nbctl lsp-set-type sw1-lr0 router                        
    ovn-nbctl lsp-set-addresses sw1-lr0 router                                                        
    ovn-nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1
                                              
                                                                                                      
    ovn-nbctl ls-add sw2                                           
    ovn-nbctl lsp-add sw2 sw2-port1                                                                   
    ovn-nbctl lsp-set-addresses sw2-port1 "50:54:00:00:00:03 30.0.0.3"
                                                                      
    ovn-nbctl lrp-add lr0 lr0-sw2 00:00:00:00:ff:03 30.0.0.1/24                                                                                                                                            
    ovn-nbctl lsp-add sw2 sw2-lr0                                                                     
    ovn-nbctl lsp-set-type sw2-lr0 router
    ovn-nbctl lsp-set-addresses sw2-lr0 router                                                        
    ovn-nbctl lsp-set-options sw2-lr0 router-port=lr0-sw2                                             
                                 
                                         
    ovn-nbctl lb-add lb0 "30.0.0.10:80" "20.0.0.4:80"                                                 
                                                                   
    ovn-nbctl ls-lb-add sw1 lb0                          
    ovn-nbctl ls-lb-add sw2 lb0                                                                       
    ovn-nbctl lr-lb-add lr0 lb0
ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal
    ip netns add sw0p1                                                   
    ip link set sw0p1 netns sw0p1                                  
    ip netns exec sw0p1 ip link set lo up   
    ip netns exec sw0p1 ip link set sw0p1 up             
    ip netns exec sw0p1 ip link set sw0p1 address 10:54:00:00:00:03   
    ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1          
    ip netns exec sw0p1 ip route add default via 10.0.0.1 dev sw0p1
    ovs-vsctl set Interface sw0p1 external_ids:iface-id=sw0-port1
                                         

    # Create the interface for lport sw1-port1
    ip netns exec sw0p1 ip link add link sw0p1 name sw1p1 type vlan id 10
    ip netns exec sw0p1 ip link set sw1p1 address 40:54:00:00:00:03 
    ip netns exec sw0p1 ip link set sw1p1 up      
    ip netns exec sw0p1 ip addr add 20.0.0.3/24 dev sw1p1
    ip netns exec sw0p1 ip route delete default via 10.0.0.1 dev sw0p1
    ip netns exec sw0p1 ip route add default via 20.0.0.1 dev sw1p1                                                                                                                                        
                                          

    ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal
    ip netns add sw0p2                     
    ip link set sw0p2 netns sw0p2
    ip netns exec sw0p2 ip link set lo up
    ip netns exec sw0p2 ip link set sw0p2 up
    ip netns exec sw0p2 ip link set sw0p2 address 10:54:00:00:00:04
    ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2
    ip netns exec sw0p2 ip route add default via 10.0.0.1 dev sw0p2      
    ovs-vsctl set Interface sw0p2 external_ids:iface-id=sw0-port2

    # Create the interface for lport sw1-port2
    ip netns exec sw0p2 ip link add link sw0p2 name sw1p2 type vlan id 20
    ip netns exec sw0p2 ip link set sw1p2 address 40:54:00:00:00:04 
    ip netns exec sw0p2 ip link set sw1p2 up
    ip netns exec sw0p2 ip addr add 20.0.0.4/24 dev sw1p2
    ip netns exec sw0p2 ip route delete default via 10.0.0.1 dev sw0p2
    ip netns exec sw0p2 ip route add default via 20.0.0.1 dev sw1p2

ip netns exec sw0p2 ping -c3 20.0.0.3


    # Start nc server on sw1p2 (sw0p2 is the parent)
    ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv &

    # connect to sw1p2 (which is the backend of LB VIP 30.0.0.10) from sw1p1 using  nc client. It works.
    ip netns exec sw0p1 nc -vz 20.0.0.4 80
   
    # connect to LB VIP from sw1p1 using  nc client. It works.
    ip netns exec sw0p1 nc -vz 30.0.0.10 80
    
    # Add the below ACL on sw1
    ovn-nbctl acl-add sw1 to-lport 2002 "ip" allow-related
    
    # connect to LB VIP from sw1p1 using  nc client. It doesn't work now.
    ip netns exec sw0p1 nc -vz 30.0.0.10 80
    
    # Clear the ACL and it works again
    ovn-nbctl clear logical_switch sw1 acls
    
    ip netns exec sw0p1 nc -vz 30.0.0.10 80

Actual results:
+ ip netns exec sw0p2 ping -c3 20.0.0.3                            
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.                   
64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1.44 ms                
64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=0.085 ms                 
64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.069 ms                  
                                                                      
--- 20.0.0.3 ping statistics ---            
3 packets transmitted, 3 received, 0% packet loss, time 41ms
rtt min/avg/max/mdev = 0.069/0.530/1.438/0.642 ms                     
+ ip netns exec sw0p1 nc -vz 20.0.0.4 80                           
+ ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv                     
Ncat: Version 7.70 ( https://nmap.org/ncat )                     
Ncat: Version 7.70 ( https://nmap.org/ncat )         
Ncat: Listening on 20.0.0.4:80                           
Ncat: Connected to 20.0.0.4:80.               
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.                    
Ncat: Connection from 20.0.0.3.                                     
Ncat: Connection from 20.0.0.3:38964.             
NCAT DEBUG: EOF on stdin                                              
NCAT DEBUG: Closing fd 4.                                             
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                                                                                                                                  
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.         
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.                   
Ncat: Connection from 20.0.0.3.                          
Ncat: Connection from 20.0.0.3:35544.                                 
NCAT DEBUG: Closing fd 4.                 
+ ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                             
Ncat: Version 7.70 ( https://nmap.org/ncat )             
Ncat: Connection timed out.   

<=== times out here
                                           
+ ovn-nbctl clear logical_switch sw1 acls                        
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                             
Ncat: Version 7.70 ( https://nmap.org/ncat )                            
Ncat: Connected to 30.0.0.10:80.                                         
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.                  
Ncat: Connection from 20.0.0.3.             
Ncat: Connection from 20.0.0.3:35548.                    
NCAT DEBUG: Closing fd 4.                                             
[root@hp-dl380pg8-12 test]# rpm -qa | grep -E "openvswitch|ovn"    
openvswitch2.13-2.13.0-41.el8fdb.x86_64
ovn2.13-central-20.06.1-2.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
ovn2.13-host-20.06.1-2.el8fdp.x86_64
ovn2.13-20.06.1-2.el8fdp.x86_64

[root@hp-dl380pg8-12 test]# grep failed /var/log/openvswitch/ovs-vswitchd.log                         
2020-07-17T07:41:35.462Z|00001|dpif(handler1)|WARN|system@ovs-system: execute ct(commit,label=0/0x1),push_vlan(vid=20,pcp=0),3 failed (Invalid argument) on packet tcp,vlan_tci=0x0000,dl_src=00:00:00:00:ff:02,dl_dst=40:54:00:00:00:04,nw_src=20.0.0.3,nw_dst=20.0.0.4,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=35556,tp_dst=80,tcp_flags=syn tcp_csum:400c

<=== failed in ovs-vswitchd.log

Expected results:
nc should pass

Additional info:


the issue doesn't exist on ovn2.13.0-39:

+ ip netns exec sw0p2 ping -c3 20.0.0.3                             
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.                                
64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1.44 ms                        
64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=0.091 ms                       
64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.068 ms
                                                                      
--- 20.0.0.3 ping statistics ---                                      
3 packets transmitted, 3 received, 0% packet loss, time 64ms                                                                                                                                               
rtt min/avg/max/mdev = 0.068/0.532/1.438/0.640 ms            
+ ip netns exec sw0p1 nc -vz 20.0.0.4 80                         
+ ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv                          
Ncat: Ncat: Version 7.70 ( https://nmap.org/ncat )                     
Version 7.70 ( https://nmap.org/ncat )                                
Ncat: Listening on 20.0.0.4:80                                       
Ncat: Connected to 20.0.0.4:80.                        
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.                 
Ncat: Connection from 20.0.0.3.                                  
Ncat: Connection from 20.0.0.3:38954.                                                                                                                                  
NCAT DEBUG: EOF on stdin                                         
NCAT DEBUG: Closing fd 4.                                             
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                               
Ncat: Version 7.70 ( https://nmap.org/ncat )                             
Ncat: Connected to 30.0.0.10:80.                                       
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.               
Ncat: Connection from 20.0.0.3.                                               
Ncat: Connection from 20.0.0.3:35534.                                         
NCAT DEBUG: Closing fd 4.                                               
+ ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related                        
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                     
Ncat: Version 7.70 ( https://nmap.org/ncat )                                  
Ncat: Connected to 30.0.0.10:80.                                    
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.                         
Ncat: Connection from 20.0.0.3.                                               
Ncat: Connection from 20.0.0.3:35536.                                         
NCAT DEBUG: Closing fd 4.

<=== passed here

+ ovn-nbctl clear logical_switch sw1 acls
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 30.0.0.10:80.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
Ncat: Connection from 20.0.0.3.
Ncat: Connection from 20.0.0.3:35538.
NCAT DEBUG: Closing fd 4.

[root@hp-dl380pg8-12 test]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.13-host-2.13.0-39.el8fdp.x86_64
openvswitch2.13-2.13.0-41.el8fdb.x86_64
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
ovn2.13-central-2.13.0-39.el8fdp.x86_64
ovn2.13-2.13.0-39.el8fdp.x86_64

Comment 3 Jianlin Shi 2020-07-20 01:32:20 UTC
Verified on ovn2.13-20.06.1-4.el8fdp.x86_64:

+ ip netns exec sw0p2 ping -c3 20.0.0.3                                                               
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.                                                        
64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=1.35 ms
64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1061 ms
64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.076 ms
                                                                                                      
--- 20.0.0.3 ping statistics ---                                                                      
3 packets transmitted, 3 received, 0% packet loss, time 63ms
rtt min/avg/max/mdev = 0.076/354.211/1061.205/499.920 ms, pipe 2
+ ip netns exec sw0p1 nc -vz 20.0.0.4 80                                                              
+ ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv                                                        
Ncat: Version 7.70 ( https://nmap.org/ncat )                                                          
Ncat: Version 7.70 ( https://nmap.org/ncat )                                                          
Ncat: Listening on 20.0.0.4:80                                                                        
Ncat: Connected to 20.0.0.4:80.                                                                       
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:45528.                                                                 
NCAT DEBUG: EOF on stdin                                                                              
NCAT DEBUG: Closing fd 4.                                                                             
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                             
Ncat: Version 7.70 ( https://nmap.org/ncat )                                                          
Ncat: Connected to 30.0.0.10:80.                                                                      
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:50484.                                                                 
NCAT DEBUG: Closing fd 4.                                                                             
+ ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                             
Ncat: Version 7.70 ( https://nmap.org/ncat )                                                          
Ncat: Connected to 30.0.0.10:80.                                                                      
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:50486.                                                                 
NCAT DEBUG: Closing fd 4.                                                                             
+ ovn-nbctl clear logical_switch sw1 acls                                                             
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                             
Ncat: Version 7.70 ( https://nmap.org/ncat )                                                          
Ncat: Connected to 30.0.0.10:80.                                                                      
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:50488.                                                                 
NCAT DEBUG: Closing fd 4.                                                                             
[root@dell-per740-12 bz1858191]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
openvswitch2.13-2.13.0-48.el8fdp.x86_64                                                               
ovn2.13-host-20.06.1-4.el8fdp.x86_64                                                                  
ovn2.13-20.06.1-4.el8fdp.x86_64                                                                       
ovn2.13-central-20.06.1-4.el8fdp.x86_64

Comment 4 Jianlin Shi 2020-07-20 01:35:34 UTC
also Verified on ovn2.13-20.06.1-4.el7fdp.x86_64:

+ ip netns exec sw0p2 ping -c3 20.0.0.3                                                               
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.                                                        
64 bytes from 20.0.0.3: icmp_seq=2 ttl=64 time=3.22 ms                                                
64 bytes from 20.0.0.3: icmp_seq=1 ttl=64 time=1003 ms                                                
64 bytes from 20.0.0.3: icmp_seq=3 ttl=64 time=0.069 ms                                               
                                                                                                      
--- 20.0.0.3 ping statistics ---                                                                      
3 packets transmitted, 3 received, 0% packet loss, time 2001ms                                        
rtt min/avg/max/mdev = 0.069/335.482/1003.153/472.116 ms, pipe 2                                      
+ ip netns exec sw0p1 nc -vz 20.0.0.4 80                                                              
+ ip netns exec sw0p2 nc -l 20.0.0.4 80 -k -vv                                                        
Ncat: Version 7.50 ( https://nmap.org/ncat )                                                          
Ncat: Version 7.50 ( https://nmap.org/ncat )                                                          
Ncat: Listening on 20.0.0.4:80                                                                        
Ncat: Connected to 20.0.0.4:80.                                                                       
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.                                                 
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:58956.                                                                 
NCAT DEBUG: EOF on stdin                                                                              
NCAT DEBUG: Closing fd 4.                                                                             
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                             
Ncat: Version 7.50 ( https://nmap.org/ncat )                                                          
Ncat: Connected to 30.0.0.10:80.                                                                      
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.                                                 
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:51278.                                                                 
NCAT DEBUG: Closing fd 4.                                                                             
+ ovn-nbctl acl-add sw1 to-lport 2002 ip allow-related                                                
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                             
Ncat: Version 7.50 ( https://nmap.org/ncat )                                                          
Ncat: Connected to 30.0.0.10:80.                                                                      
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.                                                 
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:51280.                                                                 
NCAT DEBUG: Closing fd 4.                                                                             
+ ovn-nbctl clear logical_switch sw1 acls                                                             
+ ip netns exec sw0p1 nc -vz 30.0.0.10 80                                                             
Ncat: Version 7.50 ( https://nmap.org/ncat )                                                          
Ncat: Connected to 30.0.0.10:80.                                                                      
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.                                                 
Ncat: Connection from 20.0.0.3.                                                                       
Ncat: Connection from 20.0.0.3:51282.                                                                 
NCAT DEBUG: Closing fd 4.                                                                             
[root@dell-per740-42 bz1858191]# rpm -qa | grep -E "openvswitch|ovn"                                  
openvswitch2.13-2.13.0-39.el7fdp.x86_64                                                               
ovn2.13-20.06.1-4.el7fdp.x86_64                                                                       
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch                                                 
ovn2.13-central-20.06.1-4.el7fdp.x86_64                                                               
ovn2.13-host-20.06.1-4.el7fdp.x86_64

Comment 8 errata-xmlrpc 2020-08-18 11:23:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3488


Note You need to log in before you can comment on or make changes to this bug.