Bug 1830485

Summary: iproute2: fully support for geneve/vxlan/erspan options
Product: Red Hat Enterprise Linux 8 Reporter: Xin Long <lxin>
Component: iprouteAssignee: Andrea Claudi <aclaudi>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: high Docs Contact:
Priority: high    
Version: 8.2CC: atragler, jishi
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: iproute-5.3.0-4.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 01:43:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xin Long 2020-05-02 06:48:33 UTC
After the backports of bz1830482 and bz1730151, RHEL-8.3 would fully support geneve/vxlan/erspan options on kernel side, this bz is to backport the iproute2 side:

https://patchwork.ozlabs.org/project/netdev/cover/cover.1587983178.git.lucien.xin@gmail.com/

Comment 1 Andrea Claudi 2020-06-04 19:39:58 UTC
An update to the uapi is also needed to make this compile and work:
- commit 7438afd2cc8d3 ("Update kernel headers")

Comment 4 Jianlin Shi 2020-06-18 02:07:09 UTC
erspan_opts is verified in https://bugzilla.redhat.com/show_bug.cgi?id=1730151#c12

Comment 5 Jianlin Shi 2020-06-18 02:08:24 UTC
test for geneve option:

[root@ibm-x3650m4-01-vm-16 bz1830485]# bash -x geneve.sh                                              
+ ip netns add a
+ ip netns add b
+ ip -n a link add eth0 type veth peer name eth0 netns b                                              
+ ip -n a link set eth0 up
+ ip -n b link set eth0 up
+ ip -n a addr add 10.1.0.1/24 dev eth0
+ ip -n b addr add 10.1.0.2/24 dev eth0
+ ip -n b link add geneve1 type geneve id 1 remote 10.1.0.1 ttl 64                                    
+ ip -n b addr add 1.1.1.1/24 dev geneve1
+ ip -n b link set geneve1 up
+ ip -n b route add 2.1.1.0/24 dev geneve1                                                            
+ ip -n a link add geneve1 type geneve external                                                       
+ ip -n a addr add 2.1.1.1/24 dev geneve1
+ ip -n a link set geneve1 up
+ ip -n a route add 1.1.1.0/24 encap ip id 1 geneve_opts 1:1:1212121234567890,1:1:1212121234567890,1:1:1212121234567890 dst 10.1.0.2 dev geneve1
+ ip -n a route show
1.1.1.0/24  encap ip id 1 src 0.0.0.0 dst 10.1.0.2 ttl 0 tos 0
        geneve_opts 1:1:1212121234567890,1:1:1212121234567890,1:1:1212121234567890 dev geneve1 scope link
2.1.1.0/24 dev geneve1 proto kernel scope link src 2.1.1.1
10.1.0.0/24 dev eth0 proto kernel scope link src 10.1.0.1
+ ip netns exec a ping 1.1.1.1 -c 1                                                                   
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.                                                          
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.046 ms
                                                                                                      
--- 1.1.1.1 ping statistics ---                                                                       
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.046/0.046/0.046/0.000 ms

and capture the packet for geneve:

22:05:44.050228 Out 42:32:11:c8:17:84 ethertype IPv4 (0x0800), length 186: (tos 0x0, id 2060, offset 0, flags [none], proto UDP (17), length 170)
    10.1.0.1.48621 > 10.1.0.2.6081: [no cksum] Geneve, Flags [none], vni 0x1, proto TEB (0x6558), options [class Standard (0x1) type 0x1 len 12 data 12121212 34567890, class Standard (0x1) type 0x1 len 12 data 12121212 34567890, class Standard (0x1) type 0x1 len 12 data 12121212 34567890]      

<=== the geneve option added by ip
           
        56:36:2c:f7:04:b7 > a2:cd:e2:ff:fd:51, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 60352, offset 0, flags [DF], proto ICMP (1), length 84)
    2.1.1.1 > 1.1.1.1: ICMP echo request, id 31340, seq 1, length 64
        0x0000:  0004 0001 0006 4232 11c8 1784 0000 0800                                              
        0x0010:  4500 00aa 080c 0000 0011 9e33 0a01 0001
        0x0020:  0a01 0002 bded 17c1 0096 0000 0900 6558
        0x0030:  0000 0100 0001 0102 1212 1212 3456 7890                                              
        0x0040:  0001 0102 1212 1212 3456 7890 0001 0102                                              
        0x0050:  1212 1212 3456 7890 a2cd e2ff fd51 5636
        0x0060:  2cf7 04b7 0800 4500 0054 ebc0 4000 4001                                              
        0x0070:  49e5 0201 0101 0101 0101 0800 d5d0 7a6c                                              
        0x0080:  0001 f8cb ea5e 0000 0000 05c4 0000 0000
        0x0090:  0000 1011 1213 1415 1617 1819 1a1b 1c1d                                              
        0x00a0:  1e1f 2021 2223 2425 2627 2829 2a2b 2c2d
        0x00b0:  2e2f 3031 3233 3435 3637

Comment 6 Jianlin Shi 2020-06-18 02:13:11 UTC
test vxlan option:

[root@ibm-x3650m4-01-vm-16 bz1830485]# bash -x vxlan.sh
+ ip netns add a                                                                                      
+ ip netns add b                                                                                      
+ ip -n a link add eth0 type veth peer name eth0 netns b                                              
+ ip -n a link set eth0 up                                                                            
+ ip -n b link set eth0 up                                                                            
+ ip -n a addr add 10.1.0.1/24 dev eth0                                                               
+ ip -n b addr add 10.1.0.2/24 dev eth0                                                               
+ ip -n b link add vxlan1 type vxlan id 1 local 10.1.0.2 remote 10.1.0.1 dev eth0 ttl 64 gbp          
vxlan: destination port not specified                                                                 
Will use Linux kernel default (non-standard value)                                                    
Use 'dstport 4789' to get the IANA assigned value                                                     
Use 'dstport 0' to get default and quiet this message                                                 
+ ip -n b addr add 1.1.1.1/24 dev vxlan1                                                              
+ ip -n b link set vxlan1 up                                                                          
+ ip -n b route add 2.1.1.0/24 dev vxlan1                                                             
+ ip -n a link add vxlan1 type vxlan local 10.1.0.1 dev eth0 ttl 64 gbp external                      
vxlan: destination port not specified                                                                 
Will use Linux kernel default (non-standard value)                                                    
Use 'dstport 4789' to get the IANA assigned value                                                     
Use 'dstport 0' to get default and quiet this message                                                 
+ ip -n a addr add 2.1.1.1/24 dev vxlan1                                                              
+ ip -n a link set vxlan1 up                                                                          
+ ip -n a route add 1.1.1.0/24 encap ip id 1 vxlan_opts 1110 dst 10.1.0.2 dev vxlan1                  
+ ip -n a route show                                                                                  
1.1.1.0/24  encap ip id 1 src 0.0.0.0 dst 10.1.0.2 ttl 0 tos 0
        vxlan_opts 1110 dev vxlan1 scope link
2.1.1.0/24 dev vxlan1 proto kernel scope link src 2.1.1.1
10.1.0.0/24 dev eth0 proto kernel scope link src 10.1.0.1
+ ip netns exec a ping 1.1.1.1 -c 1                                                                   
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.                                                          
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.069 ms                                                
                                                                                                      
--- 1.1.1.1 ping statistics ---                                                                       
1 packets transmitted, 1 received, 0% packet loss, time 0ms                                           
rtt min/avg/max/mdev = 0.069/0.069/0.069/0.000 ms

capture the packet:

22:10:08.012907 Out 3e:e6:35:df:02:3e ethertype IPv4 (0x0800), length 150: (tos 0x0, ttl 64, id 20874, offset 0, flags [none], proto UDP (17), length 134)
    10.1.0.1.53712 > 10.1.0.2.8472: [no cksum] OTV, flags [I] (0x88), overlay 1110, instance 1  

<==== 1110 option added by ip
      
26:0f:ec:9a:3c:03 > 1e:80:03:df:b0:d9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 4558, offset 0, flags [DF], proto ICMP (1), length 84)
    2.1.1.1 > 1.1.1.1: ICMP echo request, id 31484, seq 1, length 64                                  
        0x0000:  0004 0001 0006 3ee6 35df 023e 0000 0800
        0x0010:  4500 0086 518a 0000 4011 14d9 0a01 0001                                              
        0x0020:  0a01 0002 d1d0 2118 0072 0000 8800 0456                                              
        0x0030:  0000 0100 1e80 03df b0d9 260f ec9a 3c03
        0x0040:  0800 4500 0054 11ce 4000 4001 23d8 0201                                              
        0x0050:  0101 0101 0101 0800 90d1 7afc 0001 00cd
        0x0060:  ea5e 0000 0000 4232 0000 0000 0000 1011                                              
        0x0070:  1213 1415 1617 1819 1a1b 1c1d 1e1f 2021
        0x0080:  2223 2425 2627 2829 2a2b 2c2d 2e2f 3031
        0x0090:  3233 3435 3637

Comment 7 Jianlin Shi 2020-06-18 02:18:11 UTC
vxlan option for tc:  [root@ibm-x3650m4-01-vm-16 bz1830485]# bash -x tc_vxlan.sh                                            
+ ip link add name vxlan1 type vxlan dstport 0 external                                               
+ tc qdisc add dev ens3 ingress
+ tc filter add dev ens3 protocol ip parent ffff: flower indev ens3 ip_proto udp action tunnel_key set src_ip 10.0.99.192 dst_ip 10.0.99.193 dst_port 6081 id 11 vxlan_opts 65793 action mirred egress redirect dev vxlan1                                                                                        
+ tc -s filter show dev ens3 parent ffff:                                                             
filter protocol ip pref 49152 flower chain 0
filter protocol ip pref 49152 flower chain 0 handle 0x1
  indev ens3                                                                                          
  eth_type ipv4                                                                                       
  ip_proto udp                                                                                        
  not_in_hw                                                                                           
        action order 1: tunnel_key  set                                                               
        src_ip 10.0.99.192                                                                            
        dst_ip 10.0.99.193                                                                            
        key_id 11                                                                                     
        dst_port 6081                                                                                 
        vxlan_opts 65793                                                                              
        csum pipe                                                                                     
         index 1 ref 1 bind 1                                                                         
        Action statistics:                                                                            
        Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0                                                                      
                                                                                                      
        action order 2: mirred (Egress Redirect to device vxlan1) stolen                              
        index 1 ref 1 bind 1                                                                          
        Action statistics:                                                                            
        Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0

Comment 8 Jianlin Shi 2020-06-18 02:20:28 UTC
erspan options for tc:

[root@ibm-x3650m4-01-vm-16 bz1830485]# bash -x tc_erspan.sh 
+ ip link add name erspan1 type erspan external
+ tc qdisc add dev ens3 ingress
+ tc filter add dev ens3 protocol ip parent ffff: flower indev ens3 ip_proto udp action tunnel_key set src_ip 10.0.99.192 dst_ip 10.0.99.193 dst_port 6081 id 11 erspan_opts 1:2:0:0 action mirred egress redirect dev erspan1
+ tc -s filter show dev ens3 parent ffff:
filter protocol ip pref 49152 flower chain 0 
filter protocol ip pref 49152 flower chain 0 handle 0x1 
  indev ens3
  eth_type ipv4
  ip_proto udp
  not_in_hw
        action order 1: tunnel_key  set
        src_ip 10.0.99.192
        dst_ip 10.0.99.193
        key_id 11
        dst_port 6081
        erspan_opts 1:2:0:0
        csum pipe
         index 1 ref 1 bind 1
        Action statistics:
        Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
        backlog 0b 0p requeues 0
        
        action order 2: mirred (Egress Redirect to device erspan1) stolen
        index 1 ref 1 bind 1
        Action statistics:
        Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
        backlog 0b 0p requeues 0

Comment 9 Jianlin Shi 2020-06-18 02:22:55 UTC
vxlan option in flower:

[root@ibm-x3650m4-01-vm-16 bz1830485]# bash -x flower_vxlan.sh 
+ ip link add name vxlan1 type vxlan dstport 0 external
+ tc qdisc add dev vxlan1 ingress
+ tc filter add dev vxlan1 protocol ip parent ffff: flower enc_src_ip 10.0.99.192 enc_dst_ip 10.0.99.193 enc_key_id 11 vxlan_opts 65793/4008635966 ip_proto udp action mirred egress redirect dev ens3
+ tc -s filter show dev vxlan1 parent ffff:
filter protocol ip pref 49152 flower chain 0 
filter protocol ip pref 49152 flower chain 0 handle 0x1 
  eth_type ipv4
  ip_proto udp
  enc_dst_ip 10.0.99.193
  enc_src_ip 10.0.99.192
  enc_key_id 11  vxlan_opts 65793/4008635966
  not_in_hw
        action order 1: mirred (Egress Redirect to device ens3) stolen
        index 1 ref 1 bind 1
        Action statistics:
        Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
        backlog 0b 0p requeues 0

Comment 10 Jianlin Shi 2020-06-18 02:24:06 UTC
erspan option for flower:

[root@ibm-x3650m4-01-vm-16 bz1830485]# bash -x flower_erspan.sh 
+ ip link add name erspan1 type erspan external
+ tc qdisc add dev erspan1 ingress
+ tc filter add dev erspan1 protocol ip parent ffff: flower enc_src_ip 10.0.99.192 enc_dst_ip 10.0.99.193 enc_key_id 11 erspan_opts 1:2:0:0/1:255:0:0 ip_proto udp action mirred egress redirect dev ens3
+ tc -s filter show dev erspan1 parent ffff:
filter protocol ip pref 49152 flower chain 0 
filter protocol ip pref 49152 flower chain 0 handle 0x1 
  eth_type ipv4
  ip_proto udp
  enc_dst_ip 10.0.99.193
  enc_src_ip 10.0.99.192
  enc_key_id 11  erspan_opts 1:2:0:0/1:255:0:0
  not_in_hw
        action order 1: mirred (Egress Redirect to device ens3) stolen
        index 1 ref 1 bind 1
        Action statistics:
        Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
        backlog 0b 0p requeues 0

Comment 11 Jianlin Shi 2020-06-18 02:24:18 UTC
[root@ibm-x3650m4-01-vm-16 bz1830485]# rpm -q iproute
iproute-5.3.0-4.el8.x86_64
[root@ibm-x3650m4-01-vm-16 bz1830485]# rpm -q iproute-tc
iproute-tc-5.3.0-4.el8.x86_64

Comment 14 errata-xmlrpc 2020-11-04 01:43:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (iproute bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4478