Description of problem: Different mirrored packets format for ip6gre with ovs kernel and userspace With kernel there is an extra Destination Options for IPv6 before GRE header #################################################### userspace #################################################### [root@dell-per750-36 mirror]# ovs-vsctl show 7fc97c03-8c7b-4494-bcb6-a082dd0cb4cc Bridge br-int datapath_type: netdev Port gre123 Interface gre123 type: ip6gre options: {key="123", remote_ip="2001:db8:123::2"} Port vhost_p2 tag: 42 Interface vhost_p2 type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhost_p2"} Port vhost_p1 tag: 42 Interface vhost_p1 type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhost_p1"} Port br-int Interface br-int type: internal Bridge ovsbr0 datapath_type: netdev Port ovsbr0 Interface ovsbr0 type: internal Port dpdk0 Interface dpdk0 type: dpdk options: {dpdk-devargs="0000:b1:00.0"} ovs_version: "2.17.4" [root@dell-per750-36 mirror]# [root@dell-per750-52 mirror]# tshark -nV -i enp202s0np0 Running as user "root" and group "root". This could be dangerous. Capturing on 'enp202s0np0' Frame 1: 164 bytes on wire (1312 bits), 164 bytes captured (1312 bits) on interface enp202s0np0, id 0 Interface id: 0 (enp202s0np0) Interface name: enp202s0np0 Encapsulation type: Ethernet (1) Arrival Time: Nov 6, 2022 22:32:04.062500160 EST [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1667791924.062500160 seconds [Time delta from previous captured frame: 0.000000000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.000000000 seconds] Frame Number: 1 Frame Length: 164 bytes (1312 bits) Capture Length: 164 bytes (1312 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ipv6:gre:eth:ethertype:vlan:ethertype:ip:icmp:data] Ethernet II, Src: 3c:fd:fe:bb:1b:6c, Dst: 00:15:4d:12:2d:ac Destination: 00:15:4d:12:2d:ac Address: 00:15:4d:12:2d:ac .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: 3c:fd:fe:bb:1b:6c Address: 3c:fd:fe:bb:1b:6c .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv6 (0x86dd) Internet Protocol Version 6, Src: 2001:db8:123::1, Dst: 2001:db8:123::2 0110 .... = Version: 6 .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 (DSCP: CS0, ECN: Not-ECT) .... 0000 00.. .... .... .... .... .... = Differentiated Services Codepoint: Default (0) .... .... ..00 .... .... .... .... .... = Explicit Congestion Notification: Not ECN-Capable Transport (0) .... .... .... 0000 0000 0000 0000 0000 = Flow Label: 0x00000 Payload Length: 110 Next Header: Generic Routing Encapsulation (47) Hop Limit: 64 Source Address: 2001:db8:123::1 Destination Address: 2001:db8:123::2 Generic Routing Encapsulation (Transparent Ethernet bridging) Flags and Version: 0x2000 0... .... .... .... = Checksum Bit: No .0.. .... .... .... = Routing Bit: No ..1. .... .... .... = Key Bit: Yes ...0 .... .... .... = Sequence Number Bit: No .... 0... .... .... = Strict Source Route Bit: No .... .000 .... .... = Recursion control: 0 .... .... 0000 0... = Flags (Reserved): 0 .... .... .... .000 = Version: GRE (0) Protocol Type: Transparent Ethernet bridging (0x6558) Key: 0x0000007b Ethernet II, Src: 00:de:ad:01:00:01, Dst: 00:de:ad:01:00:02 Destination: 00:de:ad:01:00:02 Address: 00:de:ad:01:00:02 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: 00:de:ad:01:00:01 Address: 00:de:ad:01:00:01 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: 802.1Q Virtual LAN (0x8100) 802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 42 000. .... .... .... = Priority: Best Effort (default) (0) ...0 .... .... .... = DEI: Ineligible .... 0000 0010 1010 = ID: 42 Type: IPv4 (0x0800) Internet Protocol Version 4, Src: 192.168.1.1, Dst: 192.168.1.2 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 84 Identification: 0x1eef (7919) Flags: 0x40, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment Offset: 0 Time to Live: 64 Protocol: ICMP (1) Header Checksum: 0x9866 [validation disabled] [Header checksum status: Unverified] Source Address: 192.168.1.1 Destination Address: 192.168.1.2 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0x83af [correct] [Checksum Status: Good] Identifier (BE): 1 (0x0001) Identifier (LE): 256 (0x0100) Sequence Number (BE): 110 (0x006e) Sequence Number (LE): 28160 (0x6e00) Timestamp from icmp data: Nov 6, 2022 22:32:03.000000000 EST [Timestamp from icmp data (relative): 1.062500160 seconds] Data (48 bytes) 0000 14 2f 05 00 00 00 00 00 10 11 12 13 14 15 16 17 ./.............. 0010 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 ........ !"#$%&' 0020 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 ()*+,-./01234567 Data: 142f050000000000101112131415161718191a1b1c1d1e1f202122232425262728292a2b… [Length: 48] #################################################### kernel #################################################### [root@dell-per750-36 mirror]# ovs-vsctl show 7fc97c03-8c7b-4494-bcb6-a082dd0cb4cc Bridge ovsbr0 Port ens5f0 Interface ens5f0 Port ovsbr0 Interface ovsbr0 type: internal Bridge br-int Port br-int Interface br-int type: internal Port gre123 Interface gre123 type: ip6gre options: {key="123", remote_ip="2001:db8:123::2"} Port p2 tag: 42 Interface p2 Port p1 tag: 42 Interface p1 ovs_version: "2.17.4" [root@dell-per750-36 mirror]# [root@dell-per750-52 mirror]# tshark -nV -i enp202s0np0 Running as user "root" and group "root". This could be dangerous. Capturing on 'enp202s0np0' Frame 1: 172 bytes on wire (1376 bits), 172 bytes captured (1376 bits) on interface enp202s0np0, id 0 Interface id: 0 (enp202s0np0) Interface name: enp202s0np0 Encapsulation type: Ethernet (1) Arrival Time: Nov 6, 2022 22:43:24.746550991 EST [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1667792604.746550991 seconds [Time delta from previous captured frame: 0.000000000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.000000000 seconds] Frame Number: 1 Frame Length: 172 bytes (1376 bits) Capture Length: 172 bytes (1376 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ipv6:ipv6.dstopts:gre:eth:ethertype:vlan:ethertype:ip:icmp:data] Ethernet II, Src: 3c:fd:fe:bb:1b:6c, Dst: 00:15:4d:12:2d:ac Destination: 00:15:4d:12:2d:ac Address: 00:15:4d:12:2d:ac .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: 3c:fd:fe:bb:1b:6c Address: 3c:fd:fe:bb:1b:6c .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv6 (0x86dd) Internet Protocol Version 6, Src: 2001:db8:123::1, Dst: 2001:db8:123::2 0110 .... = Version: 6 .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 (DSCP: CS0, ECN: Not-ECT) .... 0000 00.. .... .... .... .... .... = Differentiated Services Codepoint: Default (0) .... .... ..00 .... .... .... .... .... = Explicit Congestion Notification: Not ECN-Capable Transport (0) .... .... .... 1001 1111 0100 1110 0001 = Flow Label: 0x9f4e1 Payload Length: 118 Next Header: Destination Options for IPv6 (60) Hop Limit: 64 Source Address: 2001:db8:123::1 Destination Address: 2001:db8:123::2 Destination Options for IPv6 Next Header: Generic Routing Encapsulation (47) Length: 0 [Length: 8 bytes] Tunnel Encapsulation Limit Type: Tunnel Encapsulation Limit (0x04) 00.. .... = Action: Skip and continue (0) ..0. .... = May Change: No ...0 0100 = Low-Order Bits: 0x04 Length: 1 Tunnel Encapsulation Limit: 0 PadN Type: PadN (0x01) 00.. .... = Action: Skip and continue (0) ..0. .... = May Change: No ...0 0001 = Low-Order Bits: 0x01 Length: 1 PadN: 00 Generic Routing Encapsulation (Transparent Ethernet bridging) Flags and Version: 0x2000 0... .... .... .... = Checksum Bit: No .0.. .... .... .... = Routing Bit: No ..1. .... .... .... = Key Bit: Yes ...0 .... .... .... = Sequence Number Bit: No .... 0... .... .... = Strict Source Route Bit: No .... .000 .... .... = Recursion control: 0 .... .... 0000 0... = Flags (Reserved): 0 .... .... .... .000 = Version: GRE (0) Protocol Type: Transparent Ethernet bridging (0x6558) Key: 0x0000007b Ethernet II, Src: 00:de:ad:01:00:01, Dst: 00:de:ad:01:00:02 Destination: 00:de:ad:01:00:02 Address: 00:de:ad:01:00:02 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: 00:de:ad:01:00:01 Address: 00:de:ad:01:00:01 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: 802.1Q Virtual LAN (0x8100) 802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 42 000. .... .... .... = Priority: Best Effort (default) (0) ...0 .... .... .... = DEI: Ineligible .... 0000 0010 1010 = ID: 42 Type: IPv4 (0x0800) Internet Protocol Version 4, Src: 192.168.1.1, Dst: 192.168.1.2 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 84 Identification: 0x97e2 (38882) Flags: 0x40, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment Offset: 0 Time to Live: 64 Protocol: ICMP (1) Header Checksum: 0x1f73 [validation disabled] [Header checksum status: Unverified] Source Address: 192.168.1.1 Destination Address: 192.168.1.2 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0xe561 [correct] [Checksum Status: Good] Identifier (BE): 1 (0x0001) Identifier (LE): 256 (0x0100) Sequence Number (BE): 125 (0x007d) Sequence Number (LE): 32000 (0x7d00) Timestamp from icmp data: Nov 6, 2022 22:43:23.000000000 EST [Timestamp from icmp data (relative): 1.746550991 seconds] Data (48 bytes) 0000 01 6b 0e 00 00 00 00 00 10 11 12 13 14 15 16 17 .k.............. 0010 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 ........ !"#$%&' 0020 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 ()*+,-./01234567 Data: 016b0e0000000000101112131415161718191a1b1c1d1e1f202122232425262728292a2b… [Length: 48] Version-Release number of selected component (if applicable): [root@dell-per750-36 mirror]# uname -r 5.14.0-185.el9.x86_64 [root@dell-per750-36 mirror]# rpm -qa | grep openvswitch kernel-kernel-networking-openvswitch-mirror-1.0-12.noarch openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch openvswitch2.17-2.17.0-52.el9fdp.x86_64 [root@dell-per750-36 mirror]# How reproducible: always Steps to Reproduce: 1. ovs-vsctl add-port br-int gre123 \ -- set interface gre123 type=ip6gre \ options:key=123 options:remote_ip=2001:db8:123::2 \ -- --id=@p0 get port gre123 \ -- --id=@m0 create mirror name=mirror1 select-all=true output-port=@p0 \ -- set bridge br-int mirrors=@m0 2. tshark -nV -i enp202s0np0 3. Actual results: Different mirrored packets format for ip6gre with ovs kernel and userspace Expected results: Same mirrored packets format for ip6gre with ovs kernel and userspace Additional info:
I have reproduced this with OVS 3.1 on RHEL 8.7. The header added in the kernel is the "Destination Options" that sets a hard limit to 4 further encapsulations, which is a default added by the kernel. This is primarily to prevent recursive encaps in a looped network where decap is not done. With user space this header is not being added, so a limit is not being placed on further encaps. As it is an ipv6 option, it is not mandatory to have this header so things should work fine without it. i.e. I don't consider this a bug. Whether this header is useful (or indeed harmful) to be added/checked for in OVS userspace and is worthwhile of an RFE needs further discussion.
I have entered a low-prio RFE BZ2210254 for aligning the packet headers. As it is not a bug I will close this Bz. Qijun, thanks for the excellent QE work to find this, as in isolation both packets are correct.