Bug 1897641

Summary: Baremetal IPI with IPv6 control plane: nodes respond with duplicate packets to ICMP6 echo requests
Product: OpenShift Container Platform Reporter: Marius Cornea <mcornea>
Component: NetworkingAssignee: Tim Rozet <trozet>
Networking sub component: ovn-kubernetes QA Contact: Marius Cornea <mcornea>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aconstan, aojeagar
Version: 4.6   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: All ICMPv6 traffic ingressing an OVN-Kubernetes cluster was sent to both the host and OVN. Consequence: Pinging via ICMPv6 towards an OCP node would result in duplicate ICMPv6 responses. Fix: Only ICMPv6 Neighbor Advertisements and Route Advertisements are now sent to both OVN and the Host. All other ICMPv6 traffic is forwarded to the host only. Result: No duplicate responses from pinging over IPv6 to nodes in the cluster.
Story Points: ---
Clone Of:
: 1899266 (view as bug list) Environment:
Last Closed: 2021-02-24 15:33:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1897336, 1899266    

Description Marius Cornea 2020-11-13 17:06:50 UTC
Description of problem:

Baremetal IPI with IPv6 control plane: nodes respond with duplicate packets to ICMP6 echo requests:

ping -c 3 openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com
PING openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com(openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20)) 56 data bytes
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=1 ttl=64 time=0.448 ms
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=1 ttl=254 time=0.669 ms (DUP!)
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=2 ttl=64 time=0.493 ms
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=2 ttl=254 time=0.520 ms (DUP!)
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=3 ttl=64 time=0.441 ms

--- openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com ping statistics ---
3 packets transmitted, 3 received, +2 duplicates, 0% packet loss, time 14ms
rtt min/avg/max/mdev = 0.441/0.514/0.669/0.083 ms

Pinging the api address results in a loop:

[kni@ocp-edge12 ~]$ ping -c3  api.ocp-edge1.lab.eng.tlv2.redhat.com
PING api.ocp-edge1.lab.eng.tlv2.redhat.com(api.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::10)) 56 data bytes
64 bytes from api.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::10): icmp_seq=1 ttl=64 time=0.894 ms
From registry.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::1): icmp_seq=1 Time exceeded: Hop limit
From registry.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::1): icmp_seq=2 Time exceeded: Hop limit

--- api.ocp-edge1.lab.eng.tlv2.redhat.com ping statistics ---
2 packets transmitted, 1 received, +2 errors, 50% packet loss, time 2ms
rtt min/avg/max/mdev = 0.894/0.894/0.894/0.000 ms


If we run tcpdump on the interface bridged to br-ex we can see many duplicate packets:

sudo ./tcpdump -i eno6 icmp6 -nnnn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno6, link-type EN10MB (Ethernet), capture size 262144 bytes
16:53:29.417180 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.417880 IP6 2620:52:0:2e39::10 > 2620:52:0:2e39::1: ICMP6, echo reply, seq 1, length 64
16:53:29.417897 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.417955 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418232 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418253 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418453 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418484 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418689 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418729 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418949 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418998 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419189 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419211 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419373 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419397 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419540 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419571 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419781 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419815 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419964 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419990 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.420180 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64


Version-Release number of selected component (if applicable):
4.6.3

How reproducible:
100%

Steps to Reproduce:
1. Deploy baremetal IPI with IPv6 control plane environment
2. Ping nodes hostnames or api hostname

Actual results:
Replies include duplicate packets which result in packet loss.

Expected results:
No duplicate packets.

Additional info:

When removing the interface from br-ex bridge there are no duplicate packets. This issue can be reproduced on both bare metal and VM environments.

Comment 3 Tim Rozet 2020-11-18 19:09:50 UTC
This only affects icmpv6 traffic. Lowering the severity.

Comment 5 Anurag saxena 2020-12-01 19:18:19 UTC
@mcornea, assigning back to you since you were the reporter. Thanks

Comment 6 Marius Cornea 2020-12-07 18:49:53 UTC
The issue is not reproducing anymore on 4.7.0-0.nightly-2020-12-04-013308.

Comment 9 errata-xmlrpc 2021-02-24 15:33:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633