Bug 1897641 - Baremetal IPI with IPv6 control plane: nodes respond with duplicate packets to ICMP6 echo requests
Summary: Baremetal IPI with IPv6 control plane: nodes respond with duplicate packets t...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Tim Rozet
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks: 1897336 1899266
TreeView+ depends on / blocked
 
Reported: 2020-11-13 17:06 UTC by Marius Cornea
Modified: 2021-02-24 15:33 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: All ICMPv6 traffic ingressing an OVN-Kubernetes cluster was sent to both the host and OVN. Consequence: Pinging via ICMPv6 towards an OCP node would result in duplicate ICMPv6 responses. Fix: Only ICMPv6 Neighbor Advertisements and Route Advertisements are now sent to both OVN and the Host. All other ICMPv6 traffic is forwarded to the host only. Result: No duplicate responses from pinging over IPv6 to nodes in the cluster.
Clone Of:
: 1899266 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:33:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 357 0 None closed Bug 1887456: 11-20-2020 merge 2021-02-17 21:48:01 UTC
Github ovn-org ovn-kubernetes pull 1851 0 None closed Fixes ICMPv6 flooding behavior 2021-02-17 21:48:01 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:33:49 UTC

Description Marius Cornea 2020-11-13 17:06:50 UTC
Description of problem:

Baremetal IPI with IPv6 control plane: nodes respond with duplicate packets to ICMP6 echo requests:

ping -c 3 openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com
PING openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com(openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20)) 56 data bytes
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=1 ttl=64 time=0.448 ms
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=1 ttl=254 time=0.669 ms (DUP!)
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=2 ttl=64 time=0.493 ms
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=2 ttl=254 time=0.520 ms (DUP!)
64 bytes from openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::20): icmp_seq=3 ttl=64 time=0.441 ms

--- openshift-master-0.ocp-edge1.lab.eng.tlv2.redhat.com ping statistics ---
3 packets transmitted, 3 received, +2 duplicates, 0% packet loss, time 14ms
rtt min/avg/max/mdev = 0.441/0.514/0.669/0.083 ms

Pinging the api address results in a loop:

[kni@ocp-edge12 ~]$ ping -c3  api.ocp-edge1.lab.eng.tlv2.redhat.com
PING api.ocp-edge1.lab.eng.tlv2.redhat.com(api.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::10)) 56 data bytes
64 bytes from api.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::10): icmp_seq=1 ttl=64 time=0.894 ms
From registry.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::1): icmp_seq=1 Time exceeded: Hop limit
From registry.ocp-edge1.lab.eng.tlv2.redhat.com (2620:52:0:2e39::1): icmp_seq=2 Time exceeded: Hop limit

--- api.ocp-edge1.lab.eng.tlv2.redhat.com ping statistics ---
2 packets transmitted, 1 received, +2 errors, 50% packet loss, time 2ms
rtt min/avg/max/mdev = 0.894/0.894/0.894/0.000 ms


If we run tcpdump on the interface bridged to br-ex we can see many duplicate packets:

sudo ./tcpdump -i eno6 icmp6 -nnnn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno6, link-type EN10MB (Ethernet), capture size 262144 bytes
16:53:29.417180 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.417880 IP6 2620:52:0:2e39::10 > 2620:52:0:2e39::1: ICMP6, echo reply, seq 1, length 64
16:53:29.417897 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.417955 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418232 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418253 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418453 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418484 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418689 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418729 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418949 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.418998 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419189 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419211 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419373 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419397 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419540 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419571 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419781 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419815 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419964 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.419990 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64
16:53:29.420180 IP6 2620:52:0:2e39::1 > 2620:52:0:2e39::10: ICMP6, echo request, seq 1, length 64


Version-Release number of selected component (if applicable):
4.6.3

How reproducible:
100%

Steps to Reproduce:
1. Deploy baremetal IPI with IPv6 control plane environment
2. Ping nodes hostnames or api hostname

Actual results:
Replies include duplicate packets which result in packet loss.

Expected results:
No duplicate packets.

Additional info:

When removing the interface from br-ex bridge there are no duplicate packets. This issue can be reproduced on both bare metal and VM environments.

Comment 3 Tim Rozet 2020-11-18 19:09:50 UTC
This only affects icmpv6 traffic. Lowering the severity.

Comment 5 Anurag saxena 2020-12-01 19:18:19 UTC
@mcornea, assigning back to you since you were the reporter. Thanks

Comment 6 Marius Cornea 2020-12-07 18:49:53 UTC
The issue is not reproducing anymore on 4.7.0-0.nightly-2020-12-04-013308.

Comment 9 errata-xmlrpc 2021-02-24 15:33:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.