1809281 – On an IPv6 bare metal deployment node network transmit errors are reported for genev_sys interfaces

Bug 1809281 - On an IPv6 bare metal deployment node network transmit errors are reported for genev_sys interfaces

Summary: On an IPv6 bare metal deployment node network transmit errors are reported fo...

Keywords:
Status:	CLOSED DUPLICATE of bug 1834918
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.3.z
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	high
Target Milestone:	---
Target Release:	4.5.0
Assignee:	Ricardo Carrillo Cruz
QA Contact:	zhaozhanqi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-03-02 18:55 UTC by Marius Cornea
Modified:	2020-05-19 14:13 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-05-19 14:13:16 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Marius Cornea 2020-03-02 18:55:54 UTC

Description of problem:

The following warnings show up in the Status Dashboard:

openshift-worker-0.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 472 transmit errors in the last two minutes.

openshift-master-0.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 763 transmit errors in the last two minutes.

openshift-master-1.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 763 transmit errors in the last two minutes.

openshift-master-2.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 759 transmit errors in the last two minutes.


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2020-03-01-194304

How reproducible:
100%

Steps to Reproduce:
1. Deploy bare metal IPv6 cluster
2. Log in to Dashboard

Actual results:
Transmit errors are reported for nodes network interface.

Expected results:
No network transmit errors are reported.

Additional info:

Comment 1 Ricardo Carrillo Cruz 2020-03-05 08:52:00 UTC

This doesn't seem a SDN problem, but a HW issue, either the NIC or the switch it is connected to.

Comment 2 Ricardo Carrillo Cruz 2020-03-05 08:53:06 UTC

If you can consistently reproduce this, please report back to the bug with info ruling out HW issues and re-open it.

Comment 3 Marius Cornea 2020-03-05 14:46:55 UTC

(In reply to Ricardo Carrillo Cruz from comment #2)
> If you can consistently reproduce this, please report back to the bug with
> info ruling out HW issues and re-open it.

This is reproducing not only on bare metal but also on VM simulated environments with virtio vNICs.

Please let me know what info do you need in order to rule out the root cause.

Comment 4 Ricardo Carrillo Cruz 2020-03-06 15:46:04 UTC

Please provide:

oc get Network.operator.openshift.io cluster -oyaml

Underlying NICs MTU size.

Comment 5 Marius Cornea 2020-03-06 16:14:05 UTC

oc get Network.operator.openshift.io cluster -oyaml

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2020-03-04T14:32:03Z"
  generation: 2
  name: cluster
  resourceVersion: "29243"
  selfLink: /apis/operator.openshift.io/v1/networks/cluster
  uid: bff1c637-b006-4b84-bff1-be3beed44a7f
spec:
  clusterNetwork:
  - cidr: fd01::/48
    hostPrefix: 64
  defaultNetwork:
    type: OVNKubernetes
  logLevel: ""
  serviceNetwork:
  - fd02::/112
status: {}

MTU is 1500

[root@openshift-master-0 core]# ip a s dev eno2
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 98:03:9b:61:7c:81 brd ff:ff:ff:ff:ff:ff
    inet6 2620:52:0:1386::34/128 scope global dynamic noprefixroute 
       valid_lft 2820sec preferred_lft 2820sec
    inet6 fe80::9a03:9bff:fe61:7c81/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Comment 6 Michael Silich 2020-03-17 19:34:54 UTC

Same issue with 4.4 (okd) installed on OpenStack (train).

oc get Network.operator.openshift.io cluster -oyaml

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2020-03-16T20:58:33Z"
  generation: 1
  name: cluster
  resourceVersion: "3223"
  selfLink: /apis/operator.openshift.io/v1/networks/cluster
  uid: 2221d79a-484a-4645-b8bd-b4087d432946
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  defaultNetwork:
    type: OVNKubernetes
  logLevel: ""
  serviceNetwork:
  - 172.30.0.0/16
status: {}



ip a 


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:05:a9:7e brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.88/16 brd 10.0.255.255 scope global dynamic noprefixroute eth0
       valid_lft 85497sec preferred_lft 85497sec
    inet6 fe80::6c77:11:aa56:fe64/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:7d:48:7e:68 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ca:9a:e2:f5:22:a1 brd ff:ff:ff:ff:ff:ff
5: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 26:fd:69:eb:5f:89 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::24fd:69ff:feeb:5f89/64 scope link
       valid_lft forever preferred_lft forever
6: k8s-dev2-dwq29-: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b6:21:03:3f:07:b6 brd ff:ff:ff:ff:ff:ff
    inet 10.128.0.2/23 scope global k8s-dev2-dwq29-
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1350 qdisc noop state DOWN group default qlen 1000
    link/ether 36:31:53:d5:c4:4a brd ff:ff:ff:ff:ff:ff
8: br-local: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 92:39:5f:7d:96:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9039:5fff:fe7d:9643/64 scope link
       valid_lft forever preferred_lft forever
9: br-nexthop: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:00:a9:fe:21:01 brd ff:ff:ff:ff:ff:ff
    inet 169.254.33.1/24 scope global br-nexthop
       valid_lft forever preferred_lft forever
10: eb299bfd0f76f4a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 56:f3:02:9f:f3:ff brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::54f3:2ff:fe9f:f3ff/64 scope link
       valid_lft forever preferred_lft forever
11: fd231ad9de0b92c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 46:86:80:cc:0e:47 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::4486:80ff:fecc:e47/64 scope link
       valid_lft forever preferred_lft forever
12: bea2c6212b59982@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 3a:83:9e:c6:5a:db brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::3883:9eff:fec6:5adb/64 scope link
       valid_lft forever preferred_lft forever
13: 80f545edf227c66@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether de:40:f1:19:dd:04 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::dc40:f1ff:fe19:dd04/64 scope link
       valid_lft forever preferred_lft forever
14: 6784ca707bb543a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 92:08:11:37:f8:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::9008:11ff:fe37:f8d7/64 scope link
       valid_lft forever preferred_lft forever

Comment 7 Ricardo Carrillo Cruz 2020-05-18 14:32:43 UTC

Looks like dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1834918

Comment 8 Ricardo Carrillo Cruz 2020-05-19 14:13:16 UTC


*** This bug has been marked as a duplicate of bug 1834918 ***

Note You need to log in before you can comment on or make changes to this bug.