Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1809281

Summary: On an IPv6 bare metal deployment node network transmit errors are reported for genev_sys interfaces
Product: OpenShift Container Platform Reporter: Marius Cornea <mcornea>
Component: NetworkingAssignee: Ricardo Carrillo Cruz <ricarril>
Networking sub component: ovn-kubernetes QA Contact: zhaozhanqi <zzhao>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: urgent CC: bbennett, michael, ricarril, rkhan
Version: 4.3.zKeywords: Reopened
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-19 14:13:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2020-03-02 18:55:54 UTC 
Description of problem:

The following warnings show up in the Status Dashboard:

openshift-worker-0.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 472 transmit errors in the last two minutes.

openshift-master-0.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 763 transmit errors in the last two minutes.

openshift-master-1.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 763 transmit errors in the last two minutes.

openshift-master-2.qe1.kni.lab.eng.bos.redhat.com interface genev_sys_6081 has encountered 759 transmit errors in the last two minutes.


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2020-03-01-194304

How reproducible:
100%

Steps to Reproduce:
1. Deploy bare metal IPv6 cluster
2. Log in to Dashboard

Actual results:
Transmit errors are reported for nodes network interface.

Expected results:
No network transmit errors are reported.

Additional info:
Comment 1 Ricardo Carrillo Cruz 2020-03-05 08:52:00 UTC 
This doesn't seem a SDN problem, but a HW issue, either the NIC or the switch it is connected to.
Comment 2 Ricardo Carrillo Cruz 2020-03-05 08:53:06 UTC 
If you can consistently reproduce this, please report back to the bug with info ruling out HW issues and re-open it.
Comment 3 Marius Cornea 2020-03-05 14:46:55 UTC 
(In reply to Ricardo Carrillo Cruz from comment #2)
> If you can consistently reproduce this, please report back to the bug with
> info ruling out HW issues and re-open it.

This is reproducing not only on bare metal but also on VM simulated environments with virtio vNICs.

Please let me know what info do you need in order to rule out the root cause.
Comment 4 Ricardo Carrillo Cruz 2020-03-06 15:46:04 UTC 
Please provide:

oc get Network.operator.openshift.io cluster -oyaml

Underlying NICs MTU size.
Comment 5 Marius Cornea 2020-03-06 16:14:05 UTC 
oc get Network.operator.openshift.io cluster -oyaml

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2020-03-04T14:32:03Z"
  generation: 2
  name: cluster
  resourceVersion: "29243"
  selfLink: /apis/operator.openshift.io/v1/networks/cluster
  uid: bff1c637-b006-4b84-bff1-be3beed44a7f
spec:
  clusterNetwork:
  - cidr: fd01::/48
    hostPrefix: 64
  defaultNetwork:
    type: OVNKubernetes
  logLevel: ""
  serviceNetwork:
  - fd02::/112
status: {}

MTU is 1500

[root@openshift-master-0 core]# ip a s dev eno2
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 98:03:9b:61:7c:81 brd ff:ff:ff:ff:ff:ff
    inet6 2620:52:0:1386::34/128 scope global dynamic noprefixroute 
       valid_lft 2820sec preferred_lft 2820sec
    inet6 fe80::9a03:9bff:fe61:7c81/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
Comment 6 Michael Silich 2020-03-17 19:34:54 UTC 
Same issue with 4.4 (okd) installed on OpenStack (train).

oc get Network.operator.openshift.io cluster -oyaml

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2020-03-16T20:58:33Z"
  generation: 1
  name: cluster
  resourceVersion: "3223"
  selfLink: /apis/operator.openshift.io/v1/networks/cluster
  uid: 2221d79a-484a-4645-b8bd-b4087d432946
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  defaultNetwork:
    type: OVNKubernetes
  logLevel: ""
  serviceNetwork:
  - 172.30.0.0/16
status: {}



ip a 


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:05:a9:7e brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.88/16 brd 10.0.255.255 scope global dynamic noprefixroute eth0
       valid_lft 85497sec preferred_lft 85497sec
    inet6 fe80::6c77:11:aa56:fe64/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:7d:48:7e:68 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ca:9a:e2:f5:22:a1 brd ff:ff:ff:ff:ff:ff
5: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 26:fd:69:eb:5f:89 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::24fd:69ff:feeb:5f89/64 scope link
       valid_lft forever preferred_lft forever
6: k8s-dev2-dwq29-: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b6:21:03:3f:07:b6 brd ff:ff:ff:ff:ff:ff
    inet 10.128.0.2/23 scope global k8s-dev2-dwq29-
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1350 qdisc noop state DOWN group default qlen 1000
    link/ether 36:31:53:d5:c4:4a brd ff:ff:ff:ff:ff:ff
8: br-local: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 92:39:5f:7d:96:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9039:5fff:fe7d:9643/64 scope link
       valid_lft forever preferred_lft forever
9: br-nexthop: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:00:a9:fe:21:01 brd ff:ff:ff:ff:ff:ff
    inet 169.254.33.1/24 scope global br-nexthop
       valid_lft forever preferred_lft forever
10: eb299bfd0f76f4a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 56:f3:02:9f:f3:ff brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::54f3:2ff:fe9f:f3ff/64 scope link
       valid_lft forever preferred_lft forever
11: fd231ad9de0b92c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 46:86:80:cc:0e:47 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::4486:80ff:fecc:e47/64 scope link
       valid_lft forever preferred_lft forever
12: bea2c6212b59982@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 3a:83:9e:c6:5a:db brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::3883:9eff:fec6:5adb/64 scope link
       valid_lft forever preferred_lft forever
13: 80f545edf227c66@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether de:40:f1:19:dd:04 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::dc40:f1ff:fe19:dd04/64 scope link
       valid_lft forever preferred_lft forever
14: 6784ca707bb543a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
    link/ether 92:08:11:37:f8:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::9008:11ff:fe37:f8d7/64 scope link
       valid_lft forever preferred_lft forever
Comment 7 Ricardo Carrillo Cruz 2020-05-18 14:32:43 UTC 
Looks like dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1834918
Comment 8 Ricardo Carrillo Cruz 2020-05-19 14:13:16 UTC 

*** This bug has been marked as a duplicate of bug 1834918 ***