Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1898269

Summary: [OVN]: `make check` broken on 4.6
Product: OpenShift Container Platform Reporter: Alexander Constantinescu <aconstan>
Component: NetworkingAssignee: Alexander Constantinescu <aconstan>
Networking sub component: ovn-kubernetes QA Contact: Arti Sood <asood>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: anusaxen, asood
Version: 4.6Flags: asood: needinfo-
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1898268
: 1912388 (view as bug list) Environment:
Last Closed: 2020-12-14 13:50:52 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1898268    
Bug Blocks:    

Description Alexander Constantinescu 2020-11-16 17:43:24 UTC 
+++ This bug was initially created as a clone of Bug #1898268 +++

Description of problem:

When running `make check` on release-4.6 of openshift/ovn-kubernetes, the build fails. I suspect this is because https://github.com/openshift/ovn-kubernetes/commit/15661c9d1d23fe0b30d266bda3668ee848884fb5

Version-Release number of selected component (if applicable):

release-4.6 latest

How reproducible:


Steps to Reproduce:
1. Checkout openshift/ovn-kubernetes
2. cd go-controller && make check
3.

Actual results:

Build fails

Expected results:

Build should be successful 
Additional info:
Comment 2 Anurag saxena 2020-12-04 01:00:50 UTC 
@asood Can you help verifying it?
Comment 3 Arti Sood 2020-12-04 16:08:16 UTC 
1) 2 tests out of 32 tsts failed. Do we need all the tests passed to mark this bug verified? I am assuming I would need to enable IPv6 on the laptop to ensure the tests pass.

2) RHEL 8 is being used for verification. Is it acceptable for verification?

make check log excerpt below:-

Management Port Operations 
  sets up the management port for IPv6 clusters
  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:317
I1204 09:09:12.840337  316223 config.go:1373] Default config: {MTU:1400 ConntrackZone:64000 EncapType:geneve EncapIP: EncapPort:6081 InactivityProbe:100000 OpenFlowProbe:180 RawClusterSubnets:fda6::/48 ClusterSubnets:[{CIDR:fda6::/48 HostSubnetLength:64}]}
I1204 09:09:12.840396  316223 config.go:1374] Logging config: {File: CNIFile:/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log Level:5 LogFileMaxSize:100 LogFileMaxBackups:5 LogFileMaxAge:5}
I1204 09:09:12.840422  316223 config.go:1375] CNI config: {ConfDir:/etc/cni/net.d Plugin:ovn-k8s-cni-overlay}
I1204 09:09:12.840439  316223 config.go:1376] Kubernetes config: {Kubeconfig: CACert: APIServer:http://localhost:8443 Token: CompatServiceCIDR:fc95::/64 RawServiceCIDRs:fc95::/64 ServiceCIDRs:[fc95::/64] OVNConfigNamespace:ovn-kubernetes MetricsBindAddress: OVNMetricsBindAddress: MetricsEnablePprof:false OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:nil}
I1204 09:09:12.840481  316223 config.go:1377] Gateway config: {Mode: Interface: NextHop: VLANID:0 NodeportEnable:false DisableSNATMultipleGWs:false}
I1204 09:09:12.840502  316223 config.go:1378] OVN North config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:true externalID:ovn-nb exec:0xc00039a080}
I1204 09:09:12.840528  316223 config.go:1379] OVN South config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:false externalID:ovn-remote exec:0xc00039a080}
I1204 09:09:12.840551  316223 config.go:1380] Hybrid Overlay config: {Enabled:false RawClusterSubnets: ClusterSubnets:[] VXLANPort:4789}
I1204 09:09:12.840700  316223 ovs.go:166] exec(72): /fake-bin/ovs-vsctl --timeout=15 -- --if-exists del-port br-int k8s-node1 -- --may-exist add-port br-int ovn-k8s-mp0 -- set interface ovn-k8s-mp0 type=internal mtu_request=1400 external-ids:iface-id=k8s-node1
I1204 09:09:12.840727  316223 ovs.go:169] exec(72): stdout: ""
I1204 09:09:12.840742  316223 ovs.go:170] exec(72): stderr: ""
I1204 09:09:12.840770  316223 ovs.go:166] exec(73): /fake-bin/ovs-vsctl --timeout=15 --if-exists get interface ovn-k8s-mp0 mac_in_use
I1204 09:09:12.840786  316223 ovs.go:169] exec(73): stdout: "00:00:00:55:66:77"
I1204 09:09:12.840805  316223 ovs.go:170] exec(73): stderr: ""
I1204 09:09:12.840840  316223 ovs.go:166] exec(74): /fake-bin/ovs-vsctl --timeout=15 set interface ovn-k8s-mp0 mac=00\:00\:00\:55\:66\:77
I1204 09:09:12.840855  316223 ovs.go:169] exec(74): stdout: ""
I1204 09:09:12.840867  316223 ovs.go:170] exec(74): stderr: ""

• Failure [0.007 seconds]
Management Port Operations
/home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:237
  sets up the management port for IPv6 clusters [It]
  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:317

  Unexpected error:
      <*errors.errorString | 0xc00064acd0>: {
          s: "failed to add address fda6:0:0:1::2/64 on link ovn-k8s-mp0: operation not supported",
      }
      failed to add address fda6:0:0:1::2/64 on link ovn-k8s-mp0: operation not supported
  occurred

  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:142
------------------------------
Management Port Operations 
  sets up the management port for dual-stack clusters
  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:343
I1204 09:09:12.847505  316223 config.go:1373] Default config: {MTU:1400 ConntrackZone:64000 EncapType:geneve EncapIP: EncapPort:6081 InactivityProbe:100000 OpenFlowProbe:180 RawClusterSubnets:10.1.0.0/16,fda6::/48 ClusterSubnets:[{CIDR:10.1.0.0/16 HostSubnetLength:24} {CIDR:fda6::/48 HostSubnetLength:64}]}
I1204 09:09:12.847569  316223 config.go:1374] Logging config: {File: CNIFile:/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log Level:5 LogFileMaxSize:100 LogFileMaxBackups:5 LogFileMaxAge:5}
I1204 09:09:12.847594  316223 config.go:1375] CNI config: {ConfDir:/etc/cni/net.d Plugin:ovn-k8s-cni-overlay}
I1204 09:09:12.847617  316223 config.go:1376] Kubernetes config: {Kubeconfig: CACert: APIServer:http://localhost:8443 Token: CompatServiceCIDR:172.16.1.0/24,fc95::/64 RawServiceCIDRs:172.16.1.0/24,fc95::/64 ServiceCIDRs:[172.16.1.0/24 fc95::/64] OVNConfigNamespace:ovn-kubernetes MetricsBindAddress: OVNMetricsBindAddress: MetricsEnablePprof:false OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:nil}
I1204 09:09:12.847667  316223 config.go:1377] Gateway config: {Mode: Interface: NextHop: VLANID:0 NodeportEnable:false DisableSNATMultipleGWs:false}
I1204 09:09:12.847690  316223 config.go:1378] OVN North config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:true externalID:ovn-nb exec:0xc00039bb40}
I1204 09:09:12.847728  316223 config.go:1379] OVN South config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:false externalID:ovn-remote exec:0xc00039bb40}
I1204 09:09:12.847754  316223 config.go:1380] Hybrid Overlay config: {Enabled:false RawClusterSubnets: ClusterSubnets:[] VXLANPort:4789}
I1204 09:09:12.847910  316223 ovs.go:166] exec(75): /fake-bin/ovs-vsctl --timeout=15 -- --if-exists del-port br-int k8s-node1 -- --may-exist add-port br-int ovn-k8s-mp0 -- set interface ovn-k8s-mp0 type=internal mtu_request=1400 external-ids:iface-id=k8s-node1
I1204 09:09:12.847942  316223 ovs.go:169] exec(75): stdout: ""
I1204 09:09:12.847967  316223 ovs.go:170] exec(75): stderr: ""
I1204 09:09:12.848008  316223 ovs.go:166] exec(76): /fake-bin/ovs-vsctl --timeout=15 --if-exists get interface ovn-k8s-mp0 mac_in_use
I1204 09:09:12.848081  316223 ovs.go:169] exec(76): stdout: "00:00:00:55:66:77"
I1204 09:09:12.848104  316223 ovs.go:170] exec(76): stderr: ""
I1204 09:09:12.848157  316223 ovs.go:166] exec(77): /fake-bin/ovs-vsctl --timeout=15 set interface ovn-k8s-mp0 mac=00\:00\:00\:55\:66\:77
I1204 09:09:12.848179  316223 ovs.go:169] exec(77): stdout: ""
I1204 09:09:12.848196  316223 ovs.go:170] exec(77): stderr: ""

• Failure [0.008 seconds]
Management Port Operations
/home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:237
  sets up the management port for dual-stack clusters [It]
  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:343

  Unexpected error:
      <*errors.errorString | 0xc00064a210>: {
          s: "failed to add address fda6:0:0:1::2/64 on link ovn-k8s-mp0: operation not supported",
      }
      failed to add address fda6:0:0:1::2/64 on link ovn-k8s-mp0: operation not supported
  occurred

  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:142
------------------------------
Node Operations 
  sets correct OVN external IDs
  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/node_test.go:30
I1204 09:09:12.854651  316223 config.go:1373] Default config: {MTU:1400 ConntrackZone:64000 EncapType:geneve EncapIP: EncapPort:6081 InactivityProbe:100000 OpenFlowProbe:180 RawClusterSubnets:10.128.0.0/14/23 ClusterSubnets:[{CIDR:10.128.0.0/14 HostSubnetLength:23}]}
I1204 09:09:12.854704  316223 config.go:1374] Logging config: {File: CNIFile:/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log Level:5 LogFileMaxSize:100 LogFileMaxBackups:5 LogFileMaxAge:5}
I1204 09:09:12.854731  316223 config.go:1375] CNI config: {ConfDir:/etc/cni/net.d Plugin:ovn-k8s-cni-overlay}
I1204 09:09:12.854752  316223 config.go:1376] Kubernetes config: {Kubeconfig: CACert: APIServer:http://localhost:8443 Token: CompatServiceCIDR: RawServiceCIDRs:172.16.1.0/24 ServiceCIDRs:[172.16.1.0/24] OVNConfigNamespace:ovn-kubernetes MetricsBindAddress: OVNMetricsBindAddress: MetricsEnablePprof:false OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:nil}
I1204 09:09:12.854792  316223 config.go:1377] Gateway config: {Mode: Interface: NextHop: VLANID:0 NodeportEnable:false DisableSNATMultipleGWs:false}
I1204 09:09:12.854814  316223 config.go:1378] OVN North config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:true externalID:ovn-nb exec:0xc00033c0c0}
I1204 09:09:12.854840  316223 config.go:1379] OVN South config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:false externalID:ovn-remote exec:0xc00033c0c0}
I1204 09:09:12.854863  316223 config.go:1380] Hybrid Overlay config: {Enabled:false RawClusterSubnets: ClusterSubnets:[] VXLANPort:4789}
I1204 09:09:12.854917  316223 ovs.go:166] exec(78): /fake-bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.2.5.6 external_ids:ovn-remote-probe-interval=100000 external_ids:ovn-openflow-probe-interval=180 external_ids:hostname="cannot.be.resolv.ed" external_ids:ovn-monitor-all=true
I1204 09:09:12.854934  316223 ovs.go:169] exec(78): stdout: ""
I1204 09:09:12.854947  316223 ovs.go:170] exec(78): stderr: ""
•
------------------------------
Node Operations 
  sets non-default OVN encap port
  /home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/node_test.go:80
I1204 09:09:12.857335  316223 config.go:1373] Default config: {MTU:1400 ConntrackZone:64000 EncapType:geneve EncapIP: EncapPort:6081 InactivityProbe:100000 OpenFlowProbe:180 RawClusterSubnets:10.128.0.0/14/23 ClusterSubnets:[{CIDR:10.128.0.0/14 HostSubnetLength:23}]}
I1204 09:09:12.857382  316223 config.go:1374] Logging config: {File: CNIFile:/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log Level:5 LogFileMaxSize:100 LogFileMaxBackups:5 LogFileMaxAge:5}
I1204 09:09:12.857405  316223 config.go:1375] CNI config: {ConfDir:/etc/cni/net.d Plugin:ovn-k8s-cni-overlay}
I1204 09:09:12.857422  316223 config.go:1376] Kubernetes config: {Kubeconfig: CACert: APIServer:http://localhost:8443 Token: CompatServiceCIDR: RawServiceCIDRs:172.16.1.0/24 ServiceCIDRs:[172.16.1.0/24] OVNConfigNamespace:ovn-kubernetes MetricsBindAddress: OVNMetricsBindAddress: MetricsEnablePprof:false OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:nil}
I1204 09:09:12.857459  316223 config.go:1377] Gateway config: {Mode: Interface: NextHop: VLANID:0 NodeportEnable:false DisableSNATMultipleGWs:false}
I1204 09:09:12.857479  316223 config.go:1378] OVN North config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:true externalID:ovn-nb exec:0xc000248940}
I1204 09:09:12.857504  316223 config.go:1379] OVN South config: {Address: PrivKey: Cert: CACert: CertCommonName: Scheme:unix northbound:false externalID:ovn-remote exec:0xc000248940}
I1204 09:09:12.857527  316223 config.go:1380] Hybrid Overlay config: {Enabled:false RawClusterSubnets: ClusterSubnets:[] VXLANPort:4789}
I1204 09:09:12.857573  316223 ovs.go:166] exec(79): /fake-bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.2.5.6 external_ids:ovn-remote-probe-interval=100000 external_ids:ovn-openflow-probe-interval=180 external_ids:hostname="cannot.be.resolv.ed" external_ids:ovn-monitor-all=true
I1204 09:09:12.857592  316223 ovs.go:169] exec(79): stdout: ""
I1204 09:09:12.857607  316223 ovs.go:170] exec(79): stderr: ""
I1204 09:09:12.857633  316223 ovs.go:166] exec(80): /fake-bin/ovs-vsctl --timeout=15 --if-exists get Open_vSwitch . external_ids:system-id
I1204 09:09:12.857648  316223 ovs.go:169] exec(80): stdout: "1a3dfc82-2749-4931-9190-c30e7c0ecea3"
I1204 09:09:12.857665  316223 ovs.go:170] exec(80): stderr: ""
I1204 09:09:12.857693  316223 ovs.go:166] exec(81): /fake-bin/ovn-sbctl --timeout=15 --data=bare --no-heading --columns=_uuid find Encap chassis_name=1a3dfc82-2749-4931-9190-c30e7c0ecea3
I1204 09:09:12.857706  316223 ovs.go:169] exec(81): stdout: "e4437094-0094-4223-9f14-995d98d5fff8"
I1204 09:09:12.857724  316223 ovs.go:170] exec(81): stderr: ""
I1204 09:09:12.857750  316223 ovs.go:166] exec(82): /fake-bin/ovn-sbctl --timeout=15 set encap e4437094-0094-4223-9f14-995d98d5fff8 options:dst_port=666
I1204 09:09:12.857763  316223 ovs.go:169] exec(82): stdout: ""
I1204 09:09:12.857773  316223 ovs.go:170] exec(82): stderr: ""
•
JUnit path was configured: /home/asood/github-projects/ovn-kubernetes/go-controller/_artifacts/junit-pkg_node.xml

JUnit report was created: /home/asood/github-projects/ovn-kubernetes/go-controller/_artifacts/junit-pkg_node.xml


Summarizing 2 Failures:

[Fail] Management Port Operations [It] sets up the management port for IPv6 clusters 
/home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:142

[Fail] Management Port Operations [It] sets up the management port for dual-stack clusters 
/home/asood/github-projects/ovn-kubernetes/go-controller/pkg/node/management-port_linux_test.go:142

Ran 32 of 32 Specs in 0.531 seconds
FAIL! -- 30 Passed | 2 Failed | 0 Pending | 0 Skipped
--- FAIL: TestNodeSuite (0.53s)
FAIL
coverage: 38.6% of statements
make: *** [Makefile:30: check] Error 1
[asood@asood go-controller]$
Comment 5 Alexander Constantinescu 2020-12-08 09:33:16 UTC 
Hi 

I've verified that it works, you can move the bug to VERIFIED

Steps to verify on Fedora:

$ git clone https://github.com/openshift/ovn-kubernetes.git
$ git checkout origin/release-4.6
$ cd go-controller
$ make check
$ echo $?

Check that last step returns 0

Hope this is clearer

/Alexander
Comment 6 Arti Sood 2020-12-08 13:13:37 UTC 
Marking it verified based on Alexander 's comment #5.

On RHEL 8 even with Ipv6 enabled I get the output of the last step as follows:-

[asood@asood go-controller]$ echo $?
2
Comment 8 errata-xmlrpc 2020-12-14 13:50:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5259
Comment 9 Arti Sood 2020-12-15 18:42:35 UTC 
After fixing the issue of ipv6 addresses configuration on the box. I was able to get a successful execution of 'make check'



Ran 14 of 14 Specs in 2.655 seconds
SUCCESS! -- 14 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestHybridOverlayControllerSuite (2.66s)
PASS
coverage: 56.9% of statements
[asood@asood go-controller]$  echo $?
0