Bug 1789124

Summary: Router doesn't listen on ipv6 interfaces when cluster network config indicates ipv6 support
Product: OpenShift Container Platform Reporter: Dan Mace <dmace>
Component: RoutingAssignee: Dan Mace <dmace>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.0CC: aos-bugs, dhansen, hongli, jschluet
Target Milestone: ---   
Target Release: 4.3.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: ipv6
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1789121 Environment:
Last Closed: 2020-02-19 05:39:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1789121, 1796618    
Bug Blocks:    

Description Dan Mace 2020-01-08 19:42:41 UTC
+++ This bug was initially created as a clone of Bug #1789121 +++

Description of problem:

When running on an IPv6-enabled cluster (defined as the presence of an ipv6 address in the network.config.openshift.io `.status.clusterNetwork` list), the router isn't listening on an ipv6 address.

This has already been fixed in https://github.com/openshift/cluster-ingress-operator/pull/342 but we'd like to backport the fix and document the change with a bug report.

Version-Release number of selected component (if applicable):


How reproducible:

Launch a single-stack IPv6 enabled cluster on AWS without the fix.

Actual results:


The router process won't listen on any ipv6 interface.

Expected results:

The router should listen on all available ipv4 and ipv6 interfaces.

Additional info:

Comment 1 Daneyon Hansen 2020-01-22 18:12:22 UTC
Removed PR 346. Superseded by PR 352.

Comment 3 Dan Winship 2020-02-07 14:25:27 UTC
Assigning all 4.3.z IPv6 bugs to Marius Cornea for QA, as they are not yet QA-able in stock release-4.3 builds.

Comment 4 Marius Cornea 2020-02-11 21:48:33 UTC
Verified on 4.3.0-0.nightly-2020-02-10-055634(included in 4.3.0-0.nightly-2020-02-10-055634-ipv6.3) on a bare metal deployment

Image used in local disconnected registry:
[kni@provisionhost-0 ~]$ oc adm release info --image-for=cluster-ingress-operator  -a ~/combined-secret.json   registry.ocp-edge-cluster.qe.lab.redhat.com:5000/localimages/local-release-image:4.3.0-0.nightly-2020-02-10-055634-ipv6.3
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298fb2499ff98a3dab790436afb5da961c951a7d22b8357ad7d82e6739e0128f

Image used in 4.3.0-0.nightly-2020-02-10-055634:
[kni@provisionhost-0 ~]$ oc adm release info --image-for=cluster-ingress-operator  -a ~/combined-secret.json   registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-02-10-055634
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298fb2499ff98a3dab790436afb5da961c951a7d22b8357ad7d82e6739e0128f


[kni@provisionhost-0 ~]$ oc get co/ingress
NAME      VERSION                                    AVAILABLE   PROGRESSING   DEGRADED   SINCE
ingress   4.3.0-0.nightly-2020-02-10-055634-ipv6.3   True        False         False      89m

[kni@provisionhost-0 ~]$ oc get network/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2020-02-11T20:04:57Z"
  generation: 2
  name: cluster
  resourceVersion: "1861"
  selfLink: /apis/config.openshift.io/v1/networks/cluster
  uid: 0cb632ca-766c-4959-8c66-187ecbb56579
spec:
  clusterNetwork:
  - cidr: fd01::/48
    hostPrefix: 64
  externalIP:
    policy: {}
  networkType: OVNKubernetes
  serviceNetwork:
  - fd02::/112
status:
  clusterNetwork:
  - cidr: fd01::/48
    hostPrefix: 64
  clusterNetworkMTU: 1400
  networkType: OVNKubernetes
  serviceNetwork:
  - fd02::/112


Ingress hostname is reacheable:

[kni@provisionhost-0 ~]$ curl -k https://test.apps.ocp-edge-cluster.qe.lab.redhat.com -I
HTTP/1.0 503 Service Unavailable
Pragma: no-cache
Cache-Control: private, max-age=0, no-cache, no-store
Connection: close
Content-Type: text/html

[kni@provisionhost-0 ~]$ curl -k https://test.apps.ocp-edge-cluster.qe.lab.redhat.com -I -v
* Rebuilt URL to: https://test.apps.ocp-edge-cluster.qe.lab.redhat.com/
*   Trying fd2e:6f44:5dd8:c956::10...
* TCP_NODELAY set
* Connected to test.apps.ocp-edge-cluster.qe.lab.redhat.com (fd2e:6f44:5dd8:c956::10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.apps.ocp-edge-cluster.qe.lab.redhat.com
*  start date: Feb 11 20:15:38 2020 GMT
*  expire date: Feb 10 20:15:39 2022 GMT
*  issuer: CN=ingress-operator@1581452136
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> HEAD / HTTP/1.1
> Host: test.apps.ocp-edge-cluster.qe.lab.redhat.com
> User-Agent: curl/7.61.1
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
HTTP/1.0 503 Service Unavailable
< Pragma: no-cache
Pragma: no-cache
< Cache-Control: private, max-age=0, no-cache, no-store
Cache-Control: private, max-age=0, no-cache, no-store
< Connection: close
Connection: close
< Content-Type: text/html
Content-Type: text/html

< 
* Excess found in a non pipelined read: excess = 3131 url = / (zero-length body)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify

Comment 6 errata-xmlrpc 2020-02-19 05:39:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0492