Bug 1891771

Summary: dual-stack deployment failed on bootstrap due to kube-apiserver CrashLoopBackOff
Product: OpenShift Container Platform Reporter: Shelly Miron <smiron>
Component: InstallerAssignee: Dan Winship <danw>
Installer sub component: OpenShift on Bare Metal IPI QA Contact: Amit Ugol <augol>
Status: CLOSED DEFERRED Docs Contact:
Severity: medium    
Priority: unspecified CC: kiran, prabinov, shardy
Version: 4.7Keywords: Triaged, UpcomingSprint
Target Milestone: ---Flags: smiron: needinfo+
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-09 14:46:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
openshift_install.log none

Description Shelly Miron 2020-10-27 10:26:02 UTC 
Created attachment 1724518 [details]
openshift_install.log

Versions:
-----------

Client Version: 4.7.0-0.nightly-2020-10-23-185958
Server Version: 4.7.0-0.nightly-2020-10-23-185958
Kubernetes Version: v1.19.0+e67f5dc

Platform:
-----------

libvirt
IPI (automated install with `openshift-baremetal-install`) 

Description:
-------------

Deployed disconnected dual-stack OCP4.7 env, with the below install-config:


apiVersion: v1
baseDomain: qe.lab.redhat.com
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 192.168.123.0/24
  - cidr: fd2e:6f44:5dd8::/64
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  - cidr: fd01::/48
    hostPrefix: 64
  serviceNetwork:
  - 172.30.0.0/16
  - fd02::/112
metadata:
  name: ocp-edge-cluster-0
compute:
- name: worker
  replicas: 2
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    provisioningNetwork: Managed
    externalBridge: baremetal-0 
    provisioningBridge: provisioning-0 
    libvirtURI: qemu+ssh://root.qe.lab.redhat.com/system
    provisioningNetworkInterface: enp4s0
    provisioningNetworkCIDR: fd00:1101::/64
    bootstrapOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-qemu.x86_64.qcow2.gz?sha256=fb27322b0269b2732021f7b2468180a7d9ebb06f598ca469513dc12960e26608
    clusterOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-openstack.x86_64.qcow2.gz?sha256=6904c9047c26a3d37fbea97c5b03dba22c0ec1cbd8221c2857c1719ba4670365
    apiVIP: fd2e:6f44:5dd8::5
    dnsVIP: fd2e:6f44:5dd8:0:0:0:0:2
    ingressVIP: fd2e:6f44:5dd8::10
    hosts:
      - name: openshift-master-0-0
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/c5cd4050-ea52-4bed-b695-fe7cad2ed835
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:54:bc:8d
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-master-0-1
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/89da58ae-c456-48e4-bdf4-66c3440b4e05
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:a6:c3:e6
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-master-0-2
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/121032ab-e2d5-4964-95c8-0dd04f36bbef
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:8c:d9:8a
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-worker-0-0
        role: worker
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/dd74f3b3-f3c7-45dd-8f7c-cee787263b41
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:a0:ac:ec
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-worker-0-1
        role: worker
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/49d07e37-8a6a-44b4-ab7e-b6e3578c7722
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:34:9f:0c
        rootDeviceHints:
          deviceName: /dev/sda
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIEVzCCAz+gAwIBAgIUBhw083k0DKgHnWvzfblE7StgJmkwDQYJKoZIhvcNAQEL
  BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln
  aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD
  VQQDDChzZWFsdXNhMjkubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X
  DTIwMTAyNjExMTIwOFoXDTIxMTAyNjExMTIwOFowgYgxCzAJBgNVBAYTAlVTMQsw
  CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w
  YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMjkubW9iaXVz
  LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
  MIIBCgKCAQEAnClmhDo9Ni14uFFBMp0Cw8dFnHSOXnNA+tDFc2O4nqsxnWtg5jh5
  X8GQweCn34fpL9Kx7iVJemsN35WaVu4H8r5icxieEI5VgiHZLHtPgb6Gw3NokKhN
  oiwsVjaX6A6Ze9PWAKCOA3GqyKsMFo7DXTGzz8xz5krALNfNNf006vn1mueyJ3/G
  /y3UY/8O7TDVDIDD7LM/lh7UF0GvoFSDinbRJSic/0x1dGCwB6gmsyVqzp6/1nXj
  Z3LjAjBrfd87gcJ00K0dYgOFdzComGNRLG7x2M440vuB2imISJ08/pLMA+x+c+zO
  aqgv2Xicfk2bqNMmgmFSktJILHHrkhPtIQIDAQABo4G2MIGzMA8GA1UdEwQIMAYB
  Af8CAQAwgZ8GA1UdEQSBlzCBlIItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w
  LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMjmHEP6AAAAAAAAA/FQA//4N1MWH
  EP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//4SvrCHECYgAFIAAAlMJm6W
  //7Rj8qHEP6AAAAAAAAAJm6W//7Rj8owDQYJKoZIhvcNAQELBQADggEBAC8AE/df
  SYplzJWaa1bkFMrP4hLXPA5uVjA1J+rsO6YZX9DpZIDzL0pJuinXLmU14wnQDeVu
  hukjXBw+kc/If/0FOIacp78+86rluI3oIo36f3ZjnxTUoqW7iju4UfQfcC9f9jpJ
  Qo4wYCmoVEYoWimVKD/LksRJa0a7FirWwPybBz88XUJheWhfQmPAZuQtukUA+8j6
  vVb4Q+jjfNYBPb9ybAkZa5xIrAT41ivXX23t+N5rGuqRVOoNdPpSyrPszTyapxcP
  A5U59boSc1yex0mnOwPsFZXGhN3GpYu6ovOSQO6gKCC4SIopNJ7WhBZwtZLcvbmX
  6ooIsDn+zBQkpoI=
  -----END CERTIFICATE-----
pullSecret: |
  { "auths": { "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000": { "auth": "b2NwLWVkZ2U6b2NwLWVkZ2UtcGFzcw==" } }}
fips: false
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfrCLPCFmbQVI9KWRmEJtdDQs2r9wNd3xrQGilt+8pEbN/tnpwM99FV1oI+R1A/y9p06Bcycmct3jdbGbYfwhWetVoc1SkiCdO329ZXIlRIdRe1RTNgkwpB3k66waen2tnILuT/T/ClBlmZSq+x6JOvXRy11TFRuRej7d+zkd3lnnNGNzNLun/g8kmQBFwivexkTeWje9W1JYJQkUwQNYH3+ITQ/K5Tw5pf0h/ZTSABX+w/i6T3DcGy5QX1C08oSDJP92aOYFHGJFiSfvQCOm+Or6cR1LeSYLhRjuwiBMYYl4iiuHkeBuZ4RUryTYyicGLdAZ7t3S6R3hCHG7x+eU5eE6GO7Iq6C22Z6Ga29y8E3nrD1Krg+YOt2zWyyTHx6TcOy3l7xvoo1EGUv7QnJZ/+/wAQwwDE5zEtRqOUS6y3J6Rvk+l8pChlUo0gPnTjphDuwWBsyAEZt28udAGa4nz0ORTEQ7sBB70Y7yCTRLE0NZNA2Sj7BY+PgWvs8bGjvM= kni.qe.lab.redhat.com
imageContentSources:
- mirrors:
  - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image
  source: registry.svc.ci.openshift.org/ocp/release

We noticed that when the apiVIP hold IPV6 address, the deployment failed each time on the bootstrap with an error: no route to host.
(BTW, When the apiVIP hold IPV4 address, the deployment succeed but the endpoints and pods hold only IPV4 addresses). 

Then when we search for CrashLoopBackOff pods, we see this outcome:

$ oc get pods -A | grep -v Running | grep -v Complete

..
....
openshift-kube-apiserver                           kube-apiserver-master-0-0                                 3/5     CrashLoopBackOff   9          5m38s
openshift-kube-apiserver                           kube-apiserver-master-0-1                                 3/5     CrashLoopBackOff   61         97m
openshift-kube-apiserver                           kube-apiserver-master-0-2                                 3/5     CrashLoopBackOff   63         101m
..
....

and when we look into the logs of one of them:

$ oc -n openshift-kube-apiserver logs kube-apiserver-master-0-0

..
.....
..........
dial tcp [fd2e:6f44:5dd8::144]:2379: connect: connection refused". Reconnecting...
W1027 08:10:02.033873      19 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://[fd2e:6f44:5dd8::100]:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [fd2e:6f44:5dd8::100]:2379: connect: connection refused". Reconnecting...
W1027 08:10:02.060731      19 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://[fd2e:6f44:5dd8::142]:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [fd2e:6f44:5dd8::142]:2379: connect: connection refused". Reconnecting...
W1027 08:10:02.831689      19 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://[fd2e:6f44:5dd8::144]:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [fd2e:6f44:5dd8::144]:2379: connect: connection refused". Reconnecting...
..
.......
...........

Eventually when we see the logs for the etc pod, we can tell that the service expect the Internal IPV's of the nodes to be IPV4, although they have IPV6 address:

$ crictl logs 3c869c9243b0c

I1027 07:57:27.665095       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2020-10-26T11:40:00Z","message":"EtcdMembersControllerDegraded: failed to get internal IP for node: no matches found for ip family \"tcp4\" for node \"master-0-0\"\nBootstrapTeardownDegraded: failed to get internal IP for node: no matches found for ip family \"tcp4\" for node \"master-0-1\"\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-scripts, configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1,restore-etcd-pod-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nStaticPodsDegraded: pods \"etcd-master-0-2\" not found\nStaticPodsDegraded: pods \"etcd-master-0-1\" not found\nStaticPodsDegraded: pods \"etcd-master-0-0\" not found\nClusterMemberControllerDegraded: failed to get internal IP for node: no matches found for ip family \"tcp4\" for node \"master-0-0\"\nEnvVarControllerDegraded: no matches found for ip family \"tcp4\" for node \"master-0-2\"","reason":"BootstrapTeardown_Error::ClusterMemberController_Error::EnvVarController_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::InstallerController_Error::RevisionController_ContentCreationError::ScriptController_Error::StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-10-26T11:39:28Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-10-26T11:38:41Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2020-10-26T11:39:21Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I1027 07:57:27.676736       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"openshift-etcd-operator", Name:"openshift-etcd-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
E1027 07:57:27.687958       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: failed to get internal IP for node: no matches found for ip family "tcp4" for node "master-0-0"
E1027 07:57:27.688630       1 base_controller.go:250] "TargetConfigController" controller failed to sync "key", err: missing env var values
E1027 07:57:27.688657       1 envvarcontroller.go:197] key failed with : no matches found for ip family "tcp4" for node "master-0-2"
E1027 07:57:27.688816       1 base_controller.go:250] "ClusterMemberController" controller failed to sync "key", err: failed to get internal IP for node: no matches found for ip family "tcp4" for node "master-0-0"
E1027 07:57:27.688864       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: failed to get internal IP for node: no matches found for ip family "tcp4" for node "master-0-0"


$ oc get nodes -o wide

NAME         STATUS   ROLES    AGE   VERSION           INTERNAL-IP           EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
master-0-0   Ready    master   20h   v1.19.0+e67f5dc   fd2e:6f44:5dd8::100   <none>        Red Hat Enterprise Linux CoreOS 47.82.202010231442-0 (Ootpa)   4.18.0-193.28.1.el8_2.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.el8
master-0-1   Ready    master   20h   v1.19.0+e67f5dc   fd2e:6f44:5dd8::142   <none>        Red Hat Enterprise Linux CoreOS 47.82.202010231442-0 (Ootpa)   4.18.0-193.28.1.el8_2.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.el8
master-0-2   Ready    master   20h   v1.19.0+e67f5dc   fd2e:6f44:5dd8::144   <none>        Red Hat Enterprise Linux CoreOS 47.82.202010231442-0 (Ootpa)   4.18.0-193.28.1.el8_2.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.el8



Bootstrap log bundle:
-------------------------------------

https://drive.google.com/file/d/1oESIX60agCtpVXDJrlx5ekMLxJ9ucqPk/view?usp=sharing


.openshift_install.log:
-------------------------------------
attached to the bug.


Expected results:
-------------------------------------
dual-stack env deployed successfully, when the endpoints (at least) holds IPV6 and IPV4 addresses. 


Actual results:
-------------------------------------
dual-stack deployment failed on bootstrap due to kube-apiserver CrashLoopBackOff


How to reproduce:
-------------------------------------
1. Deploy dual-stack env, and config the install-config.yaml file so that the apiVIP holds IPV6 address.
Comment 2 Dan Winship 2020-10-27 18:36:36 UTC 
> (BTW, When the apiVIP hold IPV4 address, the deployment succeed but the endpoints and pods
> hold only IPV4 addresses). 

That's half right: your pods _are_ actually getting dual-stack IPs, as seen in the pods.json from the log bundle:

                "podIP": "10.129.0.15",
                "podIPs": [
                    {
                        "ip": "10.129.0.15"
                    },
                    {
                        "ip": "fd01:0:0:2::f"
                    }
                ],

But to preserve compatibility with scripts that parse oc/kubectl output, `oc get` still shows the single-valued IP columns by default, so `oc get pods` will only show IP addresses of a single family (specifically, the family listed first in your install-config's `clusterNetwork` section, which is considered the default family, at least for pods).

However, that only applies to pods: your services/endpoints/nodes really are single stack, because OCP master is still based on kubernetes 1.19, which does not have full dual-stack support. A given Service can be either IPv4 or IPv6, but all of the Services created by OCP at install time will be of the "default" family (the family listed first in the `serviceNetwork` section of the install-config). Nodes OTOH will end up getting an IP of the same family as the `apiVIP`. Thus, given the install-config above, the apiserver service will have an IPv4 Service IP, but IPv6 endpoints, which isn't going to work very well. Likewise, something about the way the etcd configuration is being generated seems to be expecting IPv4 node IPs, but getting IPv6 ones.

So:

  1. I _think_ that if you change the install-config to list the IPv6 networks first in `clusterNetwork` and `serviceNetwork`, then everything should work. (Can you test that?)

  2. We probably need to fix the installer to realize this and give an error for the install-config as you wrote it here
Comment 4 Shelly Miron 2020-10-28 10:43:21 UTC 
This is the new install-config.yaml with the changes suggested (IPv6 networks first in `clusterNetwork` and `serviceNetwork`):

[kni@provisionhost-0-0 ~]$ cat install-config.yaml 
apiVersion: v1
baseDomain: qe.lab.redhat.com
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 192.168.123.0/24
  - cidr: fd2e:6f44:5dd8::/64
  clusterNetwork:
  - cidr: fd01::/48
    hostPrefix: 64
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - fd02::/112
  - 172.30.0.0/16
metadata:
  name: ocp-edge-cluster-0
compute:
- name: worker
  replicas: 2
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    provisioningNetwork: Managed
    externalBridge: baremetal-0 
    provisioningBridge: provisioning-0 
    libvirtURI: qemu+ssh://root.qe.lab.redhat.com/system
    provisioningNetworkInterface: enp4s0
    provisioningNetworkCIDR: fd00:1101::/64
    bootstrapOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-qemu.x86_64.qcow2.gz?sha256=fb27322b0269b2732021f7b2468180a7d9ebb06f598ca469513dc12960e26608
    clusterOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-openstack.x86_64.qcow2.gz?sha256=6904c9047c26a3d37fbea97c5b03dba22c0ec1cbd8221c2857c1719ba4670365
    apiVIP: fd2e:6f44:5dd8::5
    dnsVIP: fd2e:6f44:5dd8:0:0:0:0:2
    ingressVIP: fd2e:6f44:5dd8::10
    hosts:
      - name: openshift-master-0-0
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/5262aff3-2fac-4758-9c66-ed1653bedfef
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:75:00:90
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-master-0-1
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/9738b08f-07eb-43af-8ad8-98ff6e373229
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:48:5a:4a
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-master-0-2
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/8da913c4-069c-4e72-8309-c5fdc44e7794
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:05:e6:38
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-worker-0-0
        role: worker
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/ed948fe6-aeee-4300-abbc-30d809cabd7a
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:d2:33:af
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-worker-0-1
        role: worker
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/8e041ccc-e2bd-4582-be6c-771ee009f83d
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:fa:aa:9f
        rootDeviceHints:
          deviceName: /dev/sda
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIEVzCCAz+gAwIBAgIUfeyfz6pPL4HU4brQaMW0zgQR29owDQYJKoZIhvcNAQEL
  BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln
  aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD
  VQQDDChzZWFsdXNhMjkubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X
  DTIwMTAyODA3MzYwM1oXDTIxMTAyODA3MzYwM1owgYgxCzAJBgNVBAYTAlVTMQsw
  CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w
  YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMjkubW9iaXVz
  LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
  MIIBCgKCAQEAp/wq/bJwil/4dU1AHCrhBlLMUurPQRazzFXqjbK4Ksxn3k39tIh6
  Gjdqf+Ao/IwAeInkO7jTzfy5MN5sHCKlomIlcDbvNbK/G5m0KMjEp74IV7p8g4Vr
  VI7s1Gcv5lx947kRmWx2Mm+EVZmGQKVpn4rYTUq+6CT4BGYohMo6bAxhcO7q1mK7
  +Z5l9w/YpXw+xyDubup0Ji7evzyq1Y1f8bhx5yneek57kFuNAYo6Z7pedu373jlZ
  ezr/Yhocwka5N6JIRov++4lnUlZ/A1xX8vIAaHDsP+STG3clYMVGHY1OPo5GkxoR
  zbM2ONYq+IzF2xtzzDHbpIwS3huaqa+tGwIDAQABo4G2MIGzMA8GA1UdEwQIMAYB
  Af8CAQAwgZ8GA1UdEQSBlzCBlIItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w
  LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMjmHEP6AAAAAAAAA/FQA//75E46H
  EP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//500CCHECYgAFIAAAlMJm6W
  //7Rj8qHEP6AAAAAAAAAJm6W//7Rj8owDQYJKoZIhvcNAQELBQADggEBAAgdo+KI
  OJvTZrVkROXIL+/DpPeS0/09ioiTrsGUL7Na447ebeCXK8HJMbZ9a7G7oiOjt9V+
  jN7GZtAIiMmarnkmK4qhT/1uNHd2R8JobZcbsEd2tTM6EF9JBDe9yY0QXfN0tcr3
  TK/U9y5m9KUzU82WE8NJsRoGKwKjnd//yhzMihO0e1TQ5nB5m6vIMVes4RETc3hd
  hgk/2SOJcsyS9+FEmheMGu28LM4ekjKZZhm/VBnu1LbUC8OFRVuViNesjl5HsKQS
  XPCTLdd/mlIoVdUp/tBcU3xrsFE7cJg9h5mwmWUqqVst55VEqiYceNFR9m0xg4qY
  B1ee7jZD6NtfIkI=
  -----END CERTIFICATE-----
pullSecret: |
  { "auths": { "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000": { "auth": "b2NwLWVkZ2U6b2NwLWVkZ2UtcGFzcw==" } }}
fips: false
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr8zXc3x+geVx0aKDsamBhl1mBOTbEMO3jcG6iP0KEq93x5DyyKOiPUfVHruxATkzpZlKcsuqRu+GMcFntQy7lYiMtPMQZ1kf/2J/1lQ1Y/x2CKgsKUK2oGAQrWEBpAGqomeq9c5o3hWuNFkWHQ/9z438l/Ob3TahTjyK81Gl+e7FBdgPAhEDS1fjZlSM/HSTtQUMXR58eJIR5+PVS6HtuTwKjYJFlxgLv7Fx61yRmCXfDXvDuDbYM965IrTlFLwp8VSt8XBPdAGaEMnrAqpIYswW4Ot2n1k9mmArkdEln2WCHka/oMjWl1Dg+5eN3jsAfiFOsi/CYkFglTeIAb2VdKJygkLwxWed8a3vyzLyg3mcCOXxjh5WuJUUMpv8K/I/9LRaERzSVWiH5pG0Bdp6q+UmgFSMsaCGW6r6yMM5HKrASR8Q2K2vR64CeeFZ0utQ4vQdk1oiJS788FDGLyFgs1rfdtFoq092WOhr2pKSDe89bU8+txkh6R57GAHS5qhc= kni.qe.lab.redhat.com
imageContentSources:
- mirrors:
  - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image
  source: registry.svc.ci.openshift.org/ocp/release


And yet, the deployment failed on the bootstrap with: 

FATAL Bootstrap failed to complete: failed to wait for bootstrapping to complete: timed out waiting for the condition

[kni@provisionhost-0-0 ~]$ oc get nodes -A -o wide
NAME         STATUS     ROLES    AGE   VERSION           INTERNAL-IP           EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
master-0-0   NotReady   master   29m   v1.19.0+e67f5dc   fd2e:6f44:5dd8::11a   <none>        Red Hat Enterprise Linux CoreOS 47.82.202010270142-0 (Ootpa)   4.18.0-193.28.1.el8_2.x86_64   cri-o://1.20.0-0.rhaos4.7.gitec53db4.el8-dev
master-0-1   NotReady   master   29m   v1.19.0+e67f5dc   fd2e:6f44:5dd8::109   <none>        Red Hat Enterprise Linux CoreOS 47.82.202010270142-0 (Ootpa)   4.18.0-193.28.1.el8_2.x86_64   cri-o://1.20.0-0.rhaos4.7.gitec53db4.el8-dev
master-0-2   NotReady   master   29m   v1.19.0+e67f5dc   fd2e:6f44:5dd8::105   <none>        Red Hat Enterprise Linux CoreOS 47.82.202010270142-0 (Ootpa)   4.18.0-193.28.1.el8_2.x86_64   cri-o://1.20.0-0.rhaos4.7.gitec53db4.el8-dev

[kni@provisionhost-0-0 ~]$ oc get pods -A -o wide
NAMESPACE                                          NAME                                                      READY   STATUS              RESTARTS   AGE    IP                    NODE         NOMINATED NODE   READINESS GATES
openshift-apiserver-operator                       openshift-apiserver-operator-5d69cbbdf-pmxjs              0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-authentication-operator                  authentication-operator-7954964f8d-fvdvf                  0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-cloud-credential-operator                cloud-credential-operator-7fb659d948-ltxpf                0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-cluster-machine-approver                 machine-approver-7b45cc85f-lp4z9                          0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-cluster-node-tuning-operator             cluster-node-tuning-operator-7c5dd85979-nhcch             0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-cluster-storage-operator                 cluster-storage-operator-598dbcb96-k8hqj                  0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-cluster-storage-operator                 csi-snapshot-controller-operator-789c9877f-7mcq8          0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-cluster-version                          cluster-version-operator-868f7c8bcb-smds6                 0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-config-operator                          openshift-config-operator-84dbd98479-ghmtv                0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-controller-manager-operator              openshift-controller-manager-operator-bf7b9697c-ppm2m     0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-dns-operator                             dns-operator-574b845588-jtw7z                             0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-etcd-operator                            etcd-operator-7f86f6cc49-ss8fl                            0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-image-registry                           cluster-image-registry-operator-b8688d866-vstm4           0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-ingress-operator                         ingress-operator-74c8747fd4-7czxr                         0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-insights                                 insights-operator-5d6cd95c7d-6c98c                        0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-kni-infra                                coredns-master-0-0                                        2/2     Running             0          29m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-kni-infra                                coredns-master-0-1                                        2/2     Running             0          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-kni-infra                                coredns-master-0-2                                        2/2     Running             0          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-kni-infra                                haproxy-master-0-0                                        2/2     Running             0          29m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-kni-infra                                haproxy-master-0-1                                        2/2     Running             0          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-kni-infra                                haproxy-master-0-2                                        2/2     Running             0          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-kni-infra                                keepalived-master-0-0                                     1/2     Running             4          28m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-kni-infra                                keepalived-master-0-1                                     1/2     Running             4          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-kni-infra                                keepalived-master-0-2                                     1/2     Running             2          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-kni-infra                                mdns-publisher-master-0-0                                 1/1     Running             0          29m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-kni-infra                                mdns-publisher-master-0-1                                 1/1     Running             0          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-kni-infra                                mdns-publisher-master-0-2                                 1/1     Running             0          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-kube-apiserver-operator                  kube-apiserver-operator-dc58dbf6d-8pcx8                   0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-kube-controller-manager-operator         kube-controller-manager-operator-785cf947f8-f79w7         0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-kube-scheduler-operator                  openshift-kube-scheduler-operator-66dcfb7d98-mv8pn        0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-kube-storage-version-migrator-operator   kube-storage-version-migrator-operator-5758648f54-nwspv   0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-machine-api                              cluster-autoscaler-operator-7759576c89-n64hj              0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-machine-api                              machine-api-operator-5b547676ff-7lpht                     0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-machine-config-operator                  machine-config-operator-75c4bb4ccf-s9bkw                  0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-marketplace                              marketplace-operator-6bf7ddc8db-ww2wn                     0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-monitoring                               cluster-monitoring-operator-d747df89b-mvdgl               0/2     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-multus                                   multus-flfph                                              1/1     Running             2          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-multus                                   multus-rkfjs                                              1/1     Running             2          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-multus                                   multus-xdh7p                                              1/1     Running             2          28m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-multus                                   network-metrics-daemon-gj9hz                              0/2     ContainerCreating   0          28m    <none>                master-0-2   <none>           <none>
openshift-multus                                   network-metrics-daemon-hzmzc                              0/2     ContainerCreating   0          28m    <none>                master-0-1   <none>           <none>
openshift-multus                                   network-metrics-daemon-lk6jc                              0/2     ContainerCreating   0          28m    <none>                master-0-0   <none>           <none>
openshift-network-operator                         network-operator-8689749bb8-bmdj7                         1/1     Running             0          9m5s   fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-operator-lifecycle-manager               catalog-operator-5566c989c9-gw8xl                         0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-operator-lifecycle-manager               olm-operator-75d46f844c-dffrg                             0/1     Pending             0          43m    <none>                <none>       <none>           <none>
openshift-ovn-kubernetes                           ovnkube-master-6nr7w                                      4/6     CrashLoopBackOff    20         28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-ovn-kubernetes                           ovnkube-master-dchdt                                      4/6     CrashLoopBackOff    20         28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-ovn-kubernetes                           ovnkube-master-thjqh                                      4/6     CrashLoopBackOff    20         28m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-ovn-kubernetes                           ovnkube-node-4wpml                                        2/3     Running             6          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-ovn-kubernetes                           ovnkube-node-l96ng                                        2/3     CrashLoopBackOff    6          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-ovn-kubernetes                           ovnkube-node-lbcvg                                        2/3     Running             6          28m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-ovn-kubernetes                           ovs-node-8hrwj                                            1/1     Running             0          28m    fd2e:6f44:5dd8::109   master-0-1   <none>           <none>
openshift-ovn-kubernetes                           ovs-node-fk72k                                            1/1     Running             0          28m    fd2e:6f44:5dd8::105   master-0-2   <none>           <none>
openshift-ovn-kubernetes                           ovs-node-p5rss                                            1/1     Running             0          28m    fd2e:6f44:5dd8::11a   master-0-0   <none>           <none>
openshift-service-ca-operator                      service-ca-operator-797dff8569-8bsw7                      0/1     Pending             0          43m    <none>                <none>       <none>           <none>


[kni@provisionhost-0-0 ~]$ oc get endpoints -A -o wide
NAMESPACE                                          NAME                                       ENDPOINTS                                                                          AGE
default                                            kubernetes                                 192.168.123.124:6443                                                               44m
kube-system                                        kube-scheduler                             <none>                                                                             45m
openshift-apiserver-operator                       metrics                                    <none>                                                                             44m
openshift-authentication-operator                  metrics                                    <none>                                                                             44m
openshift-cloud-credential-operator                cco-metrics                                <none>                                                                             44m
openshift-cluster-machine-approver                 machine-approver                           <none>                                                                             44m
openshift-cluster-storage-operator                 csi-snapshot-controller-operator-metrics   <none>                                                                             44m
openshift-cluster-version                          cluster-version-operator                   <none>                                                                             44m
openshift-config-operator                          metrics                                    <none>                                                                             44m
openshift-controller-manager-operator              metrics                                    <none>                                                                             44m
openshift-dns-operator                             metrics                                    <none>                                                                             44m
openshift-etcd-operator                            metrics                                    <none>                                                                             44m
openshift-etcd                                     etcd                                       <none>                                                                             44m
openshift-etcd                                     host-etcd-2                                <none>                                                                             44m
openshift-image-registry                           image-registry-operator                    <none>                                                                             43m
openshift-ingress-operator                         metrics                                    <none>                                                                             44m
openshift-insights                                 metrics                                    <none>                                                                             43m
openshift-kube-apiserver-operator                  metrics                                    <none>                                                                             43m
openshift-kube-controller-manager-operator         metrics                                    <none>                                                                             44m
openshift-kube-scheduler-operator                  metrics                                    <none>                                                                             44m
openshift-kube-storage-version-migrator-operator   metrics                                    <none>                                                                             44m
openshift-machine-api                              cluster-autoscaler-operator                <none>                                                                             44m
openshift-machine-api                              machine-api-controllers                    <none>                                                                             43m
openshift-machine-api                              machine-api-operator                       <none>                                                                             43m
openshift-machine-api                              machine-api-operator-webhook               <none>                                                                             43m
openshift-machine-config-operator                  machine-config-daemon                      <none>                                                                             44m
openshift-marketplace                              marketplace-operator-metrics               <none>                                                                             44m
openshift-multus                                   multus-admission-controller                <none>                                                                             29m
openshift-multus                                   network-metrics-service                    <none>                                                                             29m
openshift-operator-lifecycle-manager               catalog-operator-metrics                   <none>                                                                             44m
openshift-operator-lifecycle-manager               olm-operator-metrics                       <none>                                                                             44m
openshift-ovn-kubernetes                           ovn-kubernetes-master                      [fd2e:6f44:5dd8::105]:9102,[fd2e:6f44:5dd8::109]:9102,[fd2e:6f44:5dd8::11a]:9102   29m
openshift-ovn-kubernetes                           ovn-kubernetes-node                        [fd2e:6f44:5dd8::105]:9103,[fd2e:6f44:5dd8::109]:9103,[fd2e:6f44:5dd8::11a]:9103   29m
openshift-ovn-kubernetes                           ovnkube-db                                                                                                                    29m
openshift-service-ca-operator                      metrics                                    <none>                                                                             44m


[kni@provisionhost-0-0 ~]$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                                                                            
cloud-credential                                     True        False         False      43m
cluster-autoscaler                                                                        
config-operator                                                                           
console                                                                                   
csi-snapshot-controller                                                                   
dns                                                                                       
etcd                                                                                      
image-registry                                                                            
ingress                                                                                   
insights                                                                                  
kube-apiserver                                                                            
kube-controller-manager                                                                   
kube-scheduler                                                                            
kube-storage-version-migrator                                                             
machine-api                                                                               
machine-approver                                                                          
machine-config                                                                            
marketplace                                                                               
monitoring                                                                                
network                                              False       True          True       30m
node-tuning                                                                               
openshift-apiserver                                                                       
openshift-controller-manager                                                              
openshift-samples                                                                         
operator-lifecycle-manager                                                                
operator-lifecycle-manager-catalog                                                        
operator-lifecycle-manager-packageserver                                                  
service-ca                                                                                
storage   


Bootstrap log bundle:
-------------------------------------
https://drive.google.com/file/d/19Tk24d5CHu81rixAS_-F9E-vN0COCuEl/view?usp=sharing


openshift_install.log:
-------------------------------------

https://drive.google.com/file/d/1riNKTEhwuDvXRv1th7szQLAyBKwzQKL9/view?usp=sharing
Comment 5 Dan Winship 2020-10-29 18:58:59 UTC 
Merged the upstream fix for the specific bug that broke your last test run: https://github.com/ovn-org/ovn-kubernetes/pull/1787
Not sure exactly when that will make it into OCP, or if that's the only remaining bug with the dual-stack-IPv6-primary case. It seems that dev-scripts won't let you create a dual-stack-IPv6-primary cluster so I can't easily test this myself.

(If you want to try testing it right away, you could ask @cluster-bot on Slack to "build openshift/ovn-kubernetes#332" and then try installing the resulting image)
Comment 6 Shelly Miron 2020-11-09 07:03:34 UTC 
We are going to wait for Kube 1.20 before continue testing dual stack
Comment 7 Dan Winship 2020-11-09 14:46:34 UTC 
(In reply to Shelly Miron from comment #6)
> We are going to wait for Kube 1.20 before continue testing dual stack

ok, closing this now then