Created attachment 1724518 [details] openshift_install.log Versions: ----------- Client Version: 4.7.0-0.nightly-2020-10-23-185958 Server Version: 4.7.0-0.nightly-2020-10-23-185958 Kubernetes Version: v1.19.0+e67f5dc Platform: ----------- libvirt IPI (automated install with `openshift-baremetal-install`) Description: ------------- Deployed disconnected dual-stack OCP4.7 env, with the below install-config: apiVersion: v1 baseDomain: qe.lab.redhat.com networking: networkType: OVNKubernetes machineNetwork: - cidr: 192.168.123.0/24 - cidr: fd2e:6f44:5dd8::/64 clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 - cidr: fd01::/48 hostPrefix: 64 serviceNetwork: - 172.30.0.0/16 - fd02::/112 metadata: name: ocp-edge-cluster-0 compute: - name: worker replicas: 2 controlPlane: name: master replicas: 3 platform: baremetal: {} platform: baremetal: provisioningNetwork: Managed externalBridge: baremetal-0 provisioningBridge: provisioning-0 libvirtURI: qemu+ssh://root.qe.lab.redhat.com/system provisioningNetworkInterface: enp4s0 provisioningNetworkCIDR: fd00:1101::/64 bootstrapOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-qemu.x86_64.qcow2.gz?sha256=fb27322b0269b2732021f7b2468180a7d9ebb06f598ca469513dc12960e26608 clusterOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-openstack.x86_64.qcow2.gz?sha256=6904c9047c26a3d37fbea97c5b03dba22c0ec1cbd8221c2857c1719ba4670365 apiVIP: fd2e:6f44:5dd8::5 dnsVIP: fd2e:6f44:5dd8:0:0:0:0:2 ingressVIP: fd2e:6f44:5dd8::10 hosts: - name: openshift-master-0-0 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/c5cd4050-ea52-4bed-b695-fe7cad2ed835 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:54:bc:8d rootDeviceHints: deviceName: /dev/sda - name: openshift-master-0-1 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/89da58ae-c456-48e4-bdf4-66c3440b4e05 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:a6:c3:e6 rootDeviceHints: deviceName: /dev/sda - name: openshift-master-0-2 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/121032ab-e2d5-4964-95c8-0dd04f36bbef disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:8c:d9:8a rootDeviceHints: deviceName: /dev/sda - name: openshift-worker-0-0 role: worker bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/dd74f3b3-f3c7-45dd-8f7c-cee787263b41 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:a0:ac:ec rootDeviceHints: deviceName: /dev/sda - name: openshift-worker-0-1 role: worker bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/49d07e37-8a6a-44b4-ab7e-b6e3578c7722 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:34:9f:0c rootDeviceHints: deviceName: /dev/sda additionalTrustBundle: | -----BEGIN CERTIFICATE----- MIIEVzCCAz+gAwIBAgIUBhw083k0DKgHnWvzfblE7StgJmkwDQYJKoZIhvcNAQEL BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD VQQDDChzZWFsdXNhMjkubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X DTIwMTAyNjExMTIwOFoXDTIxMTAyNjExMTIwOFowgYgxCzAJBgNVBAYTAlVTMQsw CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMjkubW9iaXVz LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAnClmhDo9Ni14uFFBMp0Cw8dFnHSOXnNA+tDFc2O4nqsxnWtg5jh5 X8GQweCn34fpL9Kx7iVJemsN35WaVu4H8r5icxieEI5VgiHZLHtPgb6Gw3NokKhN oiwsVjaX6A6Ze9PWAKCOA3GqyKsMFo7DXTGzz8xz5krALNfNNf006vn1mueyJ3/G /y3UY/8O7TDVDIDD7LM/lh7UF0GvoFSDinbRJSic/0x1dGCwB6gmsyVqzp6/1nXj Z3LjAjBrfd87gcJ00K0dYgOFdzComGNRLG7x2M440vuB2imISJ08/pLMA+x+c+zO aqgv2Xicfk2bqNMmgmFSktJILHHrkhPtIQIDAQABo4G2MIGzMA8GA1UdEwQIMAYB Af8CAQAwgZ8GA1UdEQSBlzCBlIItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMjmHEP6AAAAAAAAA/FQA//4N1MWH EP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//4SvrCHECYgAFIAAAlMJm6W //7Rj8qHEP6AAAAAAAAAJm6W//7Rj8owDQYJKoZIhvcNAQELBQADggEBAC8AE/df SYplzJWaa1bkFMrP4hLXPA5uVjA1J+rsO6YZX9DpZIDzL0pJuinXLmU14wnQDeVu hukjXBw+kc/If/0FOIacp78+86rluI3oIo36f3ZjnxTUoqW7iju4UfQfcC9f9jpJ Qo4wYCmoVEYoWimVKD/LksRJa0a7FirWwPybBz88XUJheWhfQmPAZuQtukUA+8j6 vVb4Q+jjfNYBPb9ybAkZa5xIrAT41ivXX23t+N5rGuqRVOoNdPpSyrPszTyapxcP A5U59boSc1yex0mnOwPsFZXGhN3GpYu6ovOSQO6gKCC4SIopNJ7WhBZwtZLcvbmX 6ooIsDn+zBQkpoI= -----END CERTIFICATE----- pullSecret: | { "auths": { "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000": { "auth": "b2NwLWVkZ2U6b2NwLWVkZ2UtcGFzcw==" } }} fips: false sshKey: | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfrCLPCFmbQVI9KWRmEJtdDQs2r9wNd3xrQGilt+8pEbN/tnpwM99FV1oI+R1A/y9p06Bcycmct3jdbGbYfwhWetVoc1SkiCdO329ZXIlRIdRe1RTNgkwpB3k66waen2tnILuT/T/ClBlmZSq+x6JOvXRy11TFRuRej7d+zkd3lnnNGNzNLun/g8kmQBFwivexkTeWje9W1JYJQkUwQNYH3+ITQ/K5Tw5pf0h/ZTSABX+w/i6T3DcGy5QX1C08oSDJP92aOYFHGJFiSfvQCOm+Or6cR1LeSYLhRjuwiBMYYl4iiuHkeBuZ4RUryTYyicGLdAZ7t3S6R3hCHG7x+eU5eE6GO7Iq6C22Z6Ga29y8E3nrD1Krg+YOt2zWyyTHx6TcOy3l7xvoo1EGUv7QnJZ/+/wAQwwDE5zEtRqOUS6y3J6Rvk+l8pChlUo0gPnTjphDuwWBsyAEZt28udAGa4nz0ORTEQ7sBB70Y7yCTRLE0NZNA2Sj7BY+PgWvs8bGjvM= kni.qe.lab.redhat.com imageContentSources: - mirrors: - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image source: registry.svc.ci.openshift.org/ocp/release We noticed that when the apiVIP hold IPV6 address, the deployment failed each time on the bootstrap with an error: no route to host. (BTW, When the apiVIP hold IPV4 address, the deployment succeed but the endpoints and pods hold only IPV4 addresses). Then when we search for CrashLoopBackOff pods, we see this outcome: $ oc get pods -A | grep -v Running | grep -v Complete .. .... openshift-kube-apiserver kube-apiserver-master-0-0 3/5 CrashLoopBackOff 9 5m38s openshift-kube-apiserver kube-apiserver-master-0-1 3/5 CrashLoopBackOff 61 97m openshift-kube-apiserver kube-apiserver-master-0-2 3/5 CrashLoopBackOff 63 101m .. .... and when we look into the logs of one of them: $ oc -n openshift-kube-apiserver logs kube-apiserver-master-0-0 .. ..... .......... dial tcp [fd2e:6f44:5dd8::144]:2379: connect: connection refused". Reconnecting... W1027 08:10:02.033873 19 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://[fd2e:6f44:5dd8::100]:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [fd2e:6f44:5dd8::100]:2379: connect: connection refused". Reconnecting... W1027 08:10:02.060731 19 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://[fd2e:6f44:5dd8::142]:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [fd2e:6f44:5dd8::142]:2379: connect: connection refused". Reconnecting... W1027 08:10:02.831689 19 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://[fd2e:6f44:5dd8::144]:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [fd2e:6f44:5dd8::144]:2379: connect: connection refused". Reconnecting... .. ....... ........... Eventually when we see the logs for the etc pod, we can tell that the service expect the Internal IPV's of the nodes to be IPV4, although they have IPV6 address: $ crictl logs 3c869c9243b0c I1027 07:57:27.665095 1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2020-10-26T11:40:00Z","message":"EtcdMembersControllerDegraded: failed to get internal IP for node: no matches found for ip family \"tcp4\" for node \"master-0-0\"\nBootstrapTeardownDegraded: failed to get internal IP for node: no matches found for ip family \"tcp4\" for node \"master-0-1\"\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-scripts, configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1,restore-etcd-pod-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nStaticPodsDegraded: pods \"etcd-master-0-2\" not found\nStaticPodsDegraded: pods \"etcd-master-0-1\" not found\nStaticPodsDegraded: pods \"etcd-master-0-0\" not found\nClusterMemberControllerDegraded: failed to get internal IP for node: no matches found for ip family \"tcp4\" for node \"master-0-0\"\nEnvVarControllerDegraded: no matches found for ip family \"tcp4\" for node \"master-0-2\"","reason":"BootstrapTeardown_Error::ClusterMemberController_Error::EnvVarController_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::InstallerController_Error::RevisionController_ContentCreationError::ScriptController_Error::StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-10-26T11:39:28Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-10-26T11:38:41Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2020-10-26T11:39:21Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} I1027 07:57:27.676736 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"openshift-etcd-operator", Name:"openshift-etcd-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod E1027 07:57:27.687958 1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: failed to get internal IP for node: no matches found for ip family "tcp4" for node "master-0-0" E1027 07:57:27.688630 1 base_controller.go:250] "TargetConfigController" controller failed to sync "key", err: missing env var values E1027 07:57:27.688657 1 envvarcontroller.go:197] key failed with : no matches found for ip family "tcp4" for node "master-0-2" E1027 07:57:27.688816 1 base_controller.go:250] "ClusterMemberController" controller failed to sync "key", err: failed to get internal IP for node: no matches found for ip family "tcp4" for node "master-0-0" E1027 07:57:27.688864 1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: failed to get internal IP for node: no matches found for ip family "tcp4" for node "master-0-0" $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0-0 Ready master 20h v1.19.0+e67f5dc fd2e:6f44:5dd8::100 <none> Red Hat Enterprise Linux CoreOS 47.82.202010231442-0 (Ootpa) 4.18.0-193.28.1.el8_2.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e.el8 master-0-1 Ready master 20h v1.19.0+e67f5dc fd2e:6f44:5dd8::142 <none> Red Hat Enterprise Linux CoreOS 47.82.202010231442-0 (Ootpa) 4.18.0-193.28.1.el8_2.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e.el8 master-0-2 Ready master 20h v1.19.0+e67f5dc fd2e:6f44:5dd8::144 <none> Red Hat Enterprise Linux CoreOS 47.82.202010231442-0 (Ootpa) 4.18.0-193.28.1.el8_2.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e.el8 Bootstrap log bundle: ------------------------------------- https://drive.google.com/file/d/1oESIX60agCtpVXDJrlx5ekMLxJ9ucqPk/view?usp=sharing .openshift_install.log: ------------------------------------- attached to the bug. Expected results: ------------------------------------- dual-stack env deployed successfully, when the endpoints (at least) holds IPV6 and IPV4 addresses. Actual results: ------------------------------------- dual-stack deployment failed on bootstrap due to kube-apiserver CrashLoopBackOff How to reproduce: ------------------------------------- 1. Deploy dual-stack env, and config the install-config.yaml file so that the apiVIP holds IPV6 address.
> (BTW, When the apiVIP hold IPV4 address, the deployment succeed but the endpoints and pods > hold only IPV4 addresses). That's half right: your pods _are_ actually getting dual-stack IPs, as seen in the pods.json from the log bundle: "podIP": "10.129.0.15", "podIPs": [ { "ip": "10.129.0.15" }, { "ip": "fd01:0:0:2::f" } ], But to preserve compatibility with scripts that parse oc/kubectl output, `oc get` still shows the single-valued IP columns by default, so `oc get pods` will only show IP addresses of a single family (specifically, the family listed first in your install-config's `clusterNetwork` section, which is considered the default family, at least for pods). However, that only applies to pods: your services/endpoints/nodes really are single stack, because OCP master is still based on kubernetes 1.19, which does not have full dual-stack support. A given Service can be either IPv4 or IPv6, but all of the Services created by OCP at install time will be of the "default" family (the family listed first in the `serviceNetwork` section of the install-config). Nodes OTOH will end up getting an IP of the same family as the `apiVIP`. Thus, given the install-config above, the apiserver service will have an IPv4 Service IP, but IPv6 endpoints, which isn't going to work very well. Likewise, something about the way the etcd configuration is being generated seems to be expecting IPv4 node IPs, but getting IPv6 ones. So: 1. I _think_ that if you change the install-config to list the IPv6 networks first in `clusterNetwork` and `serviceNetwork`, then everything should work. (Can you test that?) 2. We probably need to fix the installer to realize this and give an error for the install-config as you wrote it here
This is the new install-config.yaml with the changes suggested (IPv6 networks first in `clusterNetwork` and `serviceNetwork`): [kni@provisionhost-0-0 ~]$ cat install-config.yaml apiVersion: v1 baseDomain: qe.lab.redhat.com networking: networkType: OVNKubernetes machineNetwork: - cidr: 192.168.123.0/24 - cidr: fd2e:6f44:5dd8::/64 clusterNetwork: - cidr: fd01::/48 hostPrefix: 64 - cidr: 10.128.0.0/14 hostPrefix: 23 serviceNetwork: - fd02::/112 - 172.30.0.0/16 metadata: name: ocp-edge-cluster-0 compute: - name: worker replicas: 2 controlPlane: name: master replicas: 3 platform: baremetal: {} platform: baremetal: provisioningNetwork: Managed externalBridge: baremetal-0 provisioningBridge: provisioning-0 libvirtURI: qemu+ssh://root.qe.lab.redhat.com/system provisioningNetworkInterface: enp4s0 provisioningNetworkCIDR: fd00:1101::/64 bootstrapOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-qemu.x86_64.qcow2.gz?sha256=fb27322b0269b2732021f7b2468180a7d9ebb06f598ca469513dc12960e26608 clusterOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-47.82.202010211043-0-openstack.x86_64.qcow2.gz?sha256=6904c9047c26a3d37fbea97c5b03dba22c0ec1cbd8221c2857c1719ba4670365 apiVIP: fd2e:6f44:5dd8::5 dnsVIP: fd2e:6f44:5dd8:0:0:0:0:2 ingressVIP: fd2e:6f44:5dd8::10 hosts: - name: openshift-master-0-0 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/5262aff3-2fac-4758-9c66-ed1653bedfef disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:75:00:90 rootDeviceHints: deviceName: /dev/sda - name: openshift-master-0-1 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/9738b08f-07eb-43af-8ad8-98ff6e373229 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:48:5a:4a rootDeviceHints: deviceName: /dev/sda - name: openshift-master-0-2 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/8da913c4-069c-4e72-8309-c5fdc44e7794 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:05:e6:38 rootDeviceHints: deviceName: /dev/sda - name: openshift-worker-0-0 role: worker bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/ed948fe6-aeee-4300-abbc-30d809cabd7a disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:d2:33:af rootDeviceHints: deviceName: /dev/sda - name: openshift-worker-0-1 role: worker bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/8e041ccc-e2bd-4582-be6c-771ee009f83d disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:fa:aa:9f rootDeviceHints: deviceName: /dev/sda additionalTrustBundle: | -----BEGIN CERTIFICATE----- MIIEVzCCAz+gAwIBAgIUfeyfz6pPL4HU4brQaMW0zgQR29owDQYJKoZIhvcNAQEL BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD VQQDDChzZWFsdXNhMjkubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X DTIwMTAyODA3MzYwM1oXDTIxMTAyODA3MzYwM1owgYgxCzAJBgNVBAYTAlVTMQsw CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMjkubW9iaXVz LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAp/wq/bJwil/4dU1AHCrhBlLMUurPQRazzFXqjbK4Ksxn3k39tIh6 Gjdqf+Ao/IwAeInkO7jTzfy5MN5sHCKlomIlcDbvNbK/G5m0KMjEp74IV7p8g4Vr VI7s1Gcv5lx947kRmWx2Mm+EVZmGQKVpn4rYTUq+6CT4BGYohMo6bAxhcO7q1mK7 +Z5l9w/YpXw+xyDubup0Ji7evzyq1Y1f8bhx5yneek57kFuNAYo6Z7pedu373jlZ ezr/Yhocwka5N6JIRov++4lnUlZ/A1xX8vIAaHDsP+STG3clYMVGHY1OPo5GkxoR zbM2ONYq+IzF2xtzzDHbpIwS3huaqa+tGwIDAQABo4G2MIGzMA8GA1UdEwQIMAYB Af8CAQAwgZ8GA1UdEQSBlzCBlIItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMjmHEP6AAAAAAAAA/FQA//75E46H EP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//500CCHECYgAFIAAAlMJm6W //7Rj8qHEP6AAAAAAAAAJm6W//7Rj8owDQYJKoZIhvcNAQELBQADggEBAAgdo+KI OJvTZrVkROXIL+/DpPeS0/09ioiTrsGUL7Na447ebeCXK8HJMbZ9a7G7oiOjt9V+ jN7GZtAIiMmarnkmK4qhT/1uNHd2R8JobZcbsEd2tTM6EF9JBDe9yY0QXfN0tcr3 TK/U9y5m9KUzU82WE8NJsRoGKwKjnd//yhzMihO0e1TQ5nB5m6vIMVes4RETc3hd hgk/2SOJcsyS9+FEmheMGu28LM4ekjKZZhm/VBnu1LbUC8OFRVuViNesjl5HsKQS XPCTLdd/mlIoVdUp/tBcU3xrsFE7cJg9h5mwmWUqqVst55VEqiYceNFR9m0xg4qY B1ee7jZD6NtfIkI= -----END CERTIFICATE----- pullSecret: | { "auths": { "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000": { "auth": "b2NwLWVkZ2U6b2NwLWVkZ2UtcGFzcw==" } }} fips: false sshKey: | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr8zXc3x+geVx0aKDsamBhl1mBOTbEMO3jcG6iP0KEq93x5DyyKOiPUfVHruxATkzpZlKcsuqRu+GMcFntQy7lYiMtPMQZ1kf/2J/1lQ1Y/x2CKgsKUK2oGAQrWEBpAGqomeq9c5o3hWuNFkWHQ/9z438l/Ob3TahTjyK81Gl+e7FBdgPAhEDS1fjZlSM/HSTtQUMXR58eJIR5+PVS6HtuTwKjYJFlxgLv7Fx61yRmCXfDXvDuDbYM965IrTlFLwp8VSt8XBPdAGaEMnrAqpIYswW4Ot2n1k9mmArkdEln2WCHka/oMjWl1Dg+5eN3jsAfiFOsi/CYkFglTeIAb2VdKJygkLwxWed8a3vyzLyg3mcCOXxjh5WuJUUMpv8K/I/9LRaERzSVWiH5pG0Bdp6q+UmgFSMsaCGW6r6yMM5HKrASR8Q2K2vR64CeeFZ0utQ4vQdk1oiJS788FDGLyFgs1rfdtFoq092WOhr2pKSDe89bU8+txkh6R57GAHS5qhc= kni.qe.lab.redhat.com imageContentSources: - mirrors: - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image source: registry.svc.ci.openshift.org/ocp/release And yet, the deployment failed on the bootstrap with: FATAL Bootstrap failed to complete: failed to wait for bootstrapping to complete: timed out waiting for the condition [kni@provisionhost-0-0 ~]$ oc get nodes -A -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0-0 NotReady master 29m v1.19.0+e67f5dc fd2e:6f44:5dd8::11a <none> Red Hat Enterprise Linux CoreOS 47.82.202010270142-0 (Ootpa) 4.18.0-193.28.1.el8_2.x86_64 cri-o://1.20.0-0.rhaos4.7.gitec53db4.el8-dev master-0-1 NotReady master 29m v1.19.0+e67f5dc fd2e:6f44:5dd8::109 <none> Red Hat Enterprise Linux CoreOS 47.82.202010270142-0 (Ootpa) 4.18.0-193.28.1.el8_2.x86_64 cri-o://1.20.0-0.rhaos4.7.gitec53db4.el8-dev master-0-2 NotReady master 29m v1.19.0+e67f5dc fd2e:6f44:5dd8::105 <none> Red Hat Enterprise Linux CoreOS 47.82.202010270142-0 (Ootpa) 4.18.0-193.28.1.el8_2.x86_64 cri-o://1.20.0-0.rhaos4.7.gitec53db4.el8-dev [kni@provisionhost-0-0 ~]$ oc get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES openshift-apiserver-operator openshift-apiserver-operator-5d69cbbdf-pmxjs 0/1 Pending 0 43m <none> <none> <none> <none> openshift-authentication-operator authentication-operator-7954964f8d-fvdvf 0/1 Pending 0 43m <none> <none> <none> <none> openshift-cloud-credential-operator cloud-credential-operator-7fb659d948-ltxpf 0/2 Pending 0 43m <none> <none> <none> <none> openshift-cluster-machine-approver machine-approver-7b45cc85f-lp4z9 0/2 Pending 0 43m <none> <none> <none> <none> openshift-cluster-node-tuning-operator cluster-node-tuning-operator-7c5dd85979-nhcch 0/1 Pending 0 43m <none> <none> <none> <none> openshift-cluster-storage-operator cluster-storage-operator-598dbcb96-k8hqj 0/1 Pending 0 43m <none> <none> <none> <none> openshift-cluster-storage-operator csi-snapshot-controller-operator-789c9877f-7mcq8 0/1 Pending 0 43m <none> <none> <none> <none> openshift-cluster-version cluster-version-operator-868f7c8bcb-smds6 0/1 Pending 0 43m <none> <none> <none> <none> openshift-config-operator openshift-config-operator-84dbd98479-ghmtv 0/1 Pending 0 43m <none> <none> <none> <none> openshift-controller-manager-operator openshift-controller-manager-operator-bf7b9697c-ppm2m 0/1 Pending 0 43m <none> <none> <none> <none> openshift-dns-operator dns-operator-574b845588-jtw7z 0/2 Pending 0 43m <none> <none> <none> <none> openshift-etcd-operator etcd-operator-7f86f6cc49-ss8fl 0/1 Pending 0 43m <none> <none> <none> <none> openshift-image-registry cluster-image-registry-operator-b8688d866-vstm4 0/1 Pending 0 43m <none> <none> <none> <none> openshift-ingress-operator ingress-operator-74c8747fd4-7czxr 0/2 Pending 0 43m <none> <none> <none> <none> openshift-insights insights-operator-5d6cd95c7d-6c98c 0/1 Pending 0 43m <none> <none> <none> <none> openshift-kni-infra coredns-master-0-0 2/2 Running 0 29m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-kni-infra coredns-master-0-1 2/2 Running 0 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-kni-infra coredns-master-0-2 2/2 Running 0 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-kni-infra haproxy-master-0-0 2/2 Running 0 29m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-kni-infra haproxy-master-0-1 2/2 Running 0 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-kni-infra haproxy-master-0-2 2/2 Running 0 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-kni-infra keepalived-master-0-0 1/2 Running 4 28m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-kni-infra keepalived-master-0-1 1/2 Running 4 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-kni-infra keepalived-master-0-2 1/2 Running 2 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-kni-infra mdns-publisher-master-0-0 1/1 Running 0 29m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-kni-infra mdns-publisher-master-0-1 1/1 Running 0 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-kni-infra mdns-publisher-master-0-2 1/1 Running 0 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-kube-apiserver-operator kube-apiserver-operator-dc58dbf6d-8pcx8 0/1 Pending 0 43m <none> <none> <none> <none> openshift-kube-controller-manager-operator kube-controller-manager-operator-785cf947f8-f79w7 0/1 Pending 0 43m <none> <none> <none> <none> openshift-kube-scheduler-operator openshift-kube-scheduler-operator-66dcfb7d98-mv8pn 0/1 Pending 0 43m <none> <none> <none> <none> openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-5758648f54-nwspv 0/1 Pending 0 43m <none> <none> <none> <none> openshift-machine-api cluster-autoscaler-operator-7759576c89-n64hj 0/2 Pending 0 43m <none> <none> <none> <none> openshift-machine-api machine-api-operator-5b547676ff-7lpht 0/2 Pending 0 43m <none> <none> <none> <none> openshift-machine-config-operator machine-config-operator-75c4bb4ccf-s9bkw 0/1 Pending 0 43m <none> <none> <none> <none> openshift-marketplace marketplace-operator-6bf7ddc8db-ww2wn 0/1 Pending 0 43m <none> <none> <none> <none> openshift-monitoring cluster-monitoring-operator-d747df89b-mvdgl 0/2 Pending 0 43m <none> <none> <none> <none> openshift-multus multus-flfph 1/1 Running 2 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-multus multus-rkfjs 1/1 Running 2 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-multus multus-xdh7p 1/1 Running 2 28m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-multus network-metrics-daemon-gj9hz 0/2 ContainerCreating 0 28m <none> master-0-2 <none> <none> openshift-multus network-metrics-daemon-hzmzc 0/2 ContainerCreating 0 28m <none> master-0-1 <none> <none> openshift-multus network-metrics-daemon-lk6jc 0/2 ContainerCreating 0 28m <none> master-0-0 <none> <none> openshift-network-operator network-operator-8689749bb8-bmdj7 1/1 Running 0 9m5s fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-operator-lifecycle-manager catalog-operator-5566c989c9-gw8xl 0/1 Pending 0 43m <none> <none> <none> <none> openshift-operator-lifecycle-manager olm-operator-75d46f844c-dffrg 0/1 Pending 0 43m <none> <none> <none> <none> openshift-ovn-kubernetes ovnkube-master-6nr7w 4/6 CrashLoopBackOff 20 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-ovn-kubernetes ovnkube-master-dchdt 4/6 CrashLoopBackOff 20 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-ovn-kubernetes ovnkube-master-thjqh 4/6 CrashLoopBackOff 20 28m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-ovn-kubernetes ovnkube-node-4wpml 2/3 Running 6 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-ovn-kubernetes ovnkube-node-l96ng 2/3 CrashLoopBackOff 6 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-ovn-kubernetes ovnkube-node-lbcvg 2/3 Running 6 28m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-ovn-kubernetes ovs-node-8hrwj 1/1 Running 0 28m fd2e:6f44:5dd8::109 master-0-1 <none> <none> openshift-ovn-kubernetes ovs-node-fk72k 1/1 Running 0 28m fd2e:6f44:5dd8::105 master-0-2 <none> <none> openshift-ovn-kubernetes ovs-node-p5rss 1/1 Running 0 28m fd2e:6f44:5dd8::11a master-0-0 <none> <none> openshift-service-ca-operator service-ca-operator-797dff8569-8bsw7 0/1 Pending 0 43m <none> <none> <none> <none> [kni@provisionhost-0-0 ~]$ oc get endpoints -A -o wide NAMESPACE NAME ENDPOINTS AGE default kubernetes 192.168.123.124:6443 44m kube-system kube-scheduler <none> 45m openshift-apiserver-operator metrics <none> 44m openshift-authentication-operator metrics <none> 44m openshift-cloud-credential-operator cco-metrics <none> 44m openshift-cluster-machine-approver machine-approver <none> 44m openshift-cluster-storage-operator csi-snapshot-controller-operator-metrics <none> 44m openshift-cluster-version cluster-version-operator <none> 44m openshift-config-operator metrics <none> 44m openshift-controller-manager-operator metrics <none> 44m openshift-dns-operator metrics <none> 44m openshift-etcd-operator metrics <none> 44m openshift-etcd etcd <none> 44m openshift-etcd host-etcd-2 <none> 44m openshift-image-registry image-registry-operator <none> 43m openshift-ingress-operator metrics <none> 44m openshift-insights metrics <none> 43m openshift-kube-apiserver-operator metrics <none> 43m openshift-kube-controller-manager-operator metrics <none> 44m openshift-kube-scheduler-operator metrics <none> 44m openshift-kube-storage-version-migrator-operator metrics <none> 44m openshift-machine-api cluster-autoscaler-operator <none> 44m openshift-machine-api machine-api-controllers <none> 43m openshift-machine-api machine-api-operator <none> 43m openshift-machine-api machine-api-operator-webhook <none> 43m openshift-machine-config-operator machine-config-daemon <none> 44m openshift-marketplace marketplace-operator-metrics <none> 44m openshift-multus multus-admission-controller <none> 29m openshift-multus network-metrics-service <none> 29m openshift-operator-lifecycle-manager catalog-operator-metrics <none> 44m openshift-operator-lifecycle-manager olm-operator-metrics <none> 44m openshift-ovn-kubernetes ovn-kubernetes-master [fd2e:6f44:5dd8::105]:9102,[fd2e:6f44:5dd8::109]:9102,[fd2e:6f44:5dd8::11a]:9102 29m openshift-ovn-kubernetes ovn-kubernetes-node [fd2e:6f44:5dd8::105]:9103,[fd2e:6f44:5dd8::109]:9103,[fd2e:6f44:5dd8::11a]:9103 29m openshift-ovn-kubernetes ovnkube-db 29m openshift-service-ca-operator metrics <none> 44m [kni@provisionhost-0-0 ~]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication cloud-credential True False False 43m cluster-autoscaler config-operator console csi-snapshot-controller dns etcd image-registry ingress insights kube-apiserver kube-controller-manager kube-scheduler kube-storage-version-migrator machine-api machine-approver machine-config marketplace monitoring network False True True 30m node-tuning openshift-apiserver openshift-controller-manager openshift-samples operator-lifecycle-manager operator-lifecycle-manager-catalog operator-lifecycle-manager-packageserver service-ca storage Bootstrap log bundle: ------------------------------------- https://drive.google.com/file/d/19Tk24d5CHu81rixAS_-F9E-vN0COCuEl/view?usp=sharing openshift_install.log: ------------------------------------- https://drive.google.com/file/d/1riNKTEhwuDvXRv1th7szQLAyBKwzQKL9/view?usp=sharing
Merged the upstream fix for the specific bug that broke your last test run: https://github.com/ovn-org/ovn-kubernetes/pull/1787 Not sure exactly when that will make it into OCP, or if that's the only remaining bug with the dual-stack-IPv6-primary case. It seems that dev-scripts won't let you create a dual-stack-IPv6-primary cluster so I can't easily test this myself. (If you want to try testing it right away, you could ask @cluster-bot on Slack to "build openshift/ovn-kubernetes#332" and then try installing the resulting image)
We are going to wait for Kube 1.20 before continue testing dual stack
(In reply to Shelly Miron from comment #6) > We are going to wait for Kube 1.20 before continue testing dual stack ok, closing this now then