Bug 1888862
| Summary: | [4.6] need to pass dual-stack service CIDRs to kube-apiserver in dual-stack cluster | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Dan Winship <danw> |
| Component: | kube-apiserver | Assignee: | Dan Winship <danw> |
| Status: | CLOSED ERRATA | QA Contact: | Shelly Miron <smiron> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.6 | CC: | aos-bugs, kewang, mfojtik, prabinov, rbartal, xxia |
| Target Milestone: | --- | ||
| Target Release: | 4.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1888861 | Environment: | |
| Last Closed: | 2020-12-14 13:50:18 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1888861 | ||
| Bug Blocks: | |||
|
Description
Dan Winship
2020-10-16 00:54:49 UTC
> (Not a 4.6.0 blocker because we don't officially support dual stack yet.)
Hi danw, if we don't officially support dual stack yet, how do we verify this bug?
"don't officially support" just in the sense of "it's not supported for customers", so it's ok that this missed 4.6.0 and will land in 4.6.z. The code is all there and it works and can be tested (though we only support dual stack on bare metal right now so maybe you'd have to hand the bug over to KNI QE). Hi danw, thanks for your explanation, I am doing pre-merge verification for this bug with process https://issues.redhat.com/browse/DPTP-660, since no dual-stack cluster available in hand, I couldn't go on, will try the bug after ON_QA or hand the bug over to KNI QE as you said. bumping severity to "high"; although no customers will be using this in production in 4.6, some important customers need to enable the feature for early testing of 4.7 functionality Hey all, We in KNI Deployment QE will verify this BZ using the latest 4.6.z version, And will update the test Raviv Cluster condition: --------------------- [kni@provisionhost-0-0 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.3 True False 96m Cluster version is 4.6.3 [kni@provisionhost-0-0 ~]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.6.3 True False False 23m cloud-credential 4.6.3 True False False 149m cluster-autoscaler 4.6.3 True False False 134m config-operator 4.6.3 True False False 136m console 4.6.3 True False False 104m csi-snapshot-controller 4.6.3 True False False 135m dns 4.6.3 True False False 134m etcd 4.6.3 True False False 134m image-registry 4.6.3 True False False 115m ingress 4.6.3 True False False 109m insights 4.6.3 True False False 136m kube-apiserver 4.6.3 True False False 134m kube-controller-manager 4.6.3 True False False 134m kube-scheduler 4.6.3 True False False 134m kube-storage-version-migrator 4.6.3 True False False 109m machine-api 4.6.3 True False False 120m machine-approver 4.6.3 True False False 135m machine-config 4.6.3 True False False 134m marketplace 4.6.3 True False False 135m monitoring 4.6.3 True False False 109m network 4.6.3 True False False 136m node-tuning 4.6.3 True False False 136m openshift-apiserver 4.6.3 True False False 115m openshift-controller-manager 4.6.3 True False False 135m openshift-samples 4.6.3 True False False 115m operator-lifecycle-manager 4.6.3 True False False 135m operator-lifecycle-manager-catalog 4.6.3 True False False 135m operator-lifecycle-manager-packageserver 4.6.3 True False False 114m service-ca 4.6.3 True False False 136m storage 4.6.3 True False False 136m This is the install-config.yaml file i've worked with (since deployment of apiVIP, dnsVIP and ingressVIP with IPV6 didnt work last time due to kubernetes 1.20 version - see this bug https://bugzilla.redhat.com/show_bug.cgi?id=1891771): [kni@provisionhost-0-0 ~]$ cat install-config.yaml apiVersion: v1 baseDomain: qe.lab.redhat.com networking: networkType: OVNKubernetes machineNetwork: - cidr: 192.168.123.0/24 - cidr: fd2e:6f44:5dd8::/64 clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 - cidr: fd01::/48 hostPrefix: 64 serviceNetwork: - 172.30.0.0/16 - fd02::/112 metadata: name: ocp-edge-cluster-0 compute: - name: worker replicas: 2 controlPlane: name: master replicas: 3 platform: baremetal: {} platform: baremetal: provisioningNetwork: Managed externalBridge: baremetal-0 provisioningBridge: provisioning-0 libvirtURI: qemu+ssh://root.qe.lab.redhat.com/system provisioningNetworkInterface: enp4s0 provisioningNetworkCIDR: fd00:1101::/64 bootstrapOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-46.82.202010011740-0-qemu.x86_64.qcow2.gz?sha256=cbaf2e5548af2d1d1153d9a1beca118042e770725166de427792c33a875137cc clusterOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-46.82.202010011740-0-openstack.x86_64.qcow2.gz?sha256=95a5c2dafca2dd1a59e5e99dc1b3809c9b925cd94e37acf52e29f7e694ce30ab apiVIP: 192.168.123.5 dnsVIP: 192.168.123.6 ingressVIP: 192.168.123.10 hosts: - name: openshift-master-0-0 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/6f5bb411-e1ab-4596-b40e-a7493943ed96 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:7d:2e:d1 rootDeviceHints: deviceName: /dev/sda - name: openshift-master-0-1 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/56ef77b4-0b3a-4284-b30b-4caf099bbe5e disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:91:b6:9d rootDeviceHints: deviceName: /dev/sda - name: openshift-master-0-2 role: master bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/13c8f53b-d1aa-445c-b7f1-860f2683c810 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:8d:ba:fe rootDeviceHints: deviceName: /dev/sda - name: openshift-worker-0-0 role: worker bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/8cf35875-54b1-4339-aa60-88baeae67689 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:18:df:0d rootDeviceHints: deviceName: /dev/sda - name: openshift-worker-0-1 role: worker bmc: address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/97364ce5-e353-422c-b827-901c44fe11a1 disableCertificateVerification: True username: admin password: password bootMACAddress: 52:54:00:f4:32:60 rootDeviceHints: deviceName: /dev/sda additionalTrustBundle: | -----BEGIN CERTIFICATE----- MIIEOjCCAyKgAwIBAgIUKeG1PLdj4LZdToTP6wQ5YUzWVYcwDQYJKoZIhvcNAQEL BQAwfDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo MRUwEwYDVQQKDAxUZXN0IENvbXBhbnkxEDAOBgNVBAsMB1Rlc3RpbmcxJTAjBgNV BAMMHHNlYWwzNi5xYS5sYWIudGx2LnJlZGhhdC5jb20wHhcNMjAxMTE3MDg0OTI1 WhcNMjExMTE3MDg0OTI1WjB8MQswCQYDVQQGEwJVUzELMAkGA1UECAwCTkMxEDAO BgNVBAcMB1JhbGVpZ2gxFTATBgNVBAoMDFRlc3QgQ29tcGFueTEQMA4GA1UECwwH VGVzdGluZzElMCMGA1UEAwwcc2VhbDM2LnFhLmxhYi50bHYucmVkaGF0LmNvbTCC ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALP5HQKIaozo/6XuTssktlq9 wfLgmPJXAdDNgD49VkMHlBmzmDZsYn+7MEJEP19YsH/Nqb+pLC8r7tpO7h0SxL80 WjUbNucLR0N3wIImPI1eW2UOabIlQ4X70UigEmaYbxM27EOSsqbonmOvOVzasqgj jjd+xIM4lqrsRT3KmiwI2gERyKXESJrD5py+zpc4qR05sgdQ3i4VNPnORNWg6/fs ZQOYYM9VzA4cy6EkaE9hHInzX5ALK8JHpnpb4RFIrg9IqkxzEIvTfz9efDivhtEt +SbOKipelYZScMJYS0q0ReMZCxCzFI/zcyc1UAwhhXsCUNHUGB5JTSx1muaEhHsC AwEAAaOBszCBsDAPBgNVHRMECDAGAQH/AgEAMIGcBgNVHREEgZQwgZGCLXJlZ2lz dHJ5Lm9jcC1lZGdlLWNsdXN0ZXItMC5xZS5sYWIucmVkaGF0LmNvbYIGc2VhbDM2 hxD9Lm9EXdgAAAAAAAAAAAABhxD+gAAAAAAAAFBUAP/+PRHWhxD+gAAAAAAAAPxU AP/+mMr6hxAmIABSAAAjQA7Eev/+ZqN4hxD+gAAAAAAAAA7Eev/+ZqN4MA0GCSqG SIb3DQEBCwUAA4IBAQB+XzmMNo/KkjaIX0JdVB5QZgdswSx8i7eQskYCu7QfR4Iy qcZLQ3Nst0U3I8I4TQhfN2bZKDfK6PSpySSujgliMqxfSYOPrdekBkSBQbAf5KUK 5RvFUtwQKdR8M0BAB9Q9Tvyv8nhKc9Pl3uLb3RCIfcKq9CVj+SnhwNEXBeBtTh2Y H76qhHG9CaWrbsel9fm8Il2DjrwAleRUlV2d+ypaaJcelKE9nlLkOfzxSvHrk8MJ da9qrOd4kqM0iv0mF4N3JH6NEFujiCq7zz7quV/h4F7pSIKglmsjXzq7Izl9dnOB h7FRir3pzOfR2HfV2ksUU1gCx65w22PPs0fkVut2 -----END CERTIFICATE----- pullSecret: | { "auths": { "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000": { "auth": "b2NwLWVkZ2U6b2NwLWVkZ2UtcGFzcw==" } }} fips: false sshKey: | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC76dNlV34b/Q6d9f6zFG9mxL9/S41drT6jFEbVTNG9LGREH8Co2ezBomPdGj8mb3lZ4vuG+r7l0c4/VL5ug2ejoVsRV7R/o4HoQxQtNq75ukIApZV2flPsSImJf9N+2osrLZ+EQSp8aTr6uKA/4PqkkC8JvuN9wwMQK3ey7jdj7rwqQDHLsvj4OGSo8YKxjjFlu6Dfa4/1dP4hhdV00gO9bIzlIUsBPmvUW6vKzgzphrcdN9L+iS+dHVju5/oWoFEEUfilE183nz4cERXhcDtz5UdyC62enKrPabkyB0r5pV0WnGtUIr9/6B7iXVspF1j9HgSdh++Nw6tT5sEN4tuQLzna4WMDw/uG9o+gXwDX9Ozp8rL0qWy8GdaZGHqdPK9aNvSM+018ZRUO2eE8kt/Ntj7MqcUPSUyZwLSP7UyDfmAS3qgLhzIc4Hjc/3i7v776DhvoFWYCE9Y0Woj/EOAfiDJgiPCxWFKrNcB6X4/EzvTOoxoEmR9u91+0DvLRImk= kni.qe.lab.redhat.com imageContentSources: - mirrors: - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image source: quay.io/openshift-release-dev/ocp-release - mirrors: - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image source: quay.io/openshift-release-dev/ocp-v4.0-art-dev I tried to verify it as indicated in the cloned bug (https://bugzilla.redhat.com/show_bug.cgi?id=1888861) and i don't see indication for IPV6 address here, only IPV4: $ oc rsh -n openshift-kube-apiserver kube-apiserver-master-0-0 $ cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml .... ........... ................ resources/configmaps/bound-sa-token-signing-certs"],"servicesSubnet":"172.30.0.0/16","servingInfo":{"bindAddress":"0.0.0.0:6443","bindNetwork":"tcp4","cipherSuites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA ................ ............ .... but i might be wrong testing it, so would appreciate for additional info. must-gather: ------------- Unfortunately i could not add a must-gather due to this error, but let me know if there is any additional info needed: [kni@provisionhost-0-0 ~]$ oc adm must-gather [must-gather ] OUT unable to resolve the imagestream tag openshift/must-gather:latest [must-gather ] OUT [must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest [must-gather ] OUT namespace/openshift-must-gather-w5blx created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-vxp5z created [must-gather ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created [must-gather-mbq8z] OUT gather did not start: unable to pull image: ImagePullBackOff: Back-off pulling image "quay.io/openshift/origin-must-gather:latest" [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-vxp5z deleted [must-gather ] OUT namespace/openshift-must-gather-w5blx deleted error: gather did not start for pod must-gather-mbq8z: unable to pull image: ImagePullBackOff: Back-off pulling image "quay.io/openshift/origin-must-gather:latest" > version 4.6.3 True False 96m Cluster version is 4.6.3
The fix is in 4.6.4 and later.
Verified on OCP4.6.5, using the steps described in comment 10: $ oc rsh -n openshift-kube-apiserver kube-apiserver-master-0-0.ocp-edge-cluster-0.qe.lab.redhat.com $ cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml ...... ... .. servicesSubnet":"172.30.0.0/16,fd02::/112", "servingInfo":{"bindAddress":"0.0.0.0:6443","bindNetwork":"tcp4","cipherSuites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"] .. ... ..... both IPV4, IPV6 addresses deployed for the kube-apiserver-master-0-0 pod Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.6.8 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5259 |