Bug 1888862 - [4.6] need to pass dual-stack service CIDRs to kube-apiserver in dual-stack cluster
Summary: [4.6] need to pass dual-stack service CIDRs to kube-apiserver in dual-stack c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.6.z
Assignee: Dan Winship
QA Contact: Shelly Miron
URL:
Whiteboard:
Depends On: 1888861
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-16 00:54 UTC by Dan Winship
Modified: 2020-12-14 13:50 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1888861
Environment:
Last Closed: 2020-12-14 13:50:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 985 0 None closed Bug 1888862: [4.6] set servicesSubnet correctly in dual-stack clusters 2021-02-14 19:04:50 UTC
Red Hat Product Errata RHSA-2020:5259 0 None None None 2020-12-14 13:50:44 UTC

Description Dan Winship 2020-10-16 00:54:49 UTC
+++ This bug was initially created as a clone of Bug #1888861 +++

Description of problem:
We had not yet updated cluster-kube-apiserver-operator to pass both service CIDRs to kube-apiserver when the cluster was dual-stack

Steps to Reproduce:
1. Deploy dual-stack cluster
2. Try to create a service with `ipFamily: IPv6`

Actual results:
error

Expected results:
an IPv6 service

(Not a 4.6.0 blocker because we don't officially support dual stack yet.)

Comment 1 Ke Wang 2020-10-28 08:41:37 UTC
> (Not a 4.6.0 blocker because we don't officially support dual stack yet.)

Hi danw@redhat.com, if we don't officially support dual stack yet, how do we verify this bug?

Comment 2 Dan Winship 2020-10-28 14:02:34 UTC
"don't officially support" just in the sense of "it's not supported for customers", so it's ok that this missed 4.6.0 and will land in 4.6.z. The code is all there and it works and can be tested (though we only support dual stack on bare metal right now so maybe you'd have to hand the bug over to KNI QE).

Comment 3 Ke Wang 2020-10-29 03:52:19 UTC
Hi danw@redhat.com, thanks for your explanation, I am doing pre-merge verification for this bug with process https://issues.redhat.com/browse/DPTP-660, since no dual-stack cluster available in hand, I couldn't go on, will try the bug after ON_QA or hand the bug over to KNI QE as you said.

Comment 4 Dan Winship 2020-11-03 19:40:35 UTC
bumping severity to "high"; although no customers will be using this in production in 4.6, some important customers need to enable the feature for early testing of 4.7 functionality

Comment 7 Raviv Bar-Tal 2020-11-12 13:14:28 UTC
Hey all, 
We in KNI Deployment QE will verify this BZ using the latest 4.6.z version,
And will update the test
Raviv

Comment 10 Shelly Miron 2020-11-17 11:46:44 UTC
Cluster condition:
---------------------

[kni@provisionhost-0-0 ~]$ oc get clusterversion

NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.3     True        False         96m     Cluster version is 4.6.3

[kni@provisionhost-0-0 ~]$ oc get co

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.3     True        False         False      23m
cloud-credential                           4.6.3     True        False         False      149m
cluster-autoscaler                         4.6.3     True        False         False      134m
config-operator                            4.6.3     True        False         False      136m
console                                    4.6.3     True        False         False      104m
csi-snapshot-controller                    4.6.3     True        False         False      135m
dns                                        4.6.3     True        False         False      134m
etcd                                       4.6.3     True        False         False      134m
image-registry                             4.6.3     True        False         False      115m
ingress                                    4.6.3     True        False         False      109m
insights                                   4.6.3     True        False         False      136m
kube-apiserver                             4.6.3     True        False         False      134m
kube-controller-manager                    4.6.3     True        False         False      134m
kube-scheduler                             4.6.3     True        False         False      134m
kube-storage-version-migrator              4.6.3     True        False         False      109m
machine-api                                4.6.3     True        False         False      120m
machine-approver                           4.6.3     True        False         False      135m
machine-config                             4.6.3     True        False         False      134m
marketplace                                4.6.3     True        False         False      135m
monitoring                                 4.6.3     True        False         False      109m
network                                    4.6.3     True        False         False      136m
node-tuning                                4.6.3     True        False         False      136m
openshift-apiserver                        4.6.3     True        False         False      115m
openshift-controller-manager               4.6.3     True        False         False      135m
openshift-samples                          4.6.3     True        False         False      115m
operator-lifecycle-manager                 4.6.3     True        False         False      135m
operator-lifecycle-manager-catalog         4.6.3     True        False         False      135m
operator-lifecycle-manager-packageserver   4.6.3     True        False         False      114m
service-ca                                 4.6.3     True        False         False      136m
storage                                    4.6.3     True        False         False      136m



This is the install-config.yaml file i've worked with (since deployment of apiVIP, dnsVIP and ingressVIP 
with IPV6 didnt work last time due to kubernetes 1.20 version - see this bug https://bugzilla.redhat.com/show_bug.cgi?id=1891771):


[kni@provisionhost-0-0 ~]$ cat install-config.yaml 
apiVersion: v1
baseDomain: qe.lab.redhat.com
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 192.168.123.0/24
  - cidr: fd2e:6f44:5dd8::/64
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  - cidr: fd01::/48
    hostPrefix: 64
  serviceNetwork:
  - 172.30.0.0/16
  - fd02::/112
metadata:
  name: ocp-edge-cluster-0
compute:
- name: worker
  replicas: 2
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    provisioningNetwork: Managed
    externalBridge: baremetal-0 
    provisioningBridge: provisioning-0 
    libvirtURI: qemu+ssh://root@hypervisor.ocp-edge-cluster-0.qe.lab.redhat.com/system
    provisioningNetworkInterface: enp4s0
    provisioningNetworkCIDR: fd00:1101::/64
    bootstrapOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-46.82.202010011740-0-qemu.x86_64.qcow2.gz?sha256=cbaf2e5548af2d1d1153d9a1beca118042e770725166de427792c33a875137cc
    clusterOSImage: http://registry.ocp-edge-cluster-0.qe.lab.redhat.com:8080/images/rhcos-46.82.202010011740-0-openstack.x86_64.qcow2.gz?sha256=95a5c2dafca2dd1a59e5e99dc1b3809c9b925cd94e37acf52e29f7e694ce30ab
    apiVIP: 192.168.123.5
    dnsVIP: 192.168.123.6
    ingressVIP: 192.168.123.10
    hosts:
      - name: openshift-master-0-0
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/6f5bb411-e1ab-4596-b40e-a7493943ed96
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:7d:2e:d1
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-master-0-1
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/56ef77b4-0b3a-4284-b30b-4caf099bbe5e
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:91:b6:9d
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-master-0-2
        role: master
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/13c8f53b-d1aa-445c-b7f1-860f2683c810
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:8d:ba:fe
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-worker-0-0
        role: worker
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/8cf35875-54b1-4339-aa60-88baeae67689
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:18:df:0d
        rootDeviceHints:
          deviceName: /dev/sda
      - name: openshift-worker-0-1
        role: worker
        bmc:
          address: redfish://[fd2e:6f44:5dd8::1]:8000/redfish/v1/Systems/97364ce5-e353-422c-b827-901c44fe11a1
          disableCertificateVerification: True
          username: admin
          password: password
        bootMACAddress: 52:54:00:f4:32:60
        rootDeviceHints:
          deviceName: /dev/sda
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIEOjCCAyKgAwIBAgIUKeG1PLdj4LZdToTP6wQ5YUzWVYcwDQYJKoZIhvcNAQEL
  BQAwfDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo
  MRUwEwYDVQQKDAxUZXN0IENvbXBhbnkxEDAOBgNVBAsMB1Rlc3RpbmcxJTAjBgNV
  BAMMHHNlYWwzNi5xYS5sYWIudGx2LnJlZGhhdC5jb20wHhcNMjAxMTE3MDg0OTI1
  WhcNMjExMTE3MDg0OTI1WjB8MQswCQYDVQQGEwJVUzELMAkGA1UECAwCTkMxEDAO
  BgNVBAcMB1JhbGVpZ2gxFTATBgNVBAoMDFRlc3QgQ29tcGFueTEQMA4GA1UECwwH
  VGVzdGluZzElMCMGA1UEAwwcc2VhbDM2LnFhLmxhYi50bHYucmVkaGF0LmNvbTCC
  ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALP5HQKIaozo/6XuTssktlq9
  wfLgmPJXAdDNgD49VkMHlBmzmDZsYn+7MEJEP19YsH/Nqb+pLC8r7tpO7h0SxL80
  WjUbNucLR0N3wIImPI1eW2UOabIlQ4X70UigEmaYbxM27EOSsqbonmOvOVzasqgj
  jjd+xIM4lqrsRT3KmiwI2gERyKXESJrD5py+zpc4qR05sgdQ3i4VNPnORNWg6/fs
  ZQOYYM9VzA4cy6EkaE9hHInzX5ALK8JHpnpb4RFIrg9IqkxzEIvTfz9efDivhtEt
  +SbOKipelYZScMJYS0q0ReMZCxCzFI/zcyc1UAwhhXsCUNHUGB5JTSx1muaEhHsC
  AwEAAaOBszCBsDAPBgNVHRMECDAGAQH/AgEAMIGcBgNVHREEgZQwgZGCLXJlZ2lz
  dHJ5Lm9jcC1lZGdlLWNsdXN0ZXItMC5xZS5sYWIucmVkaGF0LmNvbYIGc2VhbDM2
  hxD9Lm9EXdgAAAAAAAAAAAABhxD+gAAAAAAAAFBUAP/+PRHWhxD+gAAAAAAAAPxU
  AP/+mMr6hxAmIABSAAAjQA7Eev/+ZqN4hxD+gAAAAAAAAA7Eev/+ZqN4MA0GCSqG
  SIb3DQEBCwUAA4IBAQB+XzmMNo/KkjaIX0JdVB5QZgdswSx8i7eQskYCu7QfR4Iy
  qcZLQ3Nst0U3I8I4TQhfN2bZKDfK6PSpySSujgliMqxfSYOPrdekBkSBQbAf5KUK
  5RvFUtwQKdR8M0BAB9Q9Tvyv8nhKc9Pl3uLb3RCIfcKq9CVj+SnhwNEXBeBtTh2Y
  H76qhHG9CaWrbsel9fm8Il2DjrwAleRUlV2d+ypaaJcelKE9nlLkOfzxSvHrk8MJ
  da9qrOd4kqM0iv0mF4N3JH6NEFujiCq7zz7quV/h4F7pSIKglmsjXzq7Izl9dnOB
  h7FRir3pzOfR2HfV2ksUU1gCx65w22PPs0fkVut2
  -----END CERTIFICATE-----
pullSecret: |
  { "auths": { "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000": { "auth": "b2NwLWVkZ2U6b2NwLWVkZ2UtcGFzcw==" } }}
fips: false
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC76dNlV34b/Q6d9f6zFG9mxL9/S41drT6jFEbVTNG9LGREH8Co2ezBomPdGj8mb3lZ4vuG+r7l0c4/VL5ug2ejoVsRV7R/o4HoQxQtNq75ukIApZV2flPsSImJf9N+2osrLZ+EQSp8aTr6uKA/4PqkkC8JvuN9wwMQK3ey7jdj7rwqQDHLsvj4OGSo8YKxjjFlu6Dfa4/1dP4hhdV00gO9bIzlIUsBPmvUW6vKzgzphrcdN9L+iS+dHVju5/oWoFEEUfilE183nz4cERXhcDtz5UdyC62enKrPabkyB0r5pV0WnGtUIr9/6B7iXVspF1j9HgSdh++Nw6tT5sEN4tuQLzna4WMDw/uG9o+gXwDX9Ozp8rL0qWy8GdaZGHqdPK9aNvSM+018ZRUO2eE8kt/Ntj7MqcUPSUyZwLSP7UyDfmAS3qgLhzIc4Hjc/3i7v776DhvoFWYCE9Y0Woj/EOAfiDJgiPCxWFKrNcB6X4/EzvTOoxoEmR9u91+0DvLRImk= kni@provisionhost-0-0.ocp-edge-cluster-0.qe.lab.redhat.com
imageContentSources:
- mirrors:
  - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/localimages/local-release-image
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev


I tried to verify it as indicated in the cloned bug (https://bugzilla.redhat.com/show_bug.cgi?id=1888861) and i don't see indication for IPV6 address here, only IPV4:


$ oc rsh -n openshift-kube-apiserver kube-apiserver-master-0-0
$ cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml
....
...........
................

resources/configmaps/bound-sa-token-signing-certs"],"servicesSubnet":"172.30.0.0/16","servingInfo":{"bindAddress":"0.0.0.0:6443","bindNetwork":"tcp4","cipherSuites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA

................
............
....

but i might be wrong testing it, so would appreciate for additional info. 


must-gather:
-------------
Unfortunately i could not add a must-gather due to this error, but let me know if there is any additional info needed:

[kni@provisionhost-0-0 ~]$ oc adm must-gather
[must-gather      ] OUT unable to resolve the imagestream tag openshift/must-gather:latest
[must-gather      ] OUT 
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest
[must-gather      ] OUT namespace/openshift-must-gather-w5blx created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-vxp5z created
[must-gather      ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created
[must-gather-mbq8z] OUT gather did not start: unable to pull image: ImagePullBackOff: Back-off pulling image "quay.io/openshift/origin-must-gather:latest"
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-vxp5z deleted
[must-gather      ] OUT namespace/openshift-must-gather-w5blx deleted
error: gather did not start for pod must-gather-mbq8z: unable to pull image: ImagePullBackOff: Back-off pulling image "quay.io/openshift/origin-must-gather:latest"

Comment 12 Dan Winship 2020-11-30 13:24:16 UTC
> version   4.6.3     True        False         96m     Cluster version is 4.6.3

The fix is in 4.6.4 and later.

Comment 13 Shelly Miron 2020-12-02 13:13:13 UTC
Verified on OCP4.6.5, using the steps described in comment 10:

$ oc rsh -n openshift-kube-apiserver kube-apiserver-master-0-0.ocp-edge-cluster-0.qe.lab.redhat.com
$ cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml

......
...
..
servicesSubnet":"172.30.0.0/16,fd02::/112",
"servingInfo":{"bindAddress":"0.0.0.0:6443","bindNetwork":"tcp4","cipherSuites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"]

..
...
.....

both IPV4, IPV6 addresses deployed for the kube-apiserver-master-0-0 pod

Comment 16 errata-xmlrpc 2020-12-14 13:50:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5259


Note You need to log in before you can comment on or make changes to this bug.