Bug 2045576 - breaking API change in dual-stack Services in 1.23 / 4.10
Summary: breaking API change in dual-stack Services in 1.23 / 4.10
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.9
Hardware: Unspecified
OS: Unspecified
urgent
low
Target Milestone: ---
: 4.9.z
Assignee: Andreas Karis
QA Contact: Weibin Liang
URL:
Whiteboard:
Depends On: 2048645
Blocks: 2047676 2047687 2065303
TreeView+ depends on / blocked
 
Reported: 2022-01-25 17:34 UTC by Dan Winship
Modified: 2022-03-17 15:47 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Release Note
Doc Text:
Cause A rewrite in the kube-apiserver causes a user facing API change. Starting in kube-apiserver 1.23 / OpenShift 4.10, users MUST explicitly specify either "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" for DualStack services to be valid. Consequence Users cannot create DualStack services (services with either 2 different ipFamilies or 2 different clusterIPs) without explicitly specifying "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack". Fix Red Hat OpenShift Container Platform 4.8 and 4.9 now emit a warning through the API that warns of the imminent API change. ovn-kubernetes now emits events with level "Warning" of type "ImplicitDualStackIpFamilyPolicy". Result Users cannot create DualStack services without explicitly specifying "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" starting with Red Hat OpenShift Container Platform 4.10. However, users will be warned in previous versions of Red Hat OpenShift Container Platform about the upcoming API change, both through the API and via events that are emitted by ovn-kubernetes.
Clone Of:
: 2047676 2050632 2065303 (view as bug list)
Environment:
Last Closed: 2022-03-16 11:38:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 938 0 None open Bug 2045576: [CARRY][Downstream-only] Give warning when ipFamilyPolicy implicitly set 2022-01-31 16:39:51 UTC
Red Hat Product Errata RHBA-2022:0798 0 None None None 2022-03-16 11:39:09 UTC

Description Dan Winship 2022-01-25 17:34:44 UTC
kube 1.23 introduced a breaking API change in dual-stack services which I'm just noticing now...

In kube 1.21 and 1.22 (OCP 4.8 and 4.9), the apiserver would default the value of `ipFamilyPolicy` to `RequireDualStack` if you created a Service with two `ipFamilies` or two `clusterIPs` but no explicitly-specified `ipFamilyPolicy`:

    kind: Service
    apiVersion: v1
    metadata:
      name: my-service
    spec:
      type: ClusterIP
      ipFamilies:
        - IPv6
        - IPv4
      ports:
      - port: 80
      selector:
        foo: bar

or

    kind: Service
    apiVersion: v1
    metadata:
      name: my-service
    spec:
      type: ClusterIP
      clusterIPs:
        - 172.30.0.99
        - fd02::9999
      ports:
      - port: 80
      selector:
        foo: bar

This turned out to have some tricky and possibly unfixable broken edge cases, so in 1.23 / 4.10, you MUST explicitly specify either "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" for the service to be valid.

(This was fallout from a MASSIVE rewrite of the apiserver Service-handling code, https://github.com/kubernetes/kubernetes/pull/96684.)


It is hard to say whether any users are actually creating services in this way. Although this behavior was described in the KEP, it never appeared in the official documentation, which always implied that you had to explicitly provide an `ipFamilyPolicy` value (https://github.com/kubernetes/website/blob/release-1.22/content/en/docs/concepts/services-networking/dual-stack.md#services). (It doesn't actually say "you MUST specify ipFamilyPolicy", but it never suggests that it's possible to omit it, and doesn't describe what would happen if you did.)


If we are really concerned about this as an API break, then we could add a mutating web hook to fix things, but presumably we'd have to maintain it forever.

A simpler fix might be to just modify 4.8 and 4.9 to warn loudly if the user tries to create such a service?

We also need to release-note this, and should explicitly mention it to known large dual-stack-using customers.

Comment 1 Stefan Schimanski 2022-01-28 10:27:03 UTC
For reference, an admission webhook for services is a no-go.

Comment 3 Andreas Karis 2022-01-28 11:53:53 UTC
Tasks:
* 1. write 4.8, 4.9, 4.10 release notes
* 2. add metrics and an alert in 4.9 and 4.8 when users write these objects
* 3. potentially block upgrade when those objects have been written recently. We do that similarly with old API versions being used in the last day or so.


For 2., use metadata.managedFields:
~~~
[root@openshift-jumpserver-0 ~]# oc create -f nginx-dualstack.yaml 
service/nginx-dualstack created
[root@openshift-jumpserver-0 ~]# oc get svc nginx-dualstack -o yaml --show-managed-fields
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-01-28T11:46:20Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:internalTrafficPolicy: {}
        f:ipFamilies: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-create
    operation: Update
    time: "2022-01-28T11:46:20Z"
  name: nginx-dualstack
  namespace: openshift-nfs-storage
  resourceVersion: "470944"
  uid: 1d1d1ce1-ecc3-4806-816c-042ef9e884fe
spec:
  clusterIP: fd02::7a6b
  clusterIPs:
  - fd02::7a6b
  - 172.30.218.223
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: RequireDualStack
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
[root@openshift-jumpserver-0 ~]# cat nginx-dualstack.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx-dualstack
spec:
  type: ClusterIP
  selector:
    app: nginx
    #  ipFamilyPolicy: RequireDualStack
    #ipFamilyPolicy: SingleStack
  ipFamilies:
         - IPv6
         - IPv4
  ports:
      # By default and for convenience, the `targetPort` is set to the same value as the `port` field.
    - port: 80
      targetPort: 80
~~~

Comment 12 Weibin Liang 2022-03-03 22:28:03 UTC
Tested and verified in 4.9.0-0.nightly-2022-03-03-120755

Comment 15 errata-xmlrpc 2022-03-16 11:38:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.24 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0798


Note You need to log in before you can comment on or make changes to this bug.