Bug 2047687

Summary: breaking API change in dual-stack Services in 1.23 / 4.10
Product: OpenShift Container Platform Reporter: Andreas Karis <akaris>
Component: NetworkingAssignee: Andreas Karis <akaris>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED NOTABUG Docs Contact:
Severity: low    
Priority: urgent CC: akaris, anusaxen, danw, trozet
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Deprecated Functionality
Doc Text:
Cause A rewrite in the kube-apiserver causes a user facing API change. Starting in kube-apiserver 1.23 / OpenShift 4.10, users MUST explicitly specify either "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" for DualStack services to be valid. Consequence Users cannot create DualStack services (services with either 2 different ipFamilies or 2 different clusterIPs) without explicitly specifying "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack". Fix Red Hat OpenShift Container Platform 4.8 and 4.9 now emit a warning that warns of the imminent API change. Result Users cannot create DualStack services without explicitly specifying "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" starting with Red Hat OpenShift Container Platform 4.10. However, users will be warned in previous versions of Red Hat OpenShift Container Platform about the upcoming API change.
 Story Points: ---
Clone Of: 2047676 Environment:
Last Closed: 2022-01-28 15:25:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2045576, 2047676    
Bug Blocks:    

Description Andreas Karis 2022-01-28 09:55:41 UTC 
+++ This bug was initially created as a clone of Bug #2047676 +++

+++ This bug was initially created as a clone of Bug #2045576 +++

kube 1.23 introduced a breaking API change in dual-stack services which I'm just noticing now...

In kube 1.21 and 1.22 (OCP 4.8 and 4.9), the apiserver would default the value of `ipFamilyPolicy` to `RequireDualStack` if you created a Service with two `ipFamilies` or two `clusterIPs` but no explicitly-specified `ipFamilyPolicy`:

    kind: Service
    apiVersion: v1
    metadata:
      name: my-service
    spec:
      type: ClusterIP
      ipFamilies:
        - IPv6
        - IPv4
      ports:
      - port: 80
      selector:
        foo: bar

or

    kind: Service
    apiVersion: v1
    metadata:
      name: my-service
    spec:
      type: ClusterIP
      clusterIPs:
        - 172.30.0.99
        - fd02::9999
      ports:
      - port: 80
      selector:
        foo: bar

This turned out to have some tricky and possibly unfixable broken edge cases, so in 1.23 / 4.10, you MUST explicitly specify either "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" for the service to be valid.

(This was fallout from a MASSIVE rewrite of the apiserver Service-handling code, https://github.com/kubernetes/kubernetes/pull/96684.)


It is hard to say whether any users are actually creating services in this way. Although this behavior was described in the KEP, it never appeared in the official documentation, which always implied that you had to explicitly provide an `ipFamilyPolicy` value (https://github.com/kubernetes/website/blob/release-1.22/content/en/docs/concepts/services-networking/dual-stack.md#services). (It doesn't actually say "you MUST specify ipFamilyPolicy", but it never suggests that it's possible to omit it, and doesn't describe what would happen if you did.)


If we are really concerned about this as an API break, then we could add a mutating web hook to fix things, but presumably we'd have to maintain it forever.

A simpler fix might be to just modify 4.8 and 4.9 to warn loudly if the user tries to create such a service?

We also need to release-note this, and should explicitly mention it to known large dual-stack-using customers.