Cause
A rewrite in the kube-apiserver causes a user facing API change. Starting in kube-apiserver 1.23 / OpenShift 4.10, users MUST explicitly specify either "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" for DualStack services to be valid.
Consequence
Users cannot create DualStack services (services with either 2 different ipFamilies or 2 different clusterIPs) without explicitly specifying "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack".
Fix
Red Hat OpenShift Container Platform 4.8 and 4.9 now emit a warning that warns of the imminent API change.
Result
Users cannot create DualStack services without explicitly specifying "ipFamilyPolicy: PreferDualStack" or "ipFamilyPolicy: RequireDualStack" starting with Red Hat OpenShift Container Platform 4.10. However, users will be warned in previous versions of Red Hat OpenShift Container Platform about the upcoming API change.
Verification steps:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.0-0.nightly-2022-04-18-075504 True False 4h43m Cluster version is 4.9.0-0.nightly-2022-04-18-075504
Deploy a cluster with IPv4/IPv6 dualstack.
Create inline services.
$ cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: dualstack-ipfamilies-no-ipfamilypolicy
spec:
type: ClusterIP
selector:
app: test
ipFamilies:
- IPv6
- IPv4
ports:
- port: 80
targetPort: 80
EOF
W0419 17:46:28.379686 54500 warnings.go:70] service.spec.ipFamilyPolicy must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified, this operation will fail starting with Red Hat OpenShift Platform 4.10.
service/dualstack-ipfamilies-no-ipfamilypolicy created
$ cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: dualstack-ipfamilies-ipfamilypolicy-requiredualstack
spec:
type: ClusterIP
selector:
app: test
ipFamilyPolicy: RequireDualStack
ipFamilies:
- IPv6
- IPv4
ports:
- port: 80
targetPort: 80
EOF
service/dualstack-ipfamilies-ipfamilypolicy-requiredualstack created
$ oc get svc | grep dualstack
dualstack-ipfamilies-ipfamilypolicy-requiredualstack ClusterIP fd02::7077 <none> 80/TCP 5s
dualstack-ipfamilies-no-ipfamilypolicy ClusterIP fd02::39c9 <none> 80/TCP 40s
$ oc get events | grep dualstack
3m7s Warning ImplicitDualStackIpFamilyPolicy service/dualstack-ipfamilies-no-ipfamilypolicy DualStack ipFamilyPolicy set implicitly for service default/dualstack-ipfamilies-no-ipfamilypolicy
Results : Only event for dualstack-ipfamilies-no-ipfamilypolicy and no event for dualstack-ipfamilies-ipfamilypolicy-requiredualstack and works as expected.
Repeat with clusterIPs - make sure to choose correct IP addresses from within the cluster's IPv4 and IPv6 service range
Create inline services.
$ cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: dualstack-clusterips-ipfamilypolicy-dualstack
spec:
type: ClusterIP
selector:
app: test
ipFamilyPolicy: PreferDualStack
clusterIP: fd02::dcf9
clusterIPs:
- fd02::dcf9
- 172.30.0.3
ports:
- port: 80
targetPort: 80
EOF
service/dualstack-clusterips-ipfamilypolicy-dualstack created
$ cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: dualstack-clusterips-no-ipfamilypolicy
spec:
type: ClusterIP
selector:
app: test
clusterIP: fd02::cb76
clusterIPs:
- fd02::cb76
- 172.30.0.4
ports:
- port: 80
targetPort: 80
EOF
W0419 17:22:59.301830 52327 warnings.go:70] service.spec.ipFamilyPolicy must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified, this operation will fail starting with Red Hat OpenShift Platform 4.10.
service/dualstack-clusterips-no-ipfamilypolicy created
$ oc get services | grep dualstack
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dualstack-clusterips-ipfamilypolicy-dualstack ClusterIP fd02::dcf9 <none> 80/TCP 3m18s
dualstack-clusterips-no-ipfamilypolicy ClusterIP fd02::cb76 <none> 80/TCP 48s
$ oc get events | grep dualstack
2m37s Warning ImplicitDualStackIpFamilyPolicy service/dualstack-clusterips-no-ipfamilypolicy DualStack ipFamilyPolicy set implicitly for service default/dualstack-clusterips-no-ipfamilypolicy
Results : Only events for dualstack-clusterips-no-ipfamilypolicy and no event for dualstack-clusterips-ipfamilypolicy-dualstack, working as expected.
Create inline services with SingleStack :
$ cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: singlestack-clusterip-ipfamilypolicy-singlestack
spec:
type: ClusterIP
selector:
app: test
ipFamilyPolicy: SingleStack
clusterIP: fd02::dcf9
ports:
- port: 80
targetPort: 80
EOF
service/singlestack-clusterip-ipfamilypolicy-singlestack created
$ cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: singlestack-ipfamilypolicy-singlestack
spec:
type: ClusterIP
selector:
app: test
ipFamilyPolicy: SingleStack
ports:
- port: 80
EOF
service/singlestack-ipfamilypolicy-singlestack created
cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
name: singlestack-clusterip-noipfamilypolicy
spec:
type: ClusterIP
selector:
app: test
clusterIP: fd02::cb76
ports:
- port: 80
targetPort: 80
EOF
service/singlestack-clusterip-noipfamilypolicy created
$ oc get svc | grep singlestack
singlestack-clusterip-ipfamilypolicy-singlestack ClusterIP fd02::dcf9 <none> 80/TCP 12m
singlestack-clusterip-noipfamilypolicy ClusterIP fd02::cb76 <none> 80/TCP 69s
singlestack-ipfamilypolicy-singlestack ClusterIP 172.30.32.60 <none> 80/TCP 2m7s
$ oc get events | grep singlestack
Results : No events should be for 3 singlestack services and works as expected.
Hence all checks works as per expectation, moved bug to verified.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (OpenShift Container Platform 4.9.31 bug fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2022:1605
Verification steps: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2022-04-18-075504 True False 4h43m Cluster version is 4.9.0-0.nightly-2022-04-18-075504 Deploy a cluster with IPv4/IPv6 dualstack. Create inline services. $ cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: dualstack-ipfamilies-no-ipfamilypolicy spec: type: ClusterIP selector: app: test ipFamilies: - IPv6 - IPv4 ports: - port: 80 targetPort: 80 EOF W0419 17:46:28.379686 54500 warnings.go:70] service.spec.ipFamilyPolicy must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified, this operation will fail starting with Red Hat OpenShift Platform 4.10. service/dualstack-ipfamilies-no-ipfamilypolicy created $ cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: dualstack-ipfamilies-ipfamilypolicy-requiredualstack spec: type: ClusterIP selector: app: test ipFamilyPolicy: RequireDualStack ipFamilies: - IPv6 - IPv4 ports: - port: 80 targetPort: 80 EOF service/dualstack-ipfamilies-ipfamilypolicy-requiredualstack created $ oc get svc | grep dualstack dualstack-ipfamilies-ipfamilypolicy-requiredualstack ClusterIP fd02::7077 <none> 80/TCP 5s dualstack-ipfamilies-no-ipfamilypolicy ClusterIP fd02::39c9 <none> 80/TCP 40s $ oc get events | grep dualstack 3m7s Warning ImplicitDualStackIpFamilyPolicy service/dualstack-ipfamilies-no-ipfamilypolicy DualStack ipFamilyPolicy set implicitly for service default/dualstack-ipfamilies-no-ipfamilypolicy Results : Only event for dualstack-ipfamilies-no-ipfamilypolicy and no event for dualstack-ipfamilies-ipfamilypolicy-requiredualstack and works as expected. Repeat with clusterIPs - make sure to choose correct IP addresses from within the cluster's IPv4 and IPv6 service range Create inline services. $ cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: dualstack-clusterips-ipfamilypolicy-dualstack spec: type: ClusterIP selector: app: test ipFamilyPolicy: PreferDualStack clusterIP: fd02::dcf9 clusterIPs: - fd02::dcf9 - 172.30.0.3 ports: - port: 80 targetPort: 80 EOF service/dualstack-clusterips-ipfamilypolicy-dualstack created $ cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: dualstack-clusterips-no-ipfamilypolicy spec: type: ClusterIP selector: app: test clusterIP: fd02::cb76 clusterIPs: - fd02::cb76 - 172.30.0.4 ports: - port: 80 targetPort: 80 EOF W0419 17:22:59.301830 52327 warnings.go:70] service.spec.ipFamilyPolicy must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified, this operation will fail starting with Red Hat OpenShift Platform 4.10. service/dualstack-clusterips-no-ipfamilypolicy created $ oc get services | grep dualstack NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dualstack-clusterips-ipfamilypolicy-dualstack ClusterIP fd02::dcf9 <none> 80/TCP 3m18s dualstack-clusterips-no-ipfamilypolicy ClusterIP fd02::cb76 <none> 80/TCP 48s $ oc get events | grep dualstack 2m37s Warning ImplicitDualStackIpFamilyPolicy service/dualstack-clusterips-no-ipfamilypolicy DualStack ipFamilyPolicy set implicitly for service default/dualstack-clusterips-no-ipfamilypolicy Results : Only events for dualstack-clusterips-no-ipfamilypolicy and no event for dualstack-clusterips-ipfamilypolicy-dualstack, working as expected. Create inline services with SingleStack : $ cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: singlestack-clusterip-ipfamilypolicy-singlestack spec: type: ClusterIP selector: app: test ipFamilyPolicy: SingleStack clusterIP: fd02::dcf9 ports: - port: 80 targetPort: 80 EOF service/singlestack-clusterip-ipfamilypolicy-singlestack created $ cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: singlestack-ipfamilypolicy-singlestack spec: type: ClusterIP selector: app: test ipFamilyPolicy: SingleStack ports: - port: 80 EOF service/singlestack-ipfamilypolicy-singlestack created cat <<'EOF' | oc apply -f - apiVersion: v1 kind: Service metadata: name: singlestack-clusterip-noipfamilypolicy spec: type: ClusterIP selector: app: test clusterIP: fd02::cb76 ports: - port: 80 targetPort: 80 EOF service/singlestack-clusterip-noipfamilypolicy created $ oc get svc | grep singlestack singlestack-clusterip-ipfamilypolicy-singlestack ClusterIP fd02::dcf9 <none> 80/TCP 12m singlestack-clusterip-noipfamilypolicy ClusterIP fd02::cb76 <none> 80/TCP 69s singlestack-ipfamilypolicy-singlestack ClusterIP 172.30.32.60 <none> 80/TCP 2m7s $ oc get events | grep singlestack Results : No events should be for 3 singlestack services and works as expected. Hence all checks works as per expectation, moved bug to verified.