When converting a service from single stack to dual stack, IPv6 DNAT rules are not created (likely any other required logic isn't executed, either, to be verified). Very similar to what's reported in https://bugzilla.redhat.com/show_bug.cgi?id=2028812 for 4.9 ~~~ [root@ovnkubernetes ~]# ./kind.sh --ipv6 --ha-enabled [root@ovnkubernetes ~]# export KUBECONFIG=~/admin.conf [root@ovnkubernetes ~]# oc apply -f netshoot.yaml service/netshoot-service created deployment.apps/netshoot-deployment created [root@ovnkubernetes ~]# oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES netshoot-deployment-5b5567c8bc-5h6rm 0/1 ContainerCreating 0 4s <none> ovn-worker <none> <none> [root@ovnkubernetes ~]# oc get pods -o wide --watch NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES netshoot-deployment-5b5567c8bc-5h6rm 1/1 Running 0 36s 10.244.0.4 ovn-worker <none> <none> ^C[root@ovnkubernetes ~]# oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4m54s netshoot-service NodePort 10.96.226.193 <none> 27017:30000/TCP 40s [root@ovnkubernetes ~]# docker exec -it ovn-worker iptables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT -A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.226.193:27017 [root@ovnkubernetes ~]# docker exec -it ovn-worker ip6tables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT [root@ovnkubernetes ~]# oc edit netshoot-service error: the server doesn't have a resource type "netshoot-service" [root@ovnkubernetes ~]# oc edit svc netshoot-service service/netshoot-service edited [root@ovnkubernetes ~]# oc get -o yaml svc netshoot-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"netshoot-service","namespace":"default"},"spec":{"externalTrafficPolicy":"Cluster","internalTrafficPolicy":"Cluster","ports":[{"name":"http","nodePort":30000,"port":27017,"targetPort":8080}],"selector":{"app":"netshoot-pod"},"type":"NodePort"}} creationTimestamp: "2021-12-09T11:16:56Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubectl.kubernetes.io/last-applied-configuration: {} f:spec: f:externalTrafficPolicy: {} f:ports: .: {} k:{"port":27017,"protocol":"TCP"}: .: {} f:name: {} f:nodePort: {} f:port: {} f:protocol: {} f:targetPort: {} f:selector: .: {} f:app: {} f:sessionAffinity: {} f:type: {} manager: kubectl-client-side-apply operation: Update time: "2021-12-09T11:16:56Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:ipFamilies: {} f:ipFamilyPolicy: {} manager: kubectl-edit operation: Update time: "2021-12-09T11:18:14Z" name: netshoot-service namespace: default resourceVersion: "1387" uid: 400f22a3-71da-4fc6-8c75-2937e86d69f3 spec: clusterIP: 10.96.226.193 clusterIPs: - 10.96.226.193 - fd00:10:96::713 externalTrafficPolicy: Cluster ipFamilies: - IPv4 - IPv6 ipFamilyPolicy: RequireDualStack ports: - name: http nodePort: 30000 port: 27017 protocol: TCP targetPort: 8080 selector: app: netshoot-pod sessionAffinity: None type: NodePort status: loadBalancer: {} [root@ovnkubernetes ~]# docker exec -it ovn-worker ip6tables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT [root@ovnkubernetes ~]# docker exec -it ovn-worker iptables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT -A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.226.193:27017 [root@ovnkubernetes ~]# ~~~ In 4.10, this seems to impact only the logic where the nodeport is exposed via iptables rules. The OVN part seems to work fine after an update: ~~~ # from the outside, getting in via breth0 [root@ovnkubernetes contrib]# curl 172.18.0.4:31557 NOW: 2021-12-09 13:53:04.225855518 +0000 UTC m=+315.049015877[root@ovnkubernetes contrib]# [root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00 inet6 fc00:f853:ccd:e793::4/64 scope global nodad [root@ovnkubernetes contrib]# curl [fc00:f853:ccd:e793::4]:31557 NOW: 2021-12-09 13:53:34.803912586 +0000 UTC m=+340.976158041[root@ovnkubernetes contrib]# ~~~ From the node itself, getting in via local switching: ~~~ [root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00 inet6 fc00:f853:ccd:e793::4/64 scope global nodad [root@ovnkubernetes contrib]# docker exec -it ovn-worker ip6tables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT this seems to impact only the logic where the nodeport is exposed via iptables rules. The OVN part seems to work fine after an update: ~~~ (...) I1209 13:49:28.206265 2444 gateway_shared_intf.go:552] Skipping service update for: nodeportsvc as change does not apply to any of .Spec.Ports, .Spec.ExternalIP, .Spec.ClusterIP, .Spec.ClusterIPs, .Spec.Type, .Status.LoadBalancer.Ingress, .Spec.ExternalTrafficPolicy (...) ~~~ ~~~ # from the outside, getting in via breth0 [root@ovnkubernetes contrib]# curl 172.18.0.4:31557 NOW: 2021-12-09 13:53:04.225855518 +0000 UTC m=+315.049015877[root@ovnkubernetes contrib]# [root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00 inet6 fc00:f853:ccd:e793::4/64 scope global nodad [root@ovnkubernetes contrib]# curl [fc00:f853:ccd:e793::4]:31557 NOW: 2021-12-09 13:53:34.803912586 +0000 UTC m=+340.976158041[root@ovnkubernetes contrib]# ~~~ From the node itself, getting in via local switching: ~~~ [root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00 inet6 fc00:f853:ccd:e793::4/64 scope global nodad [root@ovnkubernetes contrib]# docker exec -it ovn-worker ip6tables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT [root@ovnkubernetes contrib]# docker exec -it ovn-worker /bin/bash root@ovn-worker:/# curl [fc00:f853:ccd:e793::4]:31557 ^C root@ovn-worker:/# ip r g fc00:f853:ccd:e793::4 local fc00:f853:ccd:e793::4 dev lo table local proto kernel src fc00:f853:ccd:e793::4 metric 0 pref medium root@ovn-worker:/# ip r g 172.18.0.4 local 172.18.0.4 dev lo src 172.18.0.4 uid 0 cache <local> root@ovn-worker:/# curl 172.18.0.4:31557 NOW: 2021-12-09 13:56:30.706767405 +0000 UTC m=+525.157635966root@ovn-worker:/# ip r g fc00:f853:ccd:e793::4 local fc00:f853:ccd:e793::4 dev lo table local proto kernel src fc00:f853:ccd:e793::4 metric 0 pref medium root@ovn-worker:/# root@ovn-worker:/# root@ovn-worker:/# timeout 10 curl [fc00:f853:ccd:e793::4]:31557 root@ovn-worker:/# ~~~ [root@ovnkubernetes contrib]# docker exec -it ovn-worker /bin/bash root@ovn-worker:/# curl [fc00:f853:ccd:e793::4]:31557 ^C root@ovn-worker:/# ip r g 172.18.0.4 local 172.18.0.4 dev lo src 172.18.0.4 uid 0 cache <local> root@ovn-worker:/# curl 172.18.0.4:31557 NOW: 2021-12-09 13:56:30.706767405 +0000 UTC m=+525.157635966root@ovn-worker:/# ip r g fc00:f853:ccd:e793::4 local fc00:f853:ccd:e793::4 dev lo table local proto kernel src fc00:f853:ccd:e793::4 metric 0 pref medium ~~~
Upstream bug: https://github.com/ovn-org/ovn-kubernetes/issues/2700
Verification: ========================= Create an ovn-kubernetes dualstack cluster (baremetal platforms only) with gateway mode shared (the default for OCP 4.9 and 4.10). Create a singlestack nginx service and deployment, e.g.: ~~~ cat <<'EOF' > nginx.yaml --- apiVersion: v1 kind: Service metadata: name: nginx-service spec: ipFamilyPolicy: SingleStack type: NodePort selector: app: nginx-pod externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ports: # By default and for convenience, the `targetPort` is set to the same value as the `port` field. - port: 27017 targetPort: 80 # Optional field # By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767) nodePort: 30000 name: http --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx-deployment spec: replicas: 1 selector: matchLabels: app: nginx-pod template: metadata: labels: app: nginx-pod spec: containers: - name: nginx image: nginx imagePullPolicy: Always EOF oc apply -f nginx.yaml ~~~ Wait until the pod and service are deployed. ~~~ oc get pod,svc ~~~ Next, edit the service: ~~~ oc edit svc nginx-service ~~~ And change the spec.ipFamilyPolicy: ~~~ spec: (...) ipFamilyPolicy: RequireDualStack (...) ~~~ The service should update now to DualStack. Get the pod's node: ~~~ oc get pods -l app=nginx-pod -o wide ~~~ Verification part 1): Connect to the node: ~~~ oc debug node/<name> chroot /host ~~~ And verify for IPv4: ~~~ iptables-save | grep NODE ~~~ And verify for IPv6: ~~~ ip6tables-save | grep NODE ~~~ Failure: Only for the IPv4 protocol, there is an IPv6 iptables rule with port 30000. ~~~ root@ovn-worker2:/# iptables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT -A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.153.179:27017 root@ovn-worker2:/# iptables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT -A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.153.179:27017 root@ovn-worker2:/# ip6tables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT ~~~ Success: For both protocols, there should be an IPv6 iptables rules with port 30000: ~~~ root@ovn-worker:/# iptables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT -A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.237.240:27017 root@ovn-worker:/# ip6tables-save | grep NODE :OVN-KUBE-NODEPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A OUTPUT -j OVN-KUBE-NODEPORT -A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination [fd00:10:96::1c77]:27017 ~~~ Verification part 2): Connect to the node: ~~~ oc debug node/<name> chroot /host ~~~ Get the node's IP and IPv6 address: ~~~ ip a ls dev br-ex # e.g., this might yield: 172.18.0.2 ip -6 a ls dev br-ex # e.g., this might yield: fc00:f853:ccd:e793::2 ~~~ Query port 30000 for the IP/IPv6 address obtained earlier, from the node itself and also from another node on the machinenetwork: ~~~ curl --max-time 10 172.18.0.2:30000 curl --max-time 10 [fc00:f853:ccd:e793::2]:30000 ~~~ Failure: curl works only for IPv4, but times out for IPv6: ~~~ root@ovn-worker2:/# curl --connect-timeout 10 172.18.0.2:30000 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> root@ovn-worker2:/# curl --max-time 10 [fc00:f853:ccd:e793::2]:30000 curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received ~~~ Success: curl works for both protocols: ~~~ [root@ovn-worker:/# curl --max-time 10 172.18.0.4:30000 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> root@ovn-worker:/# curl --max-time 10 [fc00:f853:ccd:e793::4]:30000 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> ~~~
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056