Bug 2031926 - [ipv6dualstack] After SVC conversion from single stack only to RequireDualStack, cannot curl NodePort from the node itself
Summary: [ipv6dualstack] After SVC conversion from single stack only to RequireDualSta...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.10.0
Assignee: Andreas Karis
QA Contact: Anurag saxena
URL:
Whiteboard:
Depends On:
Blocks: 2028812
TreeView+ depends on / blocked
 
Reported: 2021-12-13 17:37 UTC by Andreas Karis
Modified: 2022-03-12 04:39 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:39:26 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 924 0 None open Bug 2031926: Shared gateway: Modification of ClusterIPs shall trigger svc update 2022-01-24 13:00:29 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:39:46 UTC

Description Andreas Karis 2021-12-13 17:37:40 UTC
When converting a service from single stack to dual stack, IPv6 DNAT rules are not created (likely any other required logic isn't executed, either, to be verified).

Very similar to what's reported in https://bugzilla.redhat.com/show_bug.cgi?id=2028812 for 4.9

~~~
[root@ovnkubernetes ~]# ./kind.sh --ipv6 --ha-enabled
[root@ovnkubernetes ~]# export KUBECONFIG=~/admin.conf 
[root@ovnkubernetes ~]# oc apply -f netshoot.yaml 
service/netshoot-service created
deployment.apps/netshoot-deployment created
[root@ovnkubernetes ~]# oc get pods -o wide
NAME                                   READY   STATUS              RESTARTS   AGE   IP       NODE         NOMINATED NODE   READINESS GATES
netshoot-deployment-5b5567c8bc-5h6rm   0/1     ContainerCreating   0          4s    <none>   ovn-worker   <none>           <none>
[root@ovnkubernetes ~]# oc get pods -o wide --watch
NAME                                   READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
netshoot-deployment-5b5567c8bc-5h6rm   1/1     Running   0          36s   10.244.0.4   ovn-worker   <none>           <none>
^C[root@ovnkubernetes ~]# oc get svc
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
kubernetes         ClusterIP   10.96.0.1       <none>        443/TCP           4m54s
netshoot-service   NodePort    10.96.226.193   <none>        27017:30000/TCP   40s
[root@ovnkubernetes ~]# docker exec -it ovn-worker iptables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
-A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.226.193:27017
[root@ovnkubernetes ~]# docker exec -it ovn-worker ip6tables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
[root@ovnkubernetes ~]# oc edit netshoot-service
error: the server doesn't have a resource type "netshoot-service"
[root@ovnkubernetes ~]# oc edit svc netshoot-service
service/netshoot-service edited
[root@ovnkubernetes ~]# oc get -o yaml svc netshoot-service
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"netshoot-service","namespace":"default"},"spec":{"externalTrafficPolicy":"Cluster","internalTrafficPolicy":"Cluster","ports":[{"name":"http","nodePort":30000,"port":27017,"targetPort":8080}],"selector":{"app":"netshoot-pod"},"type":"NodePort"}}
  creationTimestamp: "2021-12-09T11:16:56Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":27017,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:nodePort: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-12-09T11:16:56Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:ipFamilies: {}
        f:ipFamilyPolicy: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-12-09T11:18:14Z"
  name: netshoot-service
  namespace: default
  resourceVersion: "1387"
  uid: 400f22a3-71da-4fc6-8c75-2937e86d69f3
spec:
  clusterIP: 10.96.226.193
  clusterIPs:
  - 10.96.226.193
  - fd00:10:96::713
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - name: http
    nodePort: 30000
    port: 27017
    protocol: TCP
    targetPort: 8080
  selector:
    app: netshoot-pod
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}
[root@ovnkubernetes ~]# docker exec -it ovn-worker ip6tables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
[root@ovnkubernetes ~]# docker exec -it ovn-worker iptables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
-A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.226.193:27017
[root@ovnkubernetes ~]# 
~~~

In 4.10, this seems to impact only the logic where the nodeport is exposed via iptables rules. The OVN part seems to work fine after an update:

~~~
# from the outside, getting in via breth0
[root@ovnkubernetes contrib]# curl 172.18.0.4:31557
NOW: 2021-12-09 13:53:04.225855518 +0000 UTC m=+315.049015877[root@ovnkubernetes contrib]# 
[root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
[root@ovnkubernetes contrib]# curl [fc00:f853:ccd:e793::4]:31557
NOW: 2021-12-09 13:53:34.803912586 +0000 UTC m=+340.976158041[root@ovnkubernetes contrib]# 
~~~

From the node itself, getting in via local switching:
~~~
[root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
[root@ovnkubernetes contrib]# docker exec -it ovn-worker ip6tables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
this seems to impact only the logic where the nodeport is exposed via iptables rules. The OVN part seems to work fine after an update:
~~~
(...)
I1209 13:49:28.206265    2444 gateway_shared_intf.go:552] Skipping service update for: nodeportsvc as change does not apply to any of .Spec.Ports, .Spec.ExternalIP, .Spec.ClusterIP, .Spec.ClusterIPs, .Spec.Type, .Status.LoadBalancer.Ingress, .Spec.ExternalTrafficPolicy
(...)
~~~

~~~
# from the outside, getting in via breth0
[root@ovnkubernetes contrib]# curl 172.18.0.4:31557
NOW: 2021-12-09 13:53:04.225855518 +0000 UTC m=+315.049015877[root@ovnkubernetes contrib]# 
[root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
[root@ovnkubernetes contrib]# curl [fc00:f853:ccd:e793::4]:31557
NOW: 2021-12-09 13:53:34.803912586 +0000 UTC m=+340.976158041[root@ovnkubernetes contrib]# 
~~~

From the node itself, getting in via local switching:
~~~
[root@ovnkubernetes contrib]# docker exec -it ovn-worker ip a | grep fc00
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
[root@ovnkubernetes contrib]# docker exec -it ovn-worker ip6tables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
[root@ovnkubernetes contrib]# docker exec -it ovn-worker /bin/bash
root@ovn-worker:/# curl [fc00:f853:ccd:e793::4]:31557

^C
root@ovn-worker:/# ip r g fc00:f853:ccd:e793::4
local fc00:f853:ccd:e793::4 dev lo table local proto kernel src fc00:f853:ccd:e793::4 metric 0 pref medium
root@ovn-worker:/# ip r g 172.18.0.4           
local 172.18.0.4 dev lo src 172.18.0.4 uid 0 
    cache <local> 
root@ovn-worker:/# curl 172.18.0.4:31557
NOW: 2021-12-09 13:56:30.706767405 +0000 UTC m=+525.157635966root@ovn-worker:/# ip r g fc00:f853:ccd:e793::4
local fc00:f853:ccd:e793::4 dev lo table local proto kernel src fc00:f853:ccd:e793::4 metric 0 pref medium
root@ovn-worker:/# 
root@ovn-worker:/# 
root@ovn-worker:/# timeout 10 curl [fc00:f853:ccd:e793::4]:31557
root@ovn-worker:/# 
~~~
[root@ovnkubernetes contrib]# docker exec -it ovn-worker /bin/bash
root@ovn-worker:/# curl [fc00:f853:ccd:e793::4]:31557

^C
root@ovn-worker:/# ip r g 172.18.0.4           
local 172.18.0.4 dev lo src 172.18.0.4 uid 0 
    cache <local> 
root@ovn-worker:/# curl 172.18.0.4:31557
NOW: 2021-12-09 13:56:30.706767405 +0000 UTC m=+525.157635966root@ovn-worker:/# ip r g fc00:f853:ccd:e793::4
local fc00:f853:ccd:e793::4 dev lo table local proto kernel src fc00:f853:ccd:e793::4 metric 0 pref medium
~~~

Comment 1 Andreas Karis 2021-12-13 17:38:07 UTC
Upstream bug: https://github.com/ovn-org/ovn-kubernetes/issues/2700

Comment 2 Andreas Karis 2022-01-24 20:57:47 UTC
Verification:
=========================

Create an ovn-kubernetes dualstack cluster (baremetal platforms only) with gateway mode shared (the default for OCP 4.9 and 4.10).

Create a singlestack nginx service and deployment, e.g.:
~~~
cat <<'EOF' > nginx.yaml 
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ipFamilyPolicy: SingleStack
  type: NodePort
  selector:
    app: nginx-pod
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
      # By default and for convenience, the `targetPort` is set to the same value as the `port` field.
    - port: 27017
      targetPort: 80
      # Optional field
      # By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)
      nodePort: 30000
      name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-pod
  template:
    metadata:
      labels:
        app: nginx-pod
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Always
EOF
oc apply -f nginx.yaml
~~~

Wait until the pod and service are deployed.
~~~
oc get pod,svc
~~~

Next, edit the service:
~~~
oc edit svc nginx-service
~~~

And change the spec.ipFamilyPolicy:
~~~
spec:
(...)
     ipFamilyPolicy: RequireDualStack
(...)
~~~

The service should update now to DualStack.

Get the pod's node:
~~~
oc get pods -l app=nginx-pod -o wide
~~~

Verification part 1):

Connect to the node:
~~~
oc debug node/<name>
chroot /host
~~~

And verify for IPv4:
~~~
iptables-save  | grep NODE
~~~

And verify for IPv6:
~~~
ip6tables-save  | grep NODE
~~~

Failure:
Only for the IPv4 protocol, there is an IPv6 iptables rule with port 30000.
~~~
root@ovn-worker2:/# iptables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
-A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.153.179:27017
root@ovn-worker2:/# iptables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
-A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.153.179:27017
root@ovn-worker2:/# ip6tables-save | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
~~~

Success:
For both protocols, there should be an IPv6 iptables rules with port 30000:
~~~
root@ovn-worker:/# iptables-save  | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
-A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination 10.96.237.240:27017
root@ovn-worker:/# ip6tables-save  | grep NODE
:OVN-KUBE-NODEPORT - [0:0]
-A PREROUTING -j OVN-KUBE-NODEPORT
-A OUTPUT -j OVN-KUBE-NODEPORT
-A OVN-KUBE-NODEPORT -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j DNAT --to-destination [fd00:10:96::1c77]:27017
~~~

Verification part 2):

Connect to the node:
~~~
oc debug node/<name>
chroot /host
~~~

Get the node's IP and IPv6 address:
~~~
ip a ls dev br-ex       # e.g., this might yield:  172.18.0.2
ip -6 a ls dev br-ex    # e.g., this might yield:  fc00:f853:ccd:e793::2
~~~

Query port 30000 for the IP/IPv6 address obtained earlier, from the node itself and also from another node on the machinenetwork:
~~~
curl --max-time 10  172.18.0.2:30000
curl --max-time 10 [fc00:f853:ccd:e793::2]:30000
~~~

Failure: 
curl works only for IPv4, but times out for IPv6:
~~~
root@ovn-worker2:/# curl --connect-timeout 10  172.18.0.2:30000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@ovn-worker2:/# curl --max-time 10 [fc00:f853:ccd:e793::2]:30000
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
~~~

Success:
curl works for both protocols:
~~~
[root@ovn-worker:/# curl --max-time 10  172.18.0.4:30000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@ovn-worker:/# curl --max-time 10 [fc00:f853:ccd:e793::4]:30000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
~~~

Comment 8 errata-xmlrpc 2022-03-12 04:39:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.