Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1808856

Summary: Networking Granular Checks tests fail when using NodePort on oVirt
Product: OpenShift Container Platform Reporter: Gal Zaidman <gzaidman>
Component: NetworkingAssignee: Ben Bennett <bbennett>
Networking sub component: openshift-sdn QA Contact: zhaozhanqi <zzhao>
Status: CLOSED DUPLICATE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: aconstan
Version: 4.4   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-02 14:56:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1798176    

Description Gal Zaidman 2020-03-01 16:21:48 UTC 
For a few months we saw failing conformance network tests on oVirt, we saw them on other platforms but they were ports issues which where fixed on:

https://bugzilla.redhat.com/show_bug.cgi?id=1763936

One very important thing to know is that when we created a new cluster with "Network Type: OVNKubernetes" the tests passed, we ran them manually and with the openshift-test tool, so we are think this issue is with SDN.

We can provide an env for debugging the issue.

This is a detailed output of our debugging:

The Test:

openshift-tests run-test "[sig-network] Networking Granular Checks: Services [Top Level] [sig-network] Networking Granular Checks: Services should function for node-Service: udp [Suite:openshift/conformance/parallel] [Suite:k8s]"

------------------------------

The Fail:
STEP: dialing(udp) 192.168.204.135 (node) --> 192.168.204.135:32146 (nodeIP)

Mar  1 09:49:36.769: INFO: ExecWithOptions {Command:[/bin/sh -c echo hostName | nc -w 2 -u 192.168.204.135 32146 | grep -v '^\s*$'] Namespace:e2e-nettest-8609 PodName:host-test-container-pod ContainerName:agnhost Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
Mar  1 09:49:39.124: INFO: Failed to execute "echo hostName | nc -w 2 -u 192.168.204.135 32146 | grep -v '^\\s*$'": command terminated with exit code 1, stdout: "", stderr: ""
Mar  1 09:49:39.125: INFO: Waiting for [netserver-0 netserver-1] endpoints (expected=[netserver-0 netserver-1], actual=[])

Mar  1 09:49:41.164: INFO: ExecWithOptions {Command:[/bin/sh -c echo hostName | nc -w 2 -u 192.168.204.135 32146 | grep -v '^\s*$'] Namespace:e2e-nettest-8609 PodName:host-test-
container-pod ContainerName:agnhost Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
Mar  1 09:49:43.497: INFO: Failed to execute "echo hostName | nc -w 2 -u 192.168.204.135 32146 | grep -v '^\\s*$'": command terminated with exit code 1, stdout: "", stderr: ""
Mar  1 09:49:43.497: INFO: Waiting for [netserver-0 netserver-1] endpoints (expected=[netserver-0 netserver-1], actual=[])

Mar  1 09:49:45.536: INFO: ExecWithOptions {Command:[/bin/sh -c echo hostName | nc -w 2 -u 192.168.204.135 32146 | grep -v '^\s*$'] Namespace:e2e-nettest-8609 PodName:host-test-container-pod ContainerName:agnhost Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
Mar  1 09:49:47.877: INFO: Waiting for [netserver-1] endpoints (expected=[netserver-0 netserver-1], actual=[netserver-0])

Mar  1 09:49:49.915: INFO: ExecWithOptions {Command:[/bin/sh -c echo hostName | nc -w 2 -u 192.168.204.135 32146 | grep -v '^\s*$'] Namespace:e2e-nettest-8609 PodName:host-test-container-pod ContainerName:agnhost Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
Mar  1 09:49:52.249: INFO: Waiting for [netserver-1] endpoints (expected=[netserver-0 netserver-1], actual=[netserver-0])


What we see in the logs:
We see that when we try to use the nodeport service we are directed only to netserver-0 and we never get directed to netserver-1.
The test expect to get both netserver-0 and netserver-1

------------------------------

To recreate the env we started a normal cluster on our CI, ran the test and paused it in the middle, so we have the exact same env as the test which is
running from the CI.

The Env:
**I added a small pic for showing how the env looks

The Nodes:
gzaidman@gzaidman ocp $ oc get nodes
NAME                           STATUS   ROLES    AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                                     
ovirt04-5pvm6-master-0         Ready    master   5h42m   v1.17.1   192.168.204.131   <none>        Red Hat Enterprise Linux CoreOS 44.81.202003010130-0 (Ootpa)   
ovirt04-5pvm6-master-1         Ready    master   5h42m   v1.17.1   192.168.204.132   <none>        Red Hat Enterprise Linux CoreOS 44.81.202003010130-0 (Ootpa)   
ovirt04-5pvm6-master-2         Ready    master   5h43m   v1.17.1   192.168.204.130   <none>        Red Hat Enterprise Linux CoreOS 44.81.202003010130-0 (Ootpa)   
ovirt04-5pvm6-worker-0-6fl5v   Ready    worker   5h29m   v1.17.1   192.168.204.135   <none>        Red Hat Enterprise Linux CoreOS 44.81.202003010130-0 (Ootpa)   
ovirt04-5pvm6-worker-0-hgwd6   Ready    worker   5h33m   v1.17.1   192.168.204.134   <none>        Red Hat Enterprise Linux CoreOS 44.81.202003010130-0 (Ootpa)   

The network (OpenShiftSDN):
gzaidman@gzaidman ocp $ oc get network cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2020-03-01T08:16:39Z"
  generation: 2
  name: cluster
  resourceVersion: "2334"
  selfLink: /apis/config.openshift.io/v1/networks/cluster
  uid: 668eb9fa-edf0-4738-9656-e6626442db3c
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
status:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  clusterNetworkMTU: 1450
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16

gzaidman@gzaidman ocp $ oc get -n openshift-sdn pods -owide
NAME                   READY   STATUS    RESTARTS   AGE     IP                NODE                           NOMINATED NODE   READINESS GATES
ovs-2rjlz              1/1     Running   0          6h41m   192.168.204.134   ovirt04-5pvm6-worker-0-hgwd6   <none>           <none>
ovs-jfsfk              1/1     Running   0          6h37m   192.168.204.135   ovirt04-5pvm6-worker-0-6fl5v   <none>           <none>
ovs-npmss              1/1     Running   0          6h50m   192.168.204.131   ovirt04-5pvm6-master-0         <none>           <none>
ovs-q4z9g              1/1     Running   0          6h50m   192.168.204.132   ovirt04-5pvm6-master-1         <none>           <none>
ovs-thtmj              1/1     Running   0          6h50m   192.168.204.130   ovirt04-5pvm6-master-2         <none>           <none>
sdn-controller-4msth   1/1     Running   0          6h50m   192.168.204.131   ovirt04-5pvm6-master-0         <none>           <none>
sdn-controller-bf6cx   1/1     Running   0          6h50m   192.168.204.132   ovirt04-5pvm6-master-1         <none>           <none>
sdn-controller-zngwv   1/1     Running   0          6h50m   192.168.204.130   ovirt04-5pvm6-master-2         <none>           <none>
sdn-kfkql              1/1     Running   0          6h50m   192.168.204.132   ovirt04-5pvm6-master-1         <none>           <none>
sdn-kkss8              1/1     Running   0          6h41m   192.168.204.134   ovirt04-5pvm6-worker-0-hgwd6   <none>           <none>
sdn-tjkvw              1/1     Running   0          6h37m   192.168.204.135   ovirt04-5pvm6-worker-0-6fl5v   <none>           <none>
sdn-w9qwv              1/1     Running   0          6h50m   192.168.204.130   ovirt04-5pvm6-master-2         <none>           <none>
sdn-zstj5              1/1     Running   0          6h50m   192.168.204.131   ovirt04-5pvm6-master-0         <none>           <none>


The Services:
gzaidman@gzaidman ocp $ oc get svc -owide
NAME                       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                     AGE     SELECTOR
node-port-service          NodePort   172.30.109.98    <none>        80:32446/TCP,90:32146/UDP   4h38m   selector-1f4cf665-9181-4ba4-9f92-1548767d689e=true
session-affinity-service   NodePort   172.30.190.106   <none>        80:31102/TCP,90:31569/UDP   4h38m   selector-1f4cf665-9181-4ba4-9f92-1548767d689e=true

The Pods:
The test creates 4 pods:

gzaidman@gzaidman ocp $ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
host-test-container-pod   1/1     Running   0          3m8s
netserver-0               1/1     Running   0          3m34s
netserver-1               1/1     Running   0          3m34s
test-container-pod        1/1     Running   0          3m8s

Debugging:

On the test output (and when we checked), we saw that we can always get netserver-0 and he is not the problem. so we deleted netserver-0 to focus on netserver-1.
So now we have:

gzaidman@gzaidman ocp $ oc get pods -owide
NAME                      READY   STATUS    RESTARTS   AGE     IP                NODE                           NOMINATED NODE   READINESS GATES
host-test-container-pod   1/1     Running   0          4h37m   192.168.204.135   ovirt04-5pvm6-worker-0-6fl5v   <none>           <none>
netserver-1               1/1     Running   0          4h37m   10.131.0.26       ovirt04-5pvm6-worker-0-hgwd6   <none>           <none>
test-container-pod        1/1     Running   0          4h37m   10.128.2.14       ovirt04-5pvm6-worker-0-6fl5v   <none>           <none>

netserver-1 is exposed under the service: node-port-service

gzaidman@gzaidman ocp $ oc get svc node-port-service -oyaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-03-01T09:49:12Z"
  name: node-port-service
  namespace: e2e-nettest-8609
  resourceVersion: "60362"
  selfLink: /api/v1/namespaces/e2e-nettest-8609/services/node-port-service
  uid: 569231ac-5461-4816-9ce9-78d50cefe22f
spec:
  clusterIP: 172.30.109.98
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 32446
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: udp
    nodePort: 32146
    port: 90
    protocol: UDP
    targetPort: 8081
  selector:
    selector-1f4cf665-9181-4ba4-9f92-1548767d689e: "true"
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

When go into test-container-pod and run :
echo hostName | nc -w 2 -u 192.168.204.135 32146
on the first time we get "netserver-1"
but if we run it again we will not get anything
only if we will wait for about 30 secs in between the commends we will get "netserver-1"

we go into ovirt04-5pvm6-worker-0-6fl5v which is the client-pod (test-container-pod) worker, and execute tcpdump with the node port

tcpdump -vvvnnei any udp port 32146
**First try (works)**
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
10:55:25.433401   P 0a:58:0a:80:02:0e ethertype IPv4 (0x0800), length 53: (tos 0x0, ttl 64, id 18160, offset 0, flags [DF], proto UDP (17), length 37)
    10.128.2.14.35188 > 192.168.204.135.32146: [bad udp cksum 0x99e0 -> 0xbd5c!] UDP, length 9
10:55:25.433720  In 0a:58:0a:80:02:0e ethertype IPv4 (0x0800), length 53: (tos 0x0, ttl 64, id 18160, offset 0, flags [DF], proto UDP (17), length 37)
    10.128.2.14.35188 > 192.168.204.135.32146: [udp sum ok] UDP, length 9
10:55:25.434886 Out 8a:74:03:82:67:89 ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 63, id 8352, offset 0, flags [DF], proto UDP (17), length 39)
    192.168.204.135.32146 > 10.128.2.14.35188: [udp sum ok] UDP, length 11
10:55:25.434972 Out 8a:74:03:82:67:89 ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 63, id 8352, offset 0, flags [DF], proto UDP (17), length 39)
    192.168.204.135.32146 > 10.128.2.14.35188: [udp sum ok] UDP, length 11


**Second one (after 2 sec no response)**
10:55:28.232940   P 0a:58:0a:80:02:0e ethertype IPv4 (0x0800), length 53: (tos 0x0, ttl 64, id 18268, offset 0, flags [DF], proto UDP (17), length 37)
    10.128.2.14.44785 > 192.168.204.135.32146: [bad udp cksum 0x99e0 -> 0x97df!] UDP, length 9
10:55:28.232963  In 0a:58:0a:80:02:0e ethertype IPv4 (0x0800), length 53: (tos 0x0, ttl 64, id 18268, offset 0, flags [DF], proto UDP (17), length 37)
    10.128.2.14.44785 > 192.168.204.135.32146: [bad udp cksum 0x99e0 -> 0x97df!] UDP, length 9

**Notice the bad udp cksum**

On the other worker when we run tcpdump there is no output.


connection track from the ovs-pod on the client node (ovirt04-5pvm6-worker-0-6fl5v)

conntrack -L | grep 8081 | grep udp
conntrack v1.4.4 (conntrack-tools): 1950 flow entries have been shown.
udp      17 110 src=10.128.2.14 dst=192.168.204.135 sport=53965 dport=32146 src=10.131.0.26 dst=10.128.2.1 sport=8081 dport=33840 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
udp      17 16 src=10.128.2.14 dst=192.168.204.135 sport=34897 dport=32146 [UNREPLIED] src=10.131.0.26 dst=10.128.2.1 sport=8081 dport=52827 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
udp      17 162 src=10.128.2.14 dst=192.168.204.135 sport=38058 dport=32146 src=10.131.0.26 dst=10.128.2.1 sport=8081 dport=35380 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
udp      17 150 src=10.128.2.14 dst=192.168.204.135 sport=46387 dport=32146 src=10.131.0.26 dst=10.128.2.1 sport=8081 dport=44640 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
udp      17 67 src=10.128.2.14 dst=192.168.204.135 sport=53198 dport=32146 src=10.131.0.26 dst=10.128.2.1 sport=8081 dport=64171 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1

**new conntrack on every call**

connection track from the ovs-pod on the server node (ovirt04-5pvm6-worker-0-hgwd6)

conntrack -L | grep 8081 | grep udp
conntrack v1.4.4 (conntrack-tools): 1929 flow entries have been shown.
udp      17 28 src=10.131.0.26 dst=10.128.2.1 sport=8081 dport=64085 [UNREPLIED] src=10.128.2.1 dst=10.131.0.26 sport=64085 dport=8081 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=2

The netcat command terminate successfully again after this line disappear (30 sec time out)
Comment 1 Alexander Constantinescu 2020-03-02 14:56:37 UTC 
Hi

Thanks for raising this issue. It's already know and tracked however by: 1794714. 

Best regards,
Alexander

*** This bug has been marked as a duplicate of bug 1794714 ***