Bug 1928164

Summary: Traffic to outside the cluster redirected when OVN is used and NodePort service is configured
Product: OpenShift Container Platform Reporter: Lucas López Montero <llopezmo>
Component: NetworkingAssignee: Alexander Constantinescu <aconstan>
Networking sub component: ovn-kubernetes QA Contact: Arti Sood <asood>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aconstan, bbennett, josef.meier, rbrattai, skrenger
Version: 4.6   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1946696 (view as bug list) Environment:
Last Closed: 2021-07-27 22:44:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1946696    

Description Lucas López Montero 2021-02-12 14:23:15 UTC 
Description of problem:

As explained on this issue [1] (fix here [2]); when OVN is being used in a cluster, its local node firewalls are redirecting the traffic to the port configured on NodePort without filtering the destination IP.


How reproducible:


Steps to Reproduce:
oc new-project nodeport-issue
oc new-app https://github.com/openshift/ruby-hello-world
oc expose deployment ruby-hello-world --type=NodePort --name=ruby-hello-world-nodeport
oc describe services ruby-hello-world-nodeport | grep "NodePort:"

Login to one on the nodes and execute:
curl -4 -v http://www.google.de:<NodePort>

Actual results:

The output contains something like "Welcome to an OpenShift v3 Demo App".


Expected results:

Traffic should be directed to www.google.de in the example above



[1] https://github.com/ovn-org/ovn-kubernetes/issues/1981
[2] https://github.com/ovn-org/ovn-kubernetes/pull/2002
Comment 3 Alexander Constantinescu 2021-03-05 09:13:49 UTC 
This made it in with the latest downstream merge: https://github.com/openshift/ovn-kubernetes/pull/440 so setting to MODIFIED
Comment 5 Alexander Constantinescu 2021-03-10 16:29:12 UTC 
*** Bug 1934737 has been marked as a duplicate of this bug. ***
Comment 6 Arti Sood 2021-03-15 22:07:08 UTC 
Verification:-

Build used for verification
oc version
Client Version: 4.7.0
Server Version: 4.8.0-0.nightly-2021-03-14-134919
Kubernetes Version: v1.20.0+e1bc274

Cluster Information

[asood@asood ~]$ oc get network -o jsonpath='{.items[*].status.networkType}'
OVNKubernetes

oc get --namespace openshift-ingress-operator ingresscontrollers/default --output jsonpath='{.status.endpointPublishingStrategy.type}'
NodePortService

Created the app and exposed the deployment.


[asood@asood ~]$ oc describe services ruby-hello-world-nodeport | grep "NodePort:"
NodePort:                 <unset>  31729/TCP


oc get nodes
NAME                                  STATUS   ROLES    AGE     VERSION
asood-03-15-2-s7gqc-compute-0         Ready    worker   5h14m   v1.20.0+e1bc274
asood-03-15-2-s7gqc-compute-1         Ready    worker   5h14m   v1.20.0+e1bc274
asood-03-15-2-s7gqc-compute-2         Ready    worker   5h14m   v1.20.0+e1bc274
asood-03-15-2-s7gqc-control-plane-0   Ready    master   5h26m   v1.20.0+e1bc274
asood-03-15-2-s7gqc-control-plane-1   Ready    master   5h26m   v1.20.0+e1bc274
asood-03-15-2-s7gqc-control-plane-2   Ready    master   5h26m   v1.20.0+e1bc274


oc debug node/asood-03-15-2-s7gqc-compute-0
Starting pod/asood-03-15-2-s7gqc-compute-0-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.97.115
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# curl -4 -v http://www.google.de:31729
* Rebuilt URL to: http://www.google.de:31729/
*   Trying 172.217.7.195...
* TCP_NODELAY set


Traffic seems to be redirected but do not receive a response.


@aconstan  Should a response be received from www.google.de when curl is issued ?
Comment 7 Josef Meier 2021-03-16 08:26:16 UTC 
In our case we could not even pull Docker Images from external registries with this bug.
Comment 8 Alexander Constantinescu 2021-03-16 09:27:47 UTC 
Hi 

I don't know what google.de:$NODE_PORT is supposed to do, it obviously should not return a response as I don't think Google has anything that will listen on a port of your choice and return a response. 

Why don't you just use an external node (boostrap maybe?), launch a server running on a dedicated port on that node, then create a new service inside the cluster using the same node port and test that you can connect to the EXTERNAL_NODE:PORT without having the internal service interfere?

/Alex
Comment 9 Arti Sood 2021-03-16 18:52:32 UTC 
Marking it verified based on comment#6
Comment 14 errata-xmlrpc 2021-07-27 22:44:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438
Comment 16 Red Hat Bugzilla 2023-09-15 01:01:11 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days