Bug 1955192

> I tested on 4.6.25 but still it does not seems to work properly.

> ExternalIP feature do not work .

This is not as simple, it does work as it was intended in this case, let me elaborate on that:
There is no a ExternalIP implementation spec, there is an API field that says:

	// ExternalIPs are used by external load balancers, or can be set by
	// users to handle external traffic that arrives at a node.
	// +optional

So, basically, what this field says is the Service with ClusterIP X can be reached externally in this ExternalIP Y, but it doesn't say how.
As far as I know, there is two common implementation models:

1. Some platforms implement ExternalIPs, adding them literally to the Node, hence, the Service implementation only forwards traffic to the IP if it is present in the Node, otherwise it sends the traffic out of the Node and "something" external "redirects" the traffic to the node with the ExternalIP, it can be a external router, keepalived, ... you need this orchestration otherwise the feature will not work if the traffic arrives in the wrong node.

2. The other implementation of ExternalIPs direct the traffic directed to the External IP directly to the Service, if it knows how to differentiate between local and external traffic, it may discriminate between both, sending the local out of the node and the external to the Service.

One of the problems of the ExternalIP implementation is how to differentiate what traffic is internal or external to that node, remember it says, "handle external traffic that arrives at a node". Per example, in kube-proxy, CNI plugins using "non-bridge" mode for pods always send traffic to the Service because it can not discriminate if the traffic is coming from one pod.

If we take a look to some implementations, there is no consistency:
- kube-proxy iptables has implementation 2. always
- kube-proxy ipvs implements 2. if the CNI plugins doesn't use bridge and 1. if the CNI uses bridge for pods
- cilium seems to use 1. always 
- ovn-kube implements 1. in this case


It is important to mention, that ExternalIPs are affected by "CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs", since ExternalIP are set by the user in Spec the mitigation is to install a webhook to control this field https://github.com/kubernetes/kubernetes/issues/97076#issue-757368290.

Using 1. you reduce the MITM attack surface because the only traffic that you can hijack is the related to the IPs that are in the nodes, I don't know if still can have different attack vectors though, but you have a dependency on an external component,

Using 2. without traffic discrimination it has a much greater attack surface, but is more flexible and independent of an orchestrator to assign the ExternalIPs to the nodes, you just need to "route" the externalIP traffic to any of the cluster nodes and it will reach the Services "automagically"

In this case, OVN-Kubernetes has implemented 1., the request is to implement 2., but there are several "architectural" questions here:
- Does Openshift want to use ExternalIPs, the community consensus is that is not a good feature with a lot of holes and better avoid it.
- Does Openshift has some mechanism to assign ExternalIPs to nodes that makes this feature work with current implementation? or does it want to support it letting the user handle those IPs?
- Does Openshift wants ExternalIP to be flexible and let the users deal with the CVE-2020-8544 ?
- ...

Ok, I justed talked with Tim Rozet and it seems OVN-Kube has a different implementation.
It does send the traffic to the Service despite the ExternalIP is not present, but the ExternalIP is implemented using OVN Load Balancers not iptables rules.

For the record, I did a mistake in my comment, Openshift does have and admission controller that only allows ExternalIPs allowed by the admin, mitigating the CVE mentioned before.
Thanks Dan Winship for the clarification.

patch sent https://github.com/ovn-org/ovn-kubernetes/pull/2232/commits/31935e41862fb0cbceb1d1ad4f4dfe1de19cac8d

This bz should also be fixed by https://github.com/ovn-org/ovn-kubernetes/pull/2244, So I am marking as a dupe.  @anusaxen I would still like this case verified once it merges though which is why I'm leaving the bug open.

@mapandey  Try to verify this bug in v4.9, just want to double check with you on above configuration steps

1. Before using new externalIP 2.2.2.2 for svc, did you update externalIP policy from system admin?
# oc patch network.config.openshift.io cluster --type=merge -p '{"spec":{"externalIP":{"policy": {"allowedCIDRs": null }}}}' 
# oc patch network.config.openshift.io cluster --type=merge -p '{"spec":{"externalIP":{"policy":{"allowedCIDRs":["22.2.2.2/32"]}}}}'

Due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1793099, the above change will take about 220 seconds to take effect.

2. When using new externalIP 2.2.2.2, did you update it from a normal user or system admin?

Tested and verified in 4.9.0-0.nightly-2021-08-07-175228: update svc externalIP from 10.73.116.64 to 2.2.2.2, OVN-KUBE-EXTERNALIP get update too and curl on pass in using new svc externalIP

[root@dell-per740-36 ~]# oc rsh externalip-pod-57f9dd7cfb-967pw
~ $ curl 10.73.116.64:27018
Customer-Blue Test ExternalIP
[root@dell-per740-36 ~]# oc get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP    PORT(S)     AGE
externalip-svc   ClusterIP   172.30.33.188   10.73.116.64   27018/TCP   29m
[root@dell-per740-36 ~]# oc edit svc externalip-svc
service/externalip-svc edited
[root@dell-per740-36 ~]# oc get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
externalip-svc   ClusterIP   172.30.33.188   2.2.2.2       27018/TCP   30m

[root@dell-per740-36 ~]# oc debug node/dell-per740-14.rhts.eng.pek2.redhat.com
Starting pod/dell-per740-14rhtsengpek2redhatcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.73.116.62
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# iptables -n -v -t nat -L OVN-KUBE-EXTERNALIP
Chain OVN-KUBE-EXTERNALIP (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            10.73.116.64         tcp dpt:27018 to:172.30.33.188:27018
sh-4.4# iptables -n -v -t nat -L OVN-KUBE-EXTERNALIP
Chain OVN-KUBE-EXTERNALIP (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            2.2.2.2              tcp dpt:27018 to:172.30.33.188:27018




[root@dell-per740-36 ~]# oc rsh externalip-pod-57f9dd7cfb-4hl58
~ $ curl 2.2.2.2:27018
Customer-Blue Test ExternalIP
[root@dell-per740-36 ~]# ip route add 2.2.2.2/32 via 10.73.116.62 dev em1
[root@dell-per740-36 ~]# curl 2.2.2.2:27018
Customer-Blue Test ExternalIP
[root@dell-per740-36 ~]# curl 10.73.116.64:27018
curl: (7) Failed connect to 10.73.116.64:27018; Connection refused
[root@dell-per740-36 ~]# 
[root@dell-per740-36 ~]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759