Bug 1907987

+++ This bug was initially created as a clone of Bug #1896880 +++

Description of problem:
For externalIP policy testing, if policy=null is set, then creating a Service with spec.ExternalIPs[] set to any IP address is allowed, but below testing results show setting policy=null from default policy={} not take effect 

Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2020-11-11-102623


How reproducible:
Always

Steps to Reproduce:
Test step one: cluster using default policy={}
[weliang@weliang ~]$ oc login -u system:admin
[weliang@weliang ~]$ oc get network.config cluster -o json
"spec": {
        "clusterNetwork": [
            {
                "cidr": "10.128.0.0/14",
                "hostPrefix": 23
            }
        ],
        "externalIP": {
            "policy": {}
        },
        "networkType": "OpenShiftSDN",
        "serviceNetwork": [
            "172.30.0.0/16"
        ]
    },
[weliang@weliang ~]$ oc login -u testuser-0 -p pp2ojnhpN9Jo
[weliang@weliang ~]$ oc new-project test
[weliang@weliang ~]$ oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/PublishingServices/external-ip.yaml
pod/hello-pod-1 created
Error from server (Forbidden): services "hello-service1" is forbidden: spec.externalIPs: Forbidden: externalIPs have been disabled

Test step two: set policy=null
[weliang@weliang ~]$ oc login -u system:admin
[weliang@weliang ~]$ oc get network.config cluster -o json
"spec": {
        "clusterNetwork": [
            {
                "cidr": "10.128.0.0/14",
                "hostPrefix": 23
            }
        ],
        "externalIP": {},
        "networkType": "OpenShiftSDN",
        "serviceNetwork": [
            "172.30.0.0/16"
        ]
    },
[weliang@weliang ~]$ oc login -u testuser-0 -p pp2ojnhpN9Jo
[weliang@weliang ~]$ oc delete all --all
[weliang@weliang ~]$ oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/PublishingServices/external-ip.yaml
pod/hello-pod-1 created
Error from server (Forbidden): services "hello-service1" is forbidden: spec.externalIPs: Forbidden: externalIPs have been disabled

Wait for 5 minutes
[weliang@weliang ~]$ oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/PublishingServices/external-ip.yaml
pod/hello-pod-1 created
Error from server (Forbidden): services "hello-service1" is forbidden: spec.externalIPs: Forbidden: externalIPs have been disabled



Actual results:
With setting policy=null, still get Error from server (Forbidden): services "hello-service1" is forbidden: spec.externalIPs: Forbidden: externalIPs have been disabled

Expected results:
If policy=null is set, then creating a Service with spec.ExternalIPs[] set to any IP address is allowed.

Additional info:

--- Additional comment from Shubhag Saxena on 2020-12-01 20:12:58 UTC ---

Hi team,

Can you please provide an update on this bug, cu is waiting for so long.

--- Additional comment from Casey Callendrello on 2020-12-02 11:59:58 UTC ---

I don't think this is a bug; the documentation says that an empty or null policy is block-all.


If they would like to allow all external ips, then all they need to do is:

externalIP:
  policy:
    allowedCIDRs: [ "0.0.0.0/0" ]

--- Additional comment from Weibin Liang on 2020-12-02 15:24:41 UTC ---

Hi Casey,

Here is what v4.6 doc mentioned:

https://docs.openshift.com/container-platform/4.6/networking/configuring_ingress_cluster_traffic/configuring-externalip.html

When configuring policy restrictions, the following rules apply:

If policy={} is set, then creating a Service with spec.ExternalIPs[] set will fail. This is the default for OpenShift Container Platform.

If policy=null is set, then creating a Service with spec.ExternalIPs[] set to any IP address is allowed.

--- Additional comment from Shubhag Saxena on 2020-12-02 15:50:42 UTC ---

Hi @Casey,

I agree with what @Weibian said.

Actually when I reproduced this issue on my end I found that, sometimes "for project admin user "policy: null" works and sometimes not (see below) after waiting for 300 secs. While this is not the case with cluster:admin user and everything works fine for it (mentioned in previous bug https://bugzilla.redhat.com/show_bug.cgi?id=1793099#c10)".
 
So I reported my observations, where @Weibian also tried and filed this bug where he found "setting policy=null from default policy={} not take effect". 

So there is absolutely something wrong with the working of "policy: null".

--- Additional comment from Shubhag Saxena on 2020-12-10 21:22:59 UTC ---

Hi @Casey, 

I got one more case 02823234, where cu pointed out that acc. to our docs, "By default, only a user with cluster-admin privileges can create a Service with spec.externalIPs[] set to IP addresses defined within an external IP address block". 
 
@Weibin I think because of this reason you might be getting the errors for project admin `testuser` but on the other hand, I am able to set Ip for svc by project admin `new user` as mentioned in previous bug https://bugzilla.redhat.com/show_bug.cgi?id=1793099#c10 & its was working. Also, the customer is able to manipulate a service and assign an external IP as a non-cluster admin.

Correction is needed in doc if project admin user is allowed to do this configurations or otherwise, if doc is correct then project admin user shouldn't be allowed to this as cu informed & on my end.

Or let me know if I misunderstood doc's statement.

--- Additional comment from Casey Callendrello on 2020-12-14 12:52:14 UTC ---

Hmm, it could be that the docs are wrong. The way it should be is:

- cluster-admin can always set external IP
- non-cluster-admin can only set external IP if inside a range in policy
- no policy is equivalent to external ips disabled.

So it seems there is a mistake in the docs.

Assigning to Jason to fix the documentation.

--- Additional comment from Weibin Liang on 2020-12-14 15:52:29 UTC ---

(In reply to Shubhag Saxena from comment #5)
> Hi @Casey, 
> 
> I got one more case 02823234, where cu pointed out that acc. to our docs,
> "By default, only a user with cluster-admin privileges can create a Service
> with spec.externalIPs[] set to IP addresses defined within an external IP
> address block". 
>  
> @Weibin I think because of this reason you might be getting the errors for
> project admin `testuser` but on the other hand, I am able to set Ip for svc
> by project admin `new user` as mentioned in previous bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1793099#c10 & its was working.
> Also, the customer is able to manipulate a service and assign an external IP
> as a non-cluster admin.
> 
> Correction is needed in doc if project admin user is allowed to do this
> configurations or otherwise, if doc is correct then project admin user
> shouldn't be allowed to this as cu informed & on my end.
> 
> Or let me know if I misunderstood doc's statement.


@shsaxena
The statement in our docs about "By default, only a user with cluster-admin privileges can create a Service with spec.externalIPs[] set to IP addresses defined within an external IP address block" is wrong, as Casey mentioned in comment6, non-cluster-admin still can set external IP if inside a range in policy. Let me open another bug to fix this doc error.

--- Additional comment from Weibin Liang on 2020-12-14 16:23:38 UTC ---

(In reply to Casey Callendrello from comment #6)
> Hmm, it could be that the docs are wrong. The way it should be is:
> 
> - cluster-admin can always set external IP
> - non-cluster-admin can only set external IP if inside a range in policy
> - no policy is equivalent to external ips disabled.
> 
> So it seems there is a mistake in the docs.
> 
> Assigning to Jason to fix the documentation.


Bug https://bugzilla.redhat.com/show_bug.cgi?id=1907505 is opened for the doc error mentioned in comment 5.


We still need figure out "Setting policy=null from default policy={} not take effect" is a feature bug or doc bug.

--- Additional comment from Shubhag Saxena on 2020-12-15 15:47:54 UTC ---

(In reply to Weibin Liang from comment #8)
> (In reply to Casey Callendrello from comment #6)
> > Hmm, it could be that the docs are wrong. The way it should be is:
> > 
> > - cluster-admin can always set external IP
> > - non-cluster-admin can only set external IP if inside a range in policy
> > - no policy is equivalent to external ips disabled.
> > 
> > So it seems there is a mistake in the docs.
> > 
> > Assigning to Jason to fix the documentation.
> 
> 
> Bug https://bugzilla.redhat.com/show_bug.cgi?id=1907505 is opened for the
> doc error mentioned in comment 5.
> 
> 
> We still need figure out "Setting policy=null from default policy={} not
> take effect" is a feature bug or doc bug.

Hi team, 

Thanks for confirming, I will keep a track of this as well as a doc bug. 

> We still need figure out "Setting policy=null from default policy={} not take effect" is a feature bug or doc bug.
Yes, this needs to be solved as cu is waiting for update since long.