Bug 1925534 - oc does not support specified proxy-url in kubeconfig during login
Summary: oc does not support specified proxy-url in kubeconfig during login
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.6
Hardware: All
OS: All
medium
medium
Target Milestone: ---
: ---
Assignee: Arda Guclu
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-05 13:33 UTC by Jason Kincl
Modified: 2022-08-16 10:46 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-16 10:46:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 751 0 None closed Bug 1925534: Add proxy to oc 2021-07-09 08:48:04 UTC

Description Jason Kincl 2021-02-05 13:33:37 UTC
Description of problem:

Kubernetes 1.19 introduced `proxy-url` in the cluster stanza of a kubeconfig file in client-go. This lets you specify a http proxy for all connections to a specific API server[1][2]. When running `oc login` if `proxy-url` already exists it is not used and it is subsequently removed once login is successful. After obtaining a valid token it is possible to add back the `proxy-url` key and all underlying client-go requests respect the setting (`oc get pods` for example)


[1] https://github.com/kubernetes/client-go/issues/351
[2] https://github.com/kubernetes/kubernetes/pull/81443

Version-Release number of selected component (if applicable):

This was introduced in 1.19+ only so OpenShift 4.6+ is affected


How reproducible:

always


Steps to Reproduce:
1. modify kubeconfig and add `proxy-url`

```
apiVersion: v1
clusters:
- cluster:
    proxy-url: http://127.0.0.1:8080
    server: https://api.testing.local:6443
  name: local-test
```

2. run `oc login` and see that it does not use the proxy
3. run `http_proxy=http://127.0.0.1:8080 oc login` to force the proxy and obtain a token
4. add back `proxy-url` to the cluster configuration in kubeconfig
5. run `oc get pods` to verify that client-go is using the configured proxy-url in kubeconfig

Actual results:


Expected results:


Additional info:

Comment 1 Mike Dame 2021-02-25 20:59:41 UTC
I have not yet had an opportunity to investigate this bug, but will work on it in a future sprint.

Comment 2 Robin Cernin 2021-02-26 02:44:35 UTC
This is not a bug, rather new feature that was added with Kubernetes 1.19+

Every time you run `oc login` or `oc project` kubeconfig will get patched and `proxy-url` gets removed.

For example `oc project` :

~~~
$ cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080

$ oc project default
Already on project "default" on server "https://example.com:6443".

$ cat ~/.kube/config | grep 8080
$ 
~~~

For example `oc login` : 

~~~
$ cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080
$ oc login --token=sha256~{XXX} --server=https://example.com:6443

Logged into "https://example.com:6443" as "kube:admin" using the token provided.

You have access to 58 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".

$ cat ~/.kube/config | grep 8080
$
~~~

As per the upstream PR for the go-client the use-case is where you are managing several OpenShift clusters, some of them does not require proxy at all and some do and there may be different proxies per cluster.

Workaround can be to use the env HTTP_PROXY or HTTPS_PROXY:

~~~
HTTP_PROXY=http://127.0.0.1:8080 oc login --token=sha256~{XXX} --server=https://example.com:6443
~~~

Comment 9 Robin Cernin 2021-06-27 21:32:21 UTC
For QA:

the --proxy cli argument on `oc login` was rejected, so the only improvement we have done here is the proxy doesn't get removed when kubeconfig is updated:


1. oc login --token=<token> --server=<api>:6443
2. oc config set clusters.<api>:6443.proxy-url http://127.0.0.1:8080

Now you should see proxy-url set on your cluster:

- cluster:
    insecure-skip-tls-verify: true
    proxy-url: http://127.0.0.1:8080
    server: https://<api>:6443
  name: <api>:6443


This upstream patch fixed that when we changed projects, the proxy-url was removed:

Without patch:

cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080
./oc project default
Already on project "default" on server "<api>:6443".
cat ~/.kube/config | grep 8080


With patch:

cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080
./oc project default
Already on project "default" on server "<api>:6443".
cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080

Comment 14 Maciej Szulik 2022-05-13 14:20:26 UTC
Arda can you check this one out?

Comment 15 Robin Cernin 2022-05-15 23:32:41 UTC
the patch doesn't change the behaviour when you use login. That required more changes like adding a flag to login command --proxy which we decided not to. Instead it does preserve the proxy when you change projects:

This upstream patch fixed that when we changed projects, the proxy-url was removed:

Without patch:

~~~
cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080
./oc project default
Already on project "default" on server "<api>:6443".
cat ~/.kube/config | grep 8080
~~~

With patch:

~~~
cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080
./oc project default
Already on project "default" on server "<api>:6443".
cat ~/.kube/config | grep 8080
    proxy-url: http://127.0.0.1:8080
~~~

Comment 16 Robin Cernin 2022-05-15 23:34:09 UTC
Note for the login command you should still be able to use environment variables "http_proxy" or "https_proxy" environment variables.

Comment 17 Simon Reber 2022-05-16 05:43:29 UTC
(In reply to Robin Cernin from comment #16)
> Note for the login command you should still be able to use environment
> variables "http_proxy" or "https_proxy" environment variables.
The environment variables do work indeed. But the problem is that when you set a proxy in `kubeconfig` and you run `oc login` the configuration will vanish which is rather ugly. Hence having a way to preserve the proxy configuration set in `kubeconfig` also during `oc login` would be very much appreciated.

Comment 18 Arda Guclu 2022-05-16 14:22:49 UTC
I think, as Robin stated that elegant way preserving proxy configuration in kubeconfig is to add new flag, namely proxy-url, which was decided not to add it. Thus, proposed workaround is a good choice by setting proxy-url via environment variables.

Comment 19 Samuel 2022-05-25 13:29:35 UTC
For context, the customer requesting this has around 30 clusters today.
There is no one way to connect all of their clusters.
Customer is well aware he can use environment variables: it's just not practical.

Also, ... environment variables won't always work.
Collaborators over there would use VSCode, and some kubectl plugin. According to them, VSCode doesn't observe environment variables proxy configuration (or they're doing it wrong?)
Having the proxy-url persisting in kubeconfigs would be very nice here. As they currently edit their configuration after each new oc login. Wich is still less painful than setting and resetting env vars in between commands.

Comment 27 Arda Guclu 2022-08-11 10:58:29 UTC
After investigating the issue further, I have a clear understanding about the problem.

`oc login` command naturally is for writing into kubeconfig rather than using it. Given that new proxy-url flag was rejected. Only way to use `oc login` command with proxy is passing environment variables. I think we have already agreed upon moving forward with this way in the manner of `oc login` usage.

For the real problem filed in this issue which is proxy-url in kubeconfig is cleared, after login command is used, there is no differentiator point to identify user trying to clear up proxy url or setting up new one, or just trying to preserve the current one(like the case you mentioned). Thereby, possible change in oc login command will have possible side effects that we want to avoid.

I totally understand that fully relying on environment variables might not be feasible solution to that. So that I highly recommend using login command such as;

$ export HTTPS_PROXY=https://testproxy ; oc login -u user ; oc config set-cluster ${CLUSTER_NAME} --proxy-url=${HTTP_PROXY}

This will update kubeconfig with the correct proxy-url and it is simply one more command after the login. 

I hope this will solve the issue. Thanks.


Note You need to log in before you can comment on or make changes to this bug.