Bug 1857202 - oc client 4.5 unauthorized when using merged kubeconfig [NEEDINFO]
Summary: oc client 4.5 unauthorized when using merged kubeconfig
Keywords:
Status: CLOSED DUPLICATE of bug 1868384
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.5
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.7.0
Assignee: Maciej Szulik
QA Contact: zhou ying
URL:
Whiteboard: LifecycleStale
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-15 12:59 UTC by Noam Manos
Modified: 2021-01-21 12:00 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-21 12:00:15 UTC
Target Upstream Version:
Embargoed:
mfojtik: needinfo?


Attachments (Terms of Use)

Description Noam Manos 2020-07-15 12:59:26 UTC
Description of problem:
Using OC client 4.5.1 with a merged KUBECONFIG for multiple clusters, will result in error: You must be logged in to the server (Unauthorized).

Version-Release number of selected component (if applicable):
OC Client 4.5.1

How reproducible:
Always

Steps to Reproduce:
1) Download OC Client 4.5.1
2) Have a "merged" KUBECONFIG - pointing to two clusters, like:
export KUBECONFIG="/path/to/kubeconfig_1:/path/to/kubeconfig_2"
3) Run: oc status

Actual results:


$ export KUBECONFIG="/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig" 
                                                              
$ /mnt/skynet-data/ocp-install/GOBIN/oc version
Client Version: 4.5.1
Server Version: 4.5.1
Kubernetes Version: v1.18.3+8b0a82f

$ export KUBECONFIG="/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig"

$ /mnt/skynet-data/ocp-install/GOBIN/oc version
Client Version: 4.5.1
Server Version: 4.4.3
Kubernetes Version: v1.17.1

$ export KUBECONFIG="/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig:/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig"

$ /mnt/skynet-data/ocp-install/GOBIN/oc config view  
        
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://api.default-cl1.devcluster.openshift.com:6443
  name: default-cl1
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://api.nmanos-cluster-a.devcluster.openshift.com:6443
  name: nmanos-cluster-a
contexts:
- context:
    cluster: nmanos-cluster-a
    namespace: test-submariner
    user: admin
  name: nmanos-cluster-a
- context:
    cluster: default-cl1
    namespace: test-submariner
    user: admin
  name: nmanos-cluster-b
current-context: nmanos-cluster-a
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED


$ /mnt/skynet-data/ocp-install/GOBIN/oc version

Client Version: 4.5.1
error: You must be logged in to the server (Unauthorized)

$ /mnt/skynet-data/ocp-install/GOBIN/oc status 

error: You must be logged in to the server (Unauthorized)


Expected results (in OC Client 4.2 for example):

$ export KUBECONFIG="/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig:/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig"

$ oc version

Client Version: version.Info{Major:"", Minor:"", GitVersion:"v4.2.0-alpha.0-4-g38b0f09", GitCommit:"38b0f09", GitTreeState:"clean", BuildDate:"2019-08-12T19:05:43Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3+8b0a82f", GitCommit:"8b0a82f", GitTreeState:"clean", BuildDate:"2020-07-10T05:34:00Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
OpenShift Version: 4.5.1

$ oc status 

In project test-submariner on server https://api.nmanos-cluster-a.devcluster.openshift.com:6443


Additional info: 
-

Comment 1 Maciej Szulik 2020-07-16 10:02:42 UTC
I just verified this with an earlier version of oc 4.5.0-rc.6 and this works as expected.
I'd suggest checking if each of the on their own are working as expected, first and then try merging them.

Comment 2 Noam Manos 2020-07-17 11:38:49 UTC
It is reproduced on my env. 
Please try to check it when one of the clusters is using older OCP version:

On Cluster A:

$ export KUBECONFIG="/mnt/skynet-data/nmanos-cluster-a/auth/kubeconfig"

$ oc version
Client Version: 4.5.2
Server Version: 4.5.2
Kubernetes Version: v1.18.3+b74c5ed


On Cluster B:

$ export KUBECONFIG="/mnt/skynet-data/nmanos-cluster-b/auth/kubeconfig"

$ oc version
Client Version: 4.5.2
Server Version: 4.4.3
Kubernetes Version: v1.17.1


$ export KUBECONFIG="/mnt/skynet-data/nmanos-cluster-a/auth/kubeconfig:/mnt/skynet-data/nmanos-cluster-b/auth/kubeconfig"

$ oc get all -A
error: You must be logged in to the server (Unauthorized)

Comment 3 Maciej Szulik 2020-07-20 08:47:10 UTC
The cluster version has nothing to do with your problem, I've verified oc binaries all the way back to 4.3 and all worked w/o any problems.
Please ensure that you can invoke oc get all with each of the KUBECONFIGs and only then verify merging. From my tests all worked just fine.

Comment 4 Noam Manos 2020-07-26 08:27:59 UTC
### Using OC client 4.2 :

$ oc version

  Client Version: version.Info{Major:"", Minor:"", GitVersion:"v4.2.0-alpha.0-4-g38b0f09", GitCommit:"38b0f09", GitTreeState:"clean", BuildDate:"2019-08-12T19:05:43Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2020-06-08T20:03:02Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

### On 1st Cluster - OC 4.2 works as expected:

$ export KUBECONFIG=/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig
$ oc status

  In project test-submariner on server https://api.default-cl1.devcluster.openshift.com:6443

  svc/nginx-cl-b - 100.96.22.141:8080
    deployment/nginx-cl-b deploys nginxinc/nginx-unprivileged:stable-alpine
      deployment #1 running for 2 hours - 1 pod

  1 info identified, use 'oc status --suggest' to see details.
  

### On 2nd Cluster - OC 4.2 works as expected:

$ export KUBECONFIG=/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig
$ oc status

  In project test-submariner on server https://api.nmanos-cluster-a.devcluster.openshift.com:6443

  pod/netshoot-cl-a runs nicolaka/netshoot

  2 infos identified, use 'oc status --suggest' to see details.


### With MERGED Kubeconfigs - OC 4.2 works:
# Note that it retrieves 1st Cluster status only, so it's not the current bug issue (but might be related).

$ export KUBECONFIG=/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig:/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig
$ oc status

  In project test-submariner on server https://api.default-cl1.devcluster.openshift.com:6443

  svc/nginx-cl-b - 100.96.22.141:8080
    deployment/nginx-cl-b deploys nginxinc/nginx-unprivileged:stable-alpine
      deployment #1 running for 2 hours - 1 pod

  1 info identified, use 'oc status --suggest' to see details.


### Now using OC client 4.5:

$ OC45=/mnt/skynet-data/ocp-install/GOBIN/oc

### On 1st Cluster - OC 4.5 works as expected:

$ export KUBECONFIG=/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig

$ $OC45 version

  Client Version: 4.5.3
  Server Version: 4.4.3
  Kubernetes Version: v1.17.1

$ $OC45 status

  In project test-submariner on server https://api.default-cl1.devcluster.openshift.com:6443

  svc/nginx-cl-b - 100.96.22.141:8080
    deployment/nginx-cl-b deploys nginxinc/nginx-unprivileged:stable-alpine
      deployment #1 running for 3 hours - 1 pod

  1 info identified, use 'oc status --suggest' to see details.
  

### On 2nd Cluster - OC 4.5 works as expected:

$ export KUBECONFIG=/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig 

$ $OC45 version

  Client Version: 4.5.3
  Server Version: 4.5.3
  Kubernetes Version: v1.18.3+3107688

$ $OC45 status

  In project test-submariner on server https://api.nmanos-cluster-a.devcluster.openshift.com:6443

  pod/netshoot-cl-a runs nicolaka/netshoot

  You have no services, deployment configs, or build configs.
  Run 'oc new-app' to create an application.
  

### But with MERGED Kubeconfigs - OC 4.5 fails on "Unauthorized" error:

$ export KUBECONFIG=/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig:/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig

$ $OC45 status

  error: You must be logged in to the server (Unauthorized)

$ $OC45 version

  Client Version: 4.5.3
  error: You must be logged in to the server (Unauthorized)


### Trying with verbose command - see how curl to the API returned error 401:

bash-4.2$ $OC45 -v=9 version
I0726 11:15:19.005285   54189 loader.go:375] Config loaded from file:  /mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig
I0726 11:15:19.007042   54189 loader.go:375] Config loaded from file:  /mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig
I0726 11:15:19.008893   54189 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/openshift (linux/amd64) kubernetes/b66f2d3" 'https://api.nmanos-cluster-a.devcluster.openshift.com:6443/version?timeout=32s'
I0726 11:15:19.087203   54189 round_trippers.go:443] GET https://api.nmanos-cluster-a.devcluster.openshift.com:6443/version?timeout=32s 401 Unauthorized in 78 milliseconds
I0726 11:15:19.087236   54189 round_trippers.go:449] Response Headers:
I0726 11:15:19.087242   54189 round_trippers.go:452]     Audit-Id: 9fd55163-9229-427a-b892-8803efe7a6e3
I0726 11:15:19.087246   54189 round_trippers.go:452]     Cache-Control: no-cache, private
I0726 11:15:19.087250   54189 round_trippers.go:452]     Content-Type: application/json
I0726 11:15:19.087254   54189 round_trippers.go:452]     Content-Length: 129
I0726 11:15:19.087258   54189 round_trippers.go:452]     Date: Sun, 26 Jul 2020 08:15:34 GMT
I0726 11:15:19.096685   54189 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0726 11:15:19.102904   54189 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/openshift (linux/amd64) kubernetes/b66f2d3" 'https://api.nmanos-cluster-a.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusteroperators/openshift-apiserver'
I0726 11:15:19.122818   54189 round_trippers.go:443] GET https://api.nmanos-cluster-a.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusteroperators/openshift-apiserver 401 Unauthorized in 19 milliseconds
I0726 11:15:19.122863   54189 round_trippers.go:449] Response Headers:
I0726 11:15:19.122871   54189 round_trippers.go:452]     Content-Type: application/json
I0726 11:15:19.122879   54189 round_trippers.go:452]     Content-Length: 129
I0726 11:15:19.122885   54189 round_trippers.go:452]     Date: Sun, 26 Jul 2020 08:15:34 GMT
I0726 11:15:19.122893   54189 round_trippers.go:452]     Audit-Id: a8d1d3bc-0c41-4a11-ad05-a9467542bd9c
I0726 11:15:19.122900   54189 round_trippers.go:452]     Cache-Control: no-cache, private
I0726 11:15:19.122956   54189 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
Client Version: 4.5.3
I0726 11:15:19.124215   54189 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}]
F0726 11:15:19.124273   54189 helpers.go:115] error: You must be logged in to the server (Unauthorized)

Comment 5 Maciej Szulik 2020-08-21 13:48:57 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 6 Michal Fojtik 2020-08-25 08:28:41 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

Comment 7 Noam Manos 2020-08-26 16:13:28 UTC
On my env this is still reproduced:

export KUBECONFIG="/mnt/skynet-data/ocp-install/nmanos-cluster-a/auth/kubeconfig:/mnt/skynet-data/ocp-install/ocpup/.config/cl1/auth/kubeconfig"
oc version

Client Version: 4.5.7
error: You must be logged in to the server (Unauthorized)

Let me know if you need more information.

Comment 8 Michal Fojtik 2020-08-26 16:59:30 UTC
The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 9 Maciej Szulik 2020-09-11 11:30:02 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 10 Michal Fojtik 2020-09-25 17:02:56 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 11 Maciej Szulik 2021-01-21 12:00:15 UTC

*** This bug has been marked as a duplicate of bug 1868384 ***


Note You need to log in before you can comment on or make changes to this bug.