Bug 1817938

Summary: Some of `oc adm inspect co/xx` failed with error
Product: OpenShift Container Platform Reporter: Maciej Szulik <maszulik>
Component: MonitoringAssignee: Pawel Krupa <pkrupa>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.5CC: alegrand, anpicker, aos-bugs, erooth, jmalde, jokerman, kakkoyun, lcosic, mfojtik, mloibl, pkrupa, surbania, yinzhou
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1817860 Environment:
Last Closed: 2020-08-04 18:07:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1857597    

Description Maciej Szulik 2020-03-27 09:19:34 UTC 
+++ This bug was initially created as a clone of Bug #1817860 +++

Description of problem:
Use `oc adm inspect co/machine-api` met error:  skipping gathering roles.rbac.authorization.k8s.io/cloud-provider-config-reader due to error: roles.rbac.authorization.k8s.io "cloud-provider-config-reader" not found

`oc adm inspect co/cluster-autoscaler` failed with error: 
error: errors ocurred while gathering data:
    [skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers"]

`oc adm inspect co/monitoring` failed with error:
error: errors ocurred while gathering data:
    skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all"


Version-Release number of selected component (if applicable):
[root@dhcp-140-138 ~]# oc version 
Client Version: 4.5.0-202003252116-11cbaa8
Server Version: 4.5.0-0.nightly-2020-03-26-233202


How reproducible:
always

Steps to Reproduce:
1. Create cluster
2. Use command : `oc adm inspect co/machine-api`
3. `oc adm inspect co/cluster-autoscaler`
4. `oc adm inspect co/monitoring`

Actual results:
2. Met error: skipping gathering roles.rbac.authorization.k8s.io/cloud-provider-config-reader due to error: roles.rbac.authorization.k8s.io "cloud-provider-config-reader" not found

3. Met error: 
skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers"

4. Met error:
skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all"

Expected results:
2-4. Should works well



Additional info:

--- Additional comment from Maciej Szulik on 2020-03-27 10:14:53 CET ---

Data gathered by oc adm inspect are coming from cluster operator's .status.relatedObjects so please fill in those BZ for appropriate operator owners.
I'm splitting this into 3 distinct and assigning them to appropriate teams.
Comment 1 Maciej Szulik 2020-03-27 09:20:32 UTC 
This bug goes to monitoring team, make sure that your .status.relatedObjects reflect reality and don't produce below error:

`oc adm inspect co/monitoring` failed with error:
error: errors ocurred while gathering data:
    skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all"
Comment 2 Junqi Zhao 2020-03-27 09:58:32 UTC 
FYI, the full log
# oc adm inspect co/monitoring
Gathering data for ns/openshift-monitoring...
E0327 05:49:03.150150    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod d069e096737d080f5fb1f11422b829d1d2ae2dabc3a058436c54e3037ca7ee6d, uid : exit status 1: 2020/03/27 09:55:39 socat[1368493] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.198606    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod d069e096737d080f5fb1f11422b829d1d2ae2dabc3a058436c54e3037ca7ee6d, uid : exit status 1: 2020/03/27 09:55:40 socat[1368522] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.243373    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod d069e096737d080f5fb1f11422b829d1d2ae2dabc3a058436c54e3037ca7ee6d, uid : exit status 1: 2020/03/27 09:55:40 socat[1368529] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.294653    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod d069e096737d080f5fb1f11422b829d1d2ae2dabc3a058436c54e3037ca7ee6d, uid : exit status 1: 2020/03/27 09:55:40 socat[1368542] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.740256    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod cb77d273bf5b679c232f7ecebf60af856066274d38cc791e3bf6f37c1867b2d6, uid : exit status 1: 2020/03/27 09:55:40 socat[631152] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.781577    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod cb77d273bf5b679c232f7ecebf60af856066274d38cc791e3bf6f37c1867b2d6, uid : exit status 1: 2020/03/27 09:55:40 socat[631159] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.823621    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod cb77d273bf5b679c232f7ecebf60af856066274d38cc791e3bf6f37c1867b2d6, uid : exit status 1: 2020/03/27 09:55:40 socat[631167] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:03.866011    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod cb77d273bf5b679c232f7ecebf60af856066274d38cc791e3bf6f37c1867b2d6, uid : exit status 1: 2020/03/27 09:55:41 socat[631174] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:05.346392    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod f3f23e33298e154ffdb3b6bbfa82808f5fe2c6b3552fe0c53605060490a819e3, uid : exit status 1: 2020/03/27 09:55:41 socat[934917] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:05.398731    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod f3f23e33298e154ffdb3b6bbfa82808f5fe2c6b3552fe0c53605060490a819e3, uid : exit status 1: 2020/03/27 09:55:41 socat[934925] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:05.452378    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod f3f23e33298e154ffdb3b6bbfa82808f5fe2c6b3552fe0c53605060490a819e3, uid : exit status 1: 2020/03/27 09:55:41 socat[934932] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0327 05:49:05.502815    1462 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod f3f23e33298e154ffdb3b6bbfa82808f5fe2c6b3552fe0c53605060490a819e3, uid : exit status 1: 2020/03/27 09:55:42 socat[934939] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
Wrote inspect data to inspect.local.1720868577461461871.
error: errors ocurred while gathering data:
    skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all"
Comment 6 Junqi Zhao 2020-03-30 02:41:27 UTC 
Tested with 4.5.0-0.nightly-2020-03-29-195504, 
# oc adm inspect co/monitoring
Gathering data for ns/openshift-monitoring...
E0329 20:46:36.483927    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 20c889a5556238a64b96322363ee1d5dbb5d90ec136889c55bc983ccb0aff294, uid : exit status 1: 2020/03/30 00:53:22 socat[132530] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:37.500473    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 20c889a5556238a64b96322363ee1d5dbb5d90ec136889c55bc983ccb0aff294, uid : exit status 1: 2020/03/30 00:53:23 socat[132540] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:38.520086    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 20c889a5556238a64b96322363ee1d5dbb5d90ec136889c55bc983ccb0aff294, uid : exit status 1: 2020/03/30 00:53:24 socat[132548] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:39.535125    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 20c889a5556238a64b96322363ee1d5dbb5d90ec136889c55bc983ccb0aff294, uid : exit status 1: 2020/03/30 00:53:25 socat[132584] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:48.718537    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod e3ddaca451689fd8d28f891bab07ea915c58f45512b4423911982cae18e294ad, uid : exit status 1: 2020/03/30 00:53:35 socat[181672] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:49.724857    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod e3ddaca451689fd8d28f891bab07ea915c58f45512b4423911982cae18e294ad, uid : exit status 1: 2020/03/30 00:53:36 socat[181717] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:50.747679    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod e3ddaca451689fd8d28f891bab07ea915c58f45512b4423911982cae18e294ad, uid : exit status 1: 2020/03/30 00:53:37 socat[181799] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:46:51.759058    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod e3ddaca451689fd8d28f891bab07ea915c58f45512b4423911982cae18e294ad, uid : exit status 1: 2020/03/30 00:53:38 socat[181870] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:47:00.852552    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 58fda0e5df4f56f9367f7a0cb1fc0cf602335ef0d2a36dd52c9afa1256653d3a, uid : exit status 1: 2020/03/30 00:53:47 socat[279428] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:47:01.862592    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 58fda0e5df4f56f9367f7a0cb1fc0cf602335ef0d2a36dd52c9afa1256653d3a, uid : exit status 1: 2020/03/30 00:53:48 socat[279499] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:47:02.875127    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 58fda0e5df4f56f9367f7a0cb1fc0cf602335ef0d2a36dd52c9afa1256653d3a, uid : exit status 1: 2020/03/30 00:53:49 socat[279574] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:47:03.892481    4351 portforward.go:400] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 58fda0e5df4f56f9367f7a0cb1fc0cf602335ef0d2a36dd52c9afa1256653d3a, uid : exit status 1: 2020/03/30 00:53:50 socat[279672] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0329 20:47:17.344507    4351 portforward.go:385] error copying from local connection to remote stream: EOF
Wrote inspect data to inspect.local.5141615055751461993.

Although there is portforward error, it does not affect the function, move to VERIFIED, feel free to reopen it if you think it is a bug
Comment 7 Maciej Szulik 2020-06-15 14:07:01 UTC 
*** Bug 1846196 has been marked as a duplicate of this bug. ***
Comment 11 errata-xmlrpc 2020-08-04 18:07:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5 image release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409