Bug 1470003 - oc adm top doesn't work out of the box
oc adm top doesn't work out of the box
Status: VERIFIED
Product: OpenShift Container Platform
Classification: Red Hat
Component: Command Line Interface (Show other bugs)
3.5.0
Unspecified Unspecified
medium Severity medium
: ---
: 3.7.0
Assigned To: Juan Vallejo
Xingxing Xia
: Reopened
Depends On: 1483845
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-12 05:09 EDT by Eduardo Minguez
Modified: 2017-09-11 10:25 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-14 14:09:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eduardo Minguez 2017-07-12 05:09:15 EDT
Description of problem:
"oc adm top" complains when using it without flags:

[edu@bastion ~]$ oc adm top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
[edu@bastion ~]$ oc adm top pod
W0712 09:07:15.849809   16135 top_pod.go:188] Metrics not available for pod openshift-infra/hawkular-cassandra-1-h9mrq, age: 19h2m22.849799868s
error: Metrics not available for pod openshift-infra/hawkular-cassandra-1-h9mrq, age: 19h2m22.849799868s


Version-Release number of selected component (if applicable):
oc v3.5.5.31
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://emingueztst.eastus2.cloudapp.azure.com:8443
openshift v3.5.5.31
kubernetes v1.5.2+43a9be4


How reproducible:
run oc adm top pods or oc adm top nodes

Steps to Reproduce:
1. install metrics
2. run oc adm top pods

Actual results:
Errors complaining about heapster service

Expected results:
Proper pods/nodes usage


Additional info:
It works with the following flags:

$ oc adm top node --heapster-namespace='openshift-infra'  --heapster-scheme="https"
NAME         CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
infranode3   372m         9%        4657Mi          33%       
master3      68m          1%        1923Mi          13%       
node02       43m          1%        1437Mi          5%        
infranode1   145m         3%        4139Mi          29%       
master2      57m          1%        1522Mi          10%       
infranode2   167m         4%        4419Mi          31%       
master1      68m          1%        1726Mi          12%       
node03       38m          0%        1395Mi          4%        
node01       43m          1%        1400Mi          4%        

$ oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme="https" --all-namespaces
NAMESPACE         NAME                                   CPU(cores)   MEMORY(bytes)   
openshift-infra   hawkular-cassandra-1-h9mrq             161m         1423Mi          
logging           logging-fluentd-g5jqw                  8m           92Mi            
logging           logging-es-ops-b44n3gav-1-zkl3r        19m          861Mi           
...
Comment 2 Juan Vallejo 2017-08-14 14:09:03 EDT
So it looks like the default namespace (project) for heapster is "kube-system" [1], however OpenShift metrics are deployed in the `openshift-infra` project [2], which means, as far as I can tell, that you have to specify `--heapster-namespace` every time you invoke this command using the `oc` binary.

You mentioned `oc adm top node` works when specifying the `--heapster-namespace` flag, which looks like the correct way to invoke this command.

@ffranz marking this as NOTABUG. Please re-open if I have missed something about this bug.

1. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/metricsutil/metrics_client.go#L33

2. https://docs.openshift.com/enterprise/3.1/install_config/cluster_metrics.html#cluster-metrics-before-you-begin
Comment 3 Marko Myllynen 2017-08-15 04:52:17 EDT
(In reply to Juan Vallejo from comment #2)
> So it looks like the default namespace (project) for heapster is
> "kube-system" [1], however OpenShift metrics are deployed in the
> `openshift-infra` project [2], which means, as far as I can tell, that you
> have to specify `--heapster-namespace` every time you invoke this command
> using the `oc` binary.
> 
> You mentioned `oc adm top node` works when specifying the
> `--heapster-namespace` flag, which looks like the correct way to invoke this
> command.

What is the point providing defaults that cannot possibly work for anyone?

oc is the OpenShift client which should work on OpenShift regardless of Heapster/Kubernetes or such upstream defaults.

The question here is first and foremost about user experience, there's no reason to force each and every user to type these defaults when the right thing to is to set them by default and make the OOTB / common use case smooth.

Please reconsider fixing this usability issue.

Thanks,
Comment 4 Juan Vallejo 2017-08-15 15:11:08 EDT
Re-purposed linked PRs [1][2] from comment #1 to allow OpenShift-specific default values to be set for `oc adm top node` and `oc adm top pod`.

1. https://github.com/openshift/origin/pull/15764
2. https://github.com/kubernetes/kubernetes/pull/50620
Comment 7 Juan Vallejo 2017-08-21 11:35:56 EDT
Upstream PR [1] has merged. Waiting on Origin [2].

1. https://github.com/kubernetes/kubernetes/pull/50620
2. https://github.com/openshift/origin/pull/15764
Comment 8 Junqi Zhao 2017-08-27 21:19:16 EDT
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1483845
Comment 10 Xingxing Xia 2017-08-28 05:24:03 EDT
Above is verified via cluster-admin.
Found an issue that cluster-reader cannot run `top pod/node`:
$ oadm policy add-cluster-role-to-user cluster-reader xxia
cluster role "cluster-reader" added: "xxia"
$ oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme="https" --all-namespaces
W0828 17:15:23.495289   23961 top_pod.go:190] Metrics not available for pod default/docker-registry-1-xmnqc, age: 8h58m31.495266742s
error: Metrics not available for pod default/docker-registry-1-xmnqc, age: 8h58m31.495266742s
$ oc adm top node --heapster-namespace='openshift-infra'  --heapster-scheme="https"
Error from server (Forbidden): unknown (get services https:heapster:)

Log info:
$ oc adm top pod --heapster-namespace='openshift-infra'  --heapster-scheme="https" --loglevel 6
I0828 17:18:42.793073   23991 loader.go:354] Config loaded from file /home/xxia/.kube/config
I0828 17:18:42.793798   23991 cached_discovery.go:118] returning cached discovery info from /home/xxia/.kube/master/servergroups.json
I0828 17:18:44.177263   23991 round_trippers.go:405] GET https://master:8443/api/v1/namespaces/openshift-infra/services/https:heapster:/proxy/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector= 403 Forbidden in 1383 milliseconds
I0828 17:18:44.489256   23991 round_trippers.go:405] GET https://master/api/v1/namespaces/default/pods 200 OK in 311 milliseconds
W0828 17:18:44.764013   23991 top_pod.go:190] Metrics not available for pod default/docker-registry-1-xmnqc, age: 9h1m52.764002178s
F0828 17:18:44.764134   23991 helpers.go:119] error: Metrics not available for pod default/docker-registry-1-xmnqc, age: 9h1m52.764002178s

Do you think it is a problem?
Comment 11 DeShuai Ma 2017-08-28 22:34:57 EDT
Please try http again.
Comment 12 Xingxing Xia 2017-08-29 03:31:37 EDT
https://bugzilla.redhat.com/show_bug.cgi?id=1486142 is used to track comment 10
Comment 14 Xingxing Xia 2017-09-11 06:24:30 EDT
v3.5 VS v3.7 have 2 releases diff, so cannot assert without test. Today tried to launch 3.5 metrics env, but unfortunately didn't succeed. Will continue check next day
Comment 15 Juan Vallejo 2017-09-11 10:25:57 EDT
@jrosenta Yes, with a 3.7 client, `oc adm top <pod|node>` should work out of the box (with no need to specify openshift-specific metrics defaults)

Note You need to log in before you can comment on or make changes to this bug.