Description of problem: "oc adm top" complains when using it without flags: [edu@bastion ~]$ oc adm top node Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) [edu@bastion ~]$ oc adm top pod W0712 09:07:15.849809 16135 top_pod.go:188] Metrics not available for pod openshift-infra/hawkular-cassandra-1-h9mrq, age: 19h2m22.849799868s error: Metrics not available for pod openshift-infra/hawkular-cassandra-1-h9mrq, age: 19h2m22.849799868s Version-Release number of selected component (if applicable): oc v3.5.5.31 kubernetes v1.5.2+43a9be4 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://emingueztst.eastus2.cloudapp.azure.com:8443 openshift v3.5.5.31 kubernetes v1.5.2+43a9be4 How reproducible: run oc adm top pods or oc adm top nodes Steps to Reproduce: 1. install metrics 2. run oc adm top pods Actual results: Errors complaining about heapster service Expected results: Proper pods/nodes usage Additional info: It works with the following flags: $ oc adm top node --heapster-namespace='openshift-infra' --heapster-scheme="https" NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% infranode3 372m 9% 4657Mi 33% master3 68m 1% 1923Mi 13% node02 43m 1% 1437Mi 5% infranode1 145m 3% 4139Mi 29% master2 57m 1% 1522Mi 10% infranode2 167m 4% 4419Mi 31% master1 68m 1% 1726Mi 12% node03 38m 0% 1395Mi 4% node01 43m 1% 1400Mi 4% $ oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme="https" --all-namespaces NAMESPACE NAME CPU(cores) MEMORY(bytes) openshift-infra hawkular-cassandra-1-h9mrq 161m 1423Mi logging logging-fluentd-g5jqw 8m 92Mi logging logging-es-ops-b44n3gav-1-zkl3r 19m 861Mi ...
PR: https://github.com/openshift/origin/pull/15764 Upstream PR: https://github.com/kubernetes/kubernetes/pull/50620
So it looks like the default namespace (project) for heapster is "kube-system" [1], however OpenShift metrics are deployed in the `openshift-infra` project [2], which means, as far as I can tell, that you have to specify `--heapster-namespace` every time you invoke this command using the `oc` binary. You mentioned `oc adm top node` works when specifying the `--heapster-namespace` flag, which looks like the correct way to invoke this command. @ffranz marking this as NOTABUG. Please re-open if I have missed something about this bug. 1. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/metricsutil/metrics_client.go#L33 2. https://docs.openshift.com/enterprise/3.1/install_config/cluster_metrics.html#cluster-metrics-before-you-begin
(In reply to Juan Vallejo from comment #2) > So it looks like the default namespace (project) for heapster is > "kube-system" [1], however OpenShift metrics are deployed in the > `openshift-infra` project [2], which means, as far as I can tell, that you > have to specify `--heapster-namespace` every time you invoke this command > using the `oc` binary. > > You mentioned `oc adm top node` works when specifying the > `--heapster-namespace` flag, which looks like the correct way to invoke this > command. What is the point providing defaults that cannot possibly work for anyone? oc is the OpenShift client which should work on OpenShift regardless of Heapster/Kubernetes or such upstream defaults. The question here is first and foremost about user experience, there's no reason to force each and every user to type these defaults when the right thing to is to set them by default and make the OOTB / common use case smooth. Please reconsider fixing this usability issue. Thanks,
Re-purposed linked PRs [1][2] from comment #1 to allow OpenShift-specific default values to be set for `oc adm top node` and `oc adm top pod`. 1. https://github.com/openshift/origin/pull/15764 2. https://github.com/kubernetes/kubernetes/pull/50620
Upstream PR [1] has merged. Waiting on Origin [2]. 1. https://github.com/kubernetes/kubernetes/pull/50620 2. https://github.com/openshift/origin/pull/15764
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1483845
Above is verified via cluster-admin. Found an issue that cluster-reader cannot run `top pod/node`: $ oadm policy add-cluster-role-to-user cluster-reader xxia cluster role "cluster-reader" added: "xxia" $ oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme="https" --all-namespaces W0828 17:15:23.495289 23961 top_pod.go:190] Metrics not available for pod default/docker-registry-1-xmnqc, age: 8h58m31.495266742s error: Metrics not available for pod default/docker-registry-1-xmnqc, age: 8h58m31.495266742s $ oc adm top node --heapster-namespace='openshift-infra' --heapster-scheme="https" Error from server (Forbidden): unknown (get services https:heapster:) Log info: $ oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme="https" --loglevel 6 I0828 17:18:42.793073 23991 loader.go:354] Config loaded from file /home/xxia/.kube/config I0828 17:18:42.793798 23991 cached_discovery.go:118] returning cached discovery info from /home/xxia/.kube/master/servergroups.json I0828 17:18:44.177263 23991 round_trippers.go:405] GET https://master:8443/api/v1/namespaces/openshift-infra/services/https:heapster:/proxy/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector= 403 Forbidden in 1383 milliseconds I0828 17:18:44.489256 23991 round_trippers.go:405] GET https://master/api/v1/namespaces/default/pods 200 OK in 311 milliseconds W0828 17:18:44.764013 23991 top_pod.go:190] Metrics not available for pod default/docker-registry-1-xmnqc, age: 9h1m52.764002178s F0828 17:18:44.764134 23991 helpers.go:119] error: Metrics not available for pod default/docker-registry-1-xmnqc, age: 9h1m52.764002178s Do you think it is a problem?
Please try http again.
https://bugzilla.redhat.com/show_bug.cgi?id=1486142 is used to track comment 10
v3.5 VS v3.7 have 2 releases diff, so cannot assert without test. Today tried to launch 3.5 metrics env, but unfortunately didn't succeed. Will continue check next day
@jrosenta Yes, with a 3.7 client, `oc adm top <pod|node>` should work out of the box (with no need to specify openshift-specific metrics defaults)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188
If you're on 3.6 or earlier, you can make these bash functions to help you run metrics from the command line: ocnodestats () { oc adm top node --heapster-namespace=openshift-infra --heapster-scheme=https } # uses dirty sed hack to ensure headers appear at the top of sorted list ocpodstats () { oc adm top pod --heapster-namespace=openshift-infra --heapster-scheme=https --all-namespaces | sed 's/^NA/0A/' | sort | sed 's/^0A/NA/' }