Bug 1470003

PR: https://github.com/openshift/origin/pull/15764
Upstream PR: https://github.com/kubernetes/kubernetes/pull/50620

So it looks like the default namespace (project) for heapster is "kube-system" [1], however OpenShift metrics are deployed in the `openshift-infra` project [2], which means, as far as I can tell, that you have to specify `--heapster-namespace` every time you invoke this command using the `oc` binary.

You mentioned `oc adm top node` works when specifying the `--heapster-namespace` flag, which looks like the correct way to invoke this command.

@ffranz marking this as NOTABUG. Please re-open if I have missed something about this bug.

1. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/metricsutil/metrics_client.go#L33

2. https://docs.openshift.com/enterprise/3.1/install_config/cluster_metrics.html#cluster-metrics-before-you-begin

(In reply to Juan Vallejo from comment #2)
> So it looks like the default namespace (project) for heapster is
> "kube-system" [1], however OpenShift metrics are deployed in the
> `openshift-infra` project [2], which means, as far as I can tell, that you
> have to specify `--heapster-namespace` every time you invoke this command
> using the `oc` binary.
> 
> You mentioned `oc adm top node` works when specifying the
> `--heapster-namespace` flag, which looks like the correct way to invoke this
> command.

What is the point providing defaults that cannot possibly work for anyone?

oc is the OpenShift client which should work on OpenShift regardless of Heapster/Kubernetes or such upstream defaults.

The question here is first and foremost about user experience, there's no reason to force each and every user to type these defaults when the right thing to is to set them by default and make the OOTB / common use case smooth.

Please reconsider fixing this usability issue.

Thanks,

Re-purposed linked PRs [1][2] from comment #1 to allow OpenShift-specific default values to be set for `oc adm top node` and `oc adm top pod`.

1. https://github.com/openshift/origin/pull/15764
2. https://github.com/kubernetes/kubernetes/pull/50620

Upstream PR [1] has merged. Waiting on Origin [2].

1. https://github.com/kubernetes/kubernetes/pull/50620
2. https://github.com/openshift/origin/pull/15764

Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1483845

Above is verified via cluster-admin.
Found an issue that cluster-reader cannot run `top pod/node`:
$ oadm policy add-cluster-role-to-user cluster-reader xxia
cluster role "cluster-reader" added: "xxia"
$ oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme="https" --all-namespaces
W0828 17:15:23.495289   23961 top_pod.go:190] Metrics not available for pod default/docker-registry-1-xmnqc, age: 8h58m31.495266742s
error: Metrics not available for pod default/docker-registry-1-xmnqc, age: 8h58m31.495266742s
$ oc adm top node --heapster-namespace='openshift-infra'  --heapster-scheme="https"
Error from server (Forbidden): unknown (get services https:heapster:)

Log info:
$ oc adm top pod --heapster-namespace='openshift-infra'  --heapster-scheme="https" --loglevel 6
I0828 17:18:42.793073   23991 loader.go:354] Config loaded from file /home/xxia/.kube/config
I0828 17:18:42.793798   23991 cached_discovery.go:118] returning cached discovery info from /home/xxia/.kube/master/servergroups.json
I0828 17:18:44.177263   23991 round_trippers.go:405] GET https://master:8443/api/v1/namespaces/openshift-infra/services/https:heapster:/proxy/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector= 403 Forbidden in 1383 milliseconds
I0828 17:18:44.489256   23991 round_trippers.go:405] GET https://master/api/v1/namespaces/default/pods 200 OK in 311 milliseconds
W0828 17:18:44.764013   23991 top_pod.go:190] Metrics not available for pod default/docker-registry-1-xmnqc, age: 9h1m52.764002178s
F0828 17:18:44.764134   23991 helpers.go:119] error: Metrics not available for pod default/docker-registry-1-xmnqc, age: 9h1m52.764002178s

Do you think it is a problem?

Please try http again.

https://bugzilla.redhat.com/show_bug.cgi?id=1486142 is used to track comment 10

v3.5 VS v3.7 have 2 releases diff, so cannot assert without test. Today tried to launch 3.5 metrics env, but unfortunately didn't succeed. Will continue check next day

@jrosenta Yes, with a 3.7 client, `oc adm top <pod|node>` should work out of the box (with no need to specify openshift-specific metrics defaults)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

If you're on 3.6 or earlier, you can make these bash functions to help you run metrics from the command line:

ocnodestats ()
{
    oc adm top node --heapster-namespace=openshift-infra --heapster-scheme=https
}

# uses dirty sed hack to ensure headers appear at the top of sorted list
ocpodstats ()
{
    oc adm top pod --heapster-namespace=openshift-infra --heapster-scheme=https --all-namespaces | sed 's/^NA/0A/' | sort | sed 's/^0A/NA/'
}