Red Hat Bugzilla – Bug 1610669
oc adm top node can not get metrics info after metrics-server deployed
Last modified: 2018-10-11 03:23:51 EDT
Description of problem: oc adm top node can not get metrics info after metrics-server deployed Version-Release number of selected component (if applicable): [root@qe-weinliu-311-master-etcd-1 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) [root@qe-weinliu-311-master-etcd-1 ~]# oc version oc v3.11.0-0.10.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-weinliu-311-master-etcd-1:8443 openshift v3.11.0-0.10.0 kubernetes v1.11.0+d4cacc0 [nathan@localhost openshift-ansible]$ rpm -qa|grep ansible-2.6.2-1.el7.ans.noarch ansible-2.6.2-1.el7.ans.noarch [nathan@localhost openshift-ansible]$ git branch * (HEAD detached at openshift-ansible-3.11.0-0.9.0) master release-3.10 release-3.9 How reproducible: always Steps to Reproduce: 1. Update qe-inventory-host-file to include metrics install parameters: openshift_metrics_install_metrics=True 2. Install metrics by $ ansible-playbook -i qe-inventory-host-file playbooks/openshift-metrics/config.yml 3. Check metris-server pods STATUS [root@qe-weinliu-311-master-etcd-1 ~]# oc get pod -n openshift-monitoring NAME READY STATUS RESTARTS AGE metrics-server-7675bf5d75-qr69d 1/1 Running 0 11m 4.Check node status by oc adm top node Actual results: [root@qe-weinliu-311-master-etcd-1 ~]# oc adm top node panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: tabwriter: panic during Flush [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xfd4f8d] goroutine 1 [running]: text/tabwriter.handlePanic(0xc4215c2d88, 0x2bc6f23, 0x5) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:458 +0x111 panic(0x27a8ce0, 0x47321e0) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/runtime/panic.go:502 +0x229 text/tabwriter.(*Writer).write0(0xc420956c00, 0xc420b12140, 0x4, 0x13c) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:230 +0x2d text/tabwriter.(*Writer).writeLines(0xc420956c00, 0x0, 0x0, 0x4, 0x20) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:297 +0x16c text/tabwriter.(*Writer).format(0xc420956c00, 0x0, 0x0, 0x4, 0x5) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:385 +0x2ac text/tabwriter.(*Writer).format(0xc420956c00, 0x0, 0x4, 0x4, 0x0) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:379 +0x1b4 text/tabwriter.(*Writer).format(0xc420956c00, 0x0, 0x4, 0x4, 0x3) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:379 +0x1b4 text/tabwriter.(*Writer).format(0xc420956c00, 0x0, 0x4, 0x4, 0x2) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:379 +0x1b4 text/tabwriter.(*Writer).format(0xc420956c00, 0x0, 0x4, 0x4, 0x1) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:379 +0x1b4 text/tabwriter.(*Writer).format(0xc420956c00, 0x0, 0x4, 0x5, 0x5) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:379 +0x1b4 text/tabwriter.(*Writer).flush(0xc420956c00, 0x0, 0x0) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:484 +0x156 text/tabwriter.(*Writer).Flush(0xc420956c00, 0xc42004c6c0, 0xc420a32000) /opt/rh/go-toolset-7/root/usr/lib/go-toolset-7-golang/src/text/tabwriter/tabwriter.go:467 +0x2b github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/metricsutil.(*TopCmdPrinter).PrintNodeMetrics(0xc42119b910, 0xc4211a0500, 0x3, 0x4, 0xc4215c30b0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/metricsutil/metrics_printer.go:80 +0x3ab github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.TopNodeOptions.RunTopNode(0x0, 0x0, 0x0, 0x0, 0x3158f00, 0xc42119a920, 0x2bdfc02, 0xf, 0x2bce13c, 0x8, ...) /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/top_node.go:219 +0x49d github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdTopNode.func1(0xc421432000, 0x49d61a8, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/top_node.go:114 +0x165 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc421432000, 0x49d61a8, 0x0, 0x0, 0xc421432000, 0x49d61a8) /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2c1 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4212cfb80, 0xc42000e010, 0xc42000e020, 0xc4212cfb80) /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x30a github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4212cfb80, 0x2, 0xc4212cfb80) /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b main.main() /builddir/build/BUILD/atomic-openshift-git-0.766dbc4/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:49 +0x365 Expected results: oc adm top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% qe-weinliu-gce-master-etcd-1 305m 30% 2131Mi 62% qe-weinliu-gce-node-1 63m 6% 1017Mi 29% qe-weinliu-gce-node-registry-router-1 60m 6% 1006Mi 29% Additional info: [root@qe-weinliu-311-master-etcd-1 ~]# oc get --raw /apis/metrics.k8s.io/v1beta1/nodes?pretty=true { "kind": "NodeMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes" }, "items": [ { "metadata": { "name": "qe-weinliu-311-master-etcd-1", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/qe-weinliu-311-master-etcd-1", "creationTimestamp": "2018-07-31T09:13:52Z" }, "timestamp": "2018-07-31T09:13:30Z", "window": "1m0s", "usage": { "cpu": "409m", "memory": "3215900Ki" } }, { "metadata": { "name": "qe-weinliu-311-node-1", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/qe-weinliu-311-node-1", "creationTimestamp": "2018-07-31T09:13:52Z" }, "timestamp": "2018-07-31T09:13:30Z", "window": "1m0s", "usage": { "cpu": "89m", "memory": "1792784Ki" } }, { "metadata": { "name": "qe-weinliu-311-node-registry-router-1", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/qe-weinliu-311-node-registry-router-1", "creationTimestamp": "2018-07-31T09:13:52Z" }, "timestamp": "2018-07-31T09:13:30Z", "window": "1m0s", "usage": { "cpu": "104m", "memory": "1804444Ki" } } ] }[root@qe-weinliu-311-master-etcd-1 ~]# [root@qe-weinliu-311-master-etcd-1 ~]# oc describe apiservices v1beta1.metrics.k8s.io Name: v1beta1.metrics.k8s.io Namespace: Labels: kubernetes.io/cluster-service=true metrics-server-infra=support Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true","metri... API Version: apiregistration.k8s.io/v1 Kind: APIService Metadata: Creation Timestamp: 2018-07-31T08:59:49Z Resource Version: 20286 Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io UID: 1468952a-94a0-11e8-b194-42010af00009 Spec: Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1akNDQWM2Z0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFrTVNJd0lBWURWUVFEREJsdFpYUnkKYVdOekxYTnBaMjVsY2tBeE5UTXpNREkzTlRBNU1CNFhEVEU0TURjek1UQTROVGd6TUZvWERUSXpNRGN6TURBNApOVGd6TVZvd0pERWlNQ0FHQTFVRUF3d1piV1YwY21samN5MXphV2R1WlhKQU1UVXpNekF5TnpVd09UQ0NBU0l3CkRRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFKKzZHQkRBaU9JYldCRkRNSTFMeDM3cVpnMUsKUHhtNXFjUXA2M3YrY2dNbmZvR3F1dEFXcWQ5aHRualV3bHgrZ1B1c0U1QU9vYU9JZE9vaW0wNUNiYklMVTVJTApHRjJ5YjRqWkw0MmM3WXZERlZQZFlqRzBlbG9yUnJSeUVFZDc1Uk5wcjk1RTNHdzk3N1MrdGhWMndWOWxLZmlrCndGNmh1ZEZ4T0U3dWFRb0dPaVlTMlVRUnV6YmdjMmlqV1dOOGVNbHFHdEIvVHdpMUI4MmtXbVVBR0FpQVhFUGIKTS9RRDNIYzNQMHhFWHVKcy9HWUprelBzeFdaQnlpUTNCWnlzd00wTDBMYm4yNFplNjBxdXU2QnhTZHFjbnZyTwprbmx2dVpQaDRNNTJ3WHU3VEYvMDl3cVU4U0JMQ09QT1BLV3BYNGJ0WmZ4K3RuNy9nY0YxaDZmQ3B4OENBd0VBCkFhTWpNQ0V3RGdZRFZSMFBBUUgvQkFRREFnS2tNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y04KQVFFTEJRQURnZ0VCQUZqSUkxQVhzV1cxbG1seVNrWjJ0T2k5U3k4MmZMRE5YdWNNV2VOZlF0RkNBWGxuV21hMgpSZEROME52WWJxN3VQMDVYeTJSTUdtK3VGTzEwb3VyWWhRbmxuNFkwaTcvajhkem9XRDIzdkZ0N1JDRFliWnowCmZydXF6RGJCbWY1R2VvOWgxNDNwS2RYdlR0aThxQmVoTkpsYXo5MkFCZVU5aktPbkFHUGVTNHdSMkJIS09yNGQKNkgvLzZVS3NHL0ZaT2ZhMW51MzFjSW9NTW02Z0RmWGJUSmJwc1BOK1FNTzYydC9TMVV0dkhYcEI3Q1NXM0ZhTwo0ZVMzUmUyRlFSY1hMSGs3TEZjeGJPQXE2QlRJNUVNU3lFUU5ZNHVjbUFxeHFtaEE3LzRVa3BZbDRtZ3YxWGJaCktIM28xYXFwRHlaOStnem54MDFETytiaFBuYkMzbGlUZldrPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== Group: metrics.k8s.io Group Priority Minimum: 100 Service: Name: metrics-server Namespace: openshift-monitoring Version: v1beta1 Version Priority: 100 Status: Conditions: Last Transition Time: 2018-07-31T09:00:27Z Message: all checks passed Reason: Passed Status: True Type: Available Events: <none> Current CPU usage fails to be retreived either: [root@qe-weinliu-311-master-etcd-1 ~]# oc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/autoscaling/hpa-v2beta1/resource-metrics-cpu.yaml -n dma horizontalpodautoscaler.autoscaling/resource-cpu created [root@qe-weinliu-311-master-etcd-1 ~]# oc get hpa.v2beta1.autoscaling resource-cpu -n dma NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE resource-cpu ReplicationController/hello-openshift <unknown>/80% 2 10 0 10s [root@qe-weinliu-311-master-etcd-1 ~]# oc describe hpa.v2beta1.autoscaling resource-cpu -n dma Name: resource-cpu Namespace: dma Labels: <none> Annotations: <none> CreationTimestamp: Tue, 31 Jul 2018 05:34:56 -0400 Reference: ReplicationController/hello-openshift Metrics: ( current / target ) resource cpu on pods (as a percentage of request): <unknown> / 80% Min replicas: 2 Max replicas: 10 ReplicationController pods: 0 current / 0 desired Events: <none>
Fix PR: https://github.com/openshift/origin/pull/20529. It's not a metrics-server issue, necessarily. A refactor was done to the kubectl code to change how IO streams were passed in, and the way the `kubectl top` command was changed made it easy to accidentally ignore those and never set streams.
Still fails on [root@qe-weinliu-311-master-etcd-1 ~]# oc version oc v3.11.0-0.11.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-weinliu-311-master-etcd-1:8443 openshift v3.11.0-0.11.0 kubernetes v1.11.0+d4cacc0 [root@qe-weinliu-311-master-etcd-1 ~]#
Just confirmed that this works on master. Please make sure that you're using the latest code before retesting.
Verified to be fixed. [root@qe-weinliu-round3-private-2-me-1 ~]# oc adm top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% qe-weinliu-round3-private-2-me-1 543m 13% 3510Mi 23% qe-weinliu-round3-private-2-node-1 172m 4% 2284Mi 15% qe-weinliu-round3-private-2-nrr-1 138m 3% 1872Mi 12% [root@qe-weinliu-round3-private-2-me-1 ~]# oc version oc v3.11.0-0.16.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-weinliu-round3-private-2-me-1:8443 openshift v3.11.0-0.16.0 kubernetes v1.11.0+d4cacc0 [root@qe-weinliu-round3-private-2-me-1 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Atomic Host release 7.4
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652