Bug 1673178
Summary: | node-exporter and kube-state metrics pods in CrashLoopBackOff | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Sai Sindhur Malleni <smalleni> |
Component: | Monitoring | Assignee: | Frederic Branczyk <fbranczy> |
Status: | CLOSED ERRATA | QA Contact: | Junqi Zhao <juzhao> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 3.11.0 | CC: | abond, cshereme, juzhao, minden, mloibl, surbania, vlaad, wmeng |
Target Milestone: | --- | Keywords: | Regression, TestBlocker |
Target Release: | 3.11.z | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-02-20 14:11:12 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Sai Sindhur Malleni
2019-02-06 20:36:15 UTC
Could you please share a Pod manifest of one of the pods in question? Thanks! Sorry about the inconvenience. What you are seeing is a defect that got in with this pull request: https://github.com/openshift/cluster-monitoring-operator/pull/210. This has already been fixed as of https://github.com/openshift/cluster-monitoring-operator/pull/225. We will just need to wait for the next OCP z stream release. Tested with v3.11.82, issue is not fixed, # oc get pod -n openshift-monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 3/3 Running 0 12m alertmanager-main-1 3/3 Running 0 12m alertmanager-main-2 3/3 Running 0 12m cluster-monitoring-operator-548fc4f6d4-pmkfh 1/1 Running 0 13m grafana-69bb9997f5-ppswq 2/2 Running 0 13m kube-state-metrics-946b9f84d-s4hzr 1/3 CrashLoopBackOff 14 12m node-exporter-h7wlb 1/2 CrashLoopBackOff 7 12m node-exporter-nztr7 1/2 CrashLoopBackOff 7 12m node-exporter-zdnzg 1/2 CrashLoopBackOff 7 12m prometheus-k8s-0 4/4 Running 1 13m prometheus-k8s-1 4/4 Running 1 13m prometheus-operator-55bbdd949b-wq7bt 1/1 Running 0 13m # oc -n openshift-monitoring logs node-exporter-h7wlb -c kube-rbac-proxy F0211 08:32:06.230291 75189 main.go:240] failed to configure http2 server: http2: TLSConfig.CipherSuites index 11 contains an HTTP/2-approved cipher suite (0xc02f), but it comes after unapproved cipher suites. With this configuration, clients that don't support previous, approved cipher suites may be given an unapproved one and reject the connection. goroutine 1 [running]: github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.stacks(0xc4202e6200, 0xc420326000, 0x163, 0x1b7) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:769 +0xcf github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.(*loggingT).output(0x1a80580, 0xc400000003, 0xc4200de630, 0x19e36de, 0x7, 0xf0, 0x0) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:720 +0x32d github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.(*loggingT).printf(0x1a80580, 0xc400000003, 0x1210047, 0x24, 0xc420319da0, 0x1, 0x1) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:655 +0x14b github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.Fatalf(0x1210047, 0x24, 0xc420319da0, 0x1, 0x1) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:1148 +0x67 main.main() /go/src/github.com/brancz/kube-rbac-proxy/main.go:240 +0x18fc **************************************************************************************************************************** # oc -n openshift-monitoring logs kube-state-metrics-946b9f84d-s4hzr -c kube-rbac-proxy-main F0211 08:33:48.829740 1 main.go:240] failed to configure http2 server: http2: TLSConfig.CipherSuites index 11 contains an HTTP/2-approved cipher suite (0xc02f), but it comes after unapproved cipher suites. With this configuration, clients that don't support previous, approved cipher suites may be given an unapproved one and reject the connection. goroutine 1 [running]: github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.stacks(0xc420070100, 0xc4205fc000, 0x163, 0x1b7) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:769 +0xcf github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.(*loggingT).output(0x1a80580, 0xc400000003, 0xc4200de630, 0x19e36de, 0x7, 0xf0, 0x0) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:720 +0x32d github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.(*loggingT).printf(0x1a80580, 0xc400000003, 0x1210047, 0x24, 0xc42032fda0, 0x1, 0x1) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:655 +0x14b github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.Fatalf(0x1210047, 0x24, 0xc42032fda0, 0x1, 0x1) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:1148 +0x67 main.main() /go/src/github.com/brancz/kube-rbac-proxy/main.go:240 +0x18fc **************************************************************************************************************************** # oc -n openshift-monitoring logs kube-state-metrics-946b9f84d-s4hzr -c kube-rbac-proxy-self F0211 08:33:49.430652 1 main.go:240] failed to configure http2 server: http2: TLSConfig.CipherSuites index 11 contains an HTTP/2-approved cipher suite (0xc02f), but it comes after unapproved cipher suites. With this configuration, clients that don't support previous, approved cipher suites may be given an unapproved one and reject the connection. goroutine 1 [running]: github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.stacks(0xc420411c00, 0xc420436000, 0x163, 0x1b7) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:769 +0xcf github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.(*loggingT).output(0x1a80580, 0xc400000003, 0xc4200de630, 0x19e36de, 0x7, 0xf0, 0x0) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:720 +0x32d github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.(*loggingT).printf(0x1a80580, 0xc400000003, 0x1210047, 0x24, 0xc420317da0, 0x1, 0x1) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:655 +0x14b github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog.Fatalf(0x1210047, 0x24, 0xc420317da0, 0x1, 0x1) /go/src/github.com/brancz/kube-rbac-proxy/vendor/github.com/golang/glog/glog.go:1148 +0x67 main.main() /go/src/github.com/brancz/kube-rbac-proxy/main.go:240 +0x18fc @Junqi could you share one of the pod manifests? Just to check if the patch [1] mentioned above ever made it into the images. [1] https://github.com/openshift/cluster-monitoring-operator/pull/225/files Looking into this some more, the described patch did not make it into the binary properly due to an issue in our build system. https://github.com/openshift/cluster-monitoring-operator/pull/241 should fix this. I just validated the fix on Openshift 3.11. Once this got another code review we will go ahead and merge. https://github.com/openshift/cluster-monitoring-operator/pull/241 is merged. Would you mind taking another look Junqi? Tested with ose-cluster-monitoring-operator:v3.11.82-4, issue is fixed and cluster monitoring works well # oc -n openshift-monitoring get po NAME READY STATUS RESTARTS AGE alertmanager-main-0 3/3 Running 0 18m alertmanager-main-1 3/3 Running 0 18m alertmanager-main-2 3/3 Running 0 17m cluster-monitoring-operator-98f84d4dd-st5v7 1/1 Running 0 25m grafana-7fb8d6b4bf-b7nqs 2/2 Running 0 22m kube-state-metrics-9bf978578-z6pwz 3/3 Running 0 16m node-exporter-887fk 2/2 Running 0 17m node-exporter-mfdvx 2/2 Running 0 17m node-exporter-w6fpx 2/2 Running 0 17m prometheus-k8s-0 4/4 Running 1 21m prometheus-k8s-1 4/4 Running 1 20m prometheus-operator-544d79d996-gmhnb 1/1 Running 0 24m Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0326 |