Attachment 1488287 Details for Bug 1634227 – cluster-monitoring-operator pod logs

cluster-monitoring-operator pod logs

cluster_monitoring_logs.txt (text/plain), 13.81 KB, created by Junqi Zhao on 2018-09-29 05:19:14 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Junqi Zhao

Created: 2018-09-29 05:19:14 UTC

Size: 13.81 KB

patch

obsolete

># oc -n openshift-monitoring get pod
>NAME                                           READY     STATUS             RESTARTS   AGE
>alertmanager-main-0                            3/3       Running            0          57m
>alertmanager-main-1                            3/3       Running            0          57m
>alertmanager-main-2                            3/3       Running            0          56m
>cluster-monitoring-operator-56bb5946c4-49v8n   0/1       CrashLoopBackOff   12         1h
>grafana-69c5976c69-gmkvg                       2/2       Running            0          1h
>kube-state-metrics-67fd45fc58-jqkrp            3/3       Running            0          55m
>node-exporter-4b6xg                            2/2       Running            0          55m
>node-exporter-5stxn                            2/2       Running            0          55m
>node-exporter-bjnl8                            2/2       Running            0          55m
>node-exporter-bnkzn                            2/2       Running            0          55m
>node-exporter-fl7cj                            2/2       Running            0          55m
>node-exporter-lf7tz                            2/2       Running            0          55m
>node-exporter-m2tbc                            2/2       Running            0          55m
>node-exporter-rq8w8                            2/2       Running            0          55m
>node-exporter-s6bvd                            2/2       Running            0          55m
>node-exporter-twfdp                            2/2       Running            0          55m
>prometheus-k8s-0                               4/4       Running            1          1h
>prometheus-k8s-1                               4/4       Running            1          59m
>prometheus-operator-dfbc5bd76-qjh82            1/1       Running            0          1h
>
>
>
># oc -n openshift-monitoring logs cluster-monitoring-operator-56bb5946c4-49v8n
>I0929 03:12:20.306691       1 operator.go:116] Connection established (cluster-version: v1.11.0+d4cacc0)
>I0929 03:12:20.319613       1 operator.go:167] Waiting for initial cache sync.
>I0929 03:12:20.322641       1 reflector.go:202] Starting reflector *v1.ConfigMap (5m0s) from github.com/openshift/cluster-monitoring-operator/pkg/operator/operator.go:132
>I0929 03:12:20.322694       1 reflector.go:240] Listing and watching *v1.ConfigMap from github.com/openshift/cluster-monitoring-operator/pkg/operator/operator.go:132
>I0929 03:12:21.207149       1 operator.go:157] ConfigMap updated: openshift-monitoring/cluster-monitoring-config
>I0929 03:12:21.207196       1 operator.go:157] ConfigMap updated: openshift-monitoring/grafana-dashboard-k8s-cluster-rsrc-use
>I0929 03:12:21.207211       1 operator.go:160] ConfigMap (openshift-monitoring/grafana-dashboard-k8s-cluster-rsrc-use) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.207240       1 operator.go:157] ConfigMap updated: openshift-monitoring/grafana-dashboard-k8s-node-rsrc-use
>I0929 03:12:21.207261       1 operator.go:160] ConfigMap (openshift-monitoring/grafana-dashboard-k8s-node-rsrc-use) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.207276       1 operator.go:157] ConfigMap updated: openshift-monitoring/grafana-dashboard-k8s-resources-cluster
>I0929 03:12:21.207287       1 operator.go:160] ConfigMap (openshift-monitoring/grafana-dashboard-k8s-resources-cluster) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.207304       1 operator.go:157] ConfigMap updated: openshift-monitoring/grafana-dashboard-k8s-resources-namespace
>I0929 03:12:21.207317       1 operator.go:160] ConfigMap (openshift-monitoring/grafana-dashboard-k8s-resources-namespace) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.207330       1 operator.go:157] ConfigMap updated: openshift-monitoring/grafana-dashboard-k8s-resources-pod
>I0929 03:12:21.207340       1 operator.go:160] ConfigMap (openshift-monitoring/grafana-dashboard-k8s-resources-pod) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.207355       1 operator.go:157] ConfigMap updated: openshift-monitoring/grafana-dashboards
>I0929 03:12:21.207394       1 operator.go:160] ConfigMap (openshift-monitoring/grafana-dashboards) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.207412       1 operator.go:157] ConfigMap updated: openshift-monitoring/prometheus-k8s-rulefiles-0
>I0929 03:12:21.207425       1 operator.go:160] ConfigMap (openshift-monitoring/prometheus-k8s-rulefiles-0) not triggering an update. Only changes to openshift-monitoring/cluster-monitoring-config configure the cluster monitoring stack.
>I0929 03:12:21.319830       1 operator.go:169] Initial cache sync done.
>I0929 03:12:21.319902       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:21.326686       1 operator.go:284] Could not fetch cluster configuration from API. Proceeding without it.
>I0929 03:12:21.326723       1 tasks.go:37] running task Updating Prometheus Operator
>I0929 03:12:21.326810       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:21.407700       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:21.807552       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:22.006594       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:22.210199       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:41.507386       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:41.623932       1 tasks.go:37] running task Updating Cluster Monitoring Operator
>I0929 03:12:41.624030       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:41.806514       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:41.912677       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:42.028838       1 tasks.go:37] running task Updating Grafana
>I0929 03:12:42.028925       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:42.050171       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:42.131557       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:43.209597       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:43.213301       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:43.312265       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:43.607357       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:46.707936       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:46.806679       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:46.907402       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:47.300325       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:48.422700       1 tasks.go:37] running task Updating Prometheus-k8s
>I0929 03:12:48.422804       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.510906       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.518901       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.525166       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.531519       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.642419       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.726643       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:49.906654       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:50.106625       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:50.332063       1 request.go:485] Throttling request took 124.746592ms, request: PUT:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/default/rolebindings/prometheus-k8s
>I0929 03:12:50.532052       1 request.go:485] Throttling request took 178.872298ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-system/rolebindings/prometheus-k8s
>I0929 03:12:50.732063       1 request.go:485] Throttling request took 197.111766ms, request: PUT:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-system/rolebindings/prometheus-k8s
>I0929 03:12:50.932511       1 request.go:485] Throttling request took 177.81908ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-monitoring/rolebindings/prometheus-k8s
>I0929 03:12:51.132045       1 request.go:485] Throttling request took 196.51842ms, request: PUT:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-monitoring/rolebindings/prometheus-k8s
>I0929 03:12:51.153844       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:51.332040       1 request.go:485] Throttling request took 177.895477ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-monitoring/rolebindings/prometheus-k8s-config
>I0929 03:12:51.532072       1 request.go:485] Throttling request took 196.580149ms, request: PUT:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-monitoring/rolebindings/prometheus-k8s-config
>I0929 03:12:51.556074       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:54.208058       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:54.406798       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:54.514807       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:54.706760       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:54.812039       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:54.917197       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:55.016709       1 prometheus.go:263] initializing Prometheus object
>I0929 03:12:55.016850       1 decoder.go:224] decoding stream as YAML
>I0929 03:12:55.106903       1 prometheus.go:269] reconciling Prometheus object
>I0929 03:12:55.306788       1 request.go:485] Throttling request took 91.123544ms, request: PUT:https://172.30.0.1:443/apis/monitoring.coreos.com/v1/namespaces/openshift-monitoring/prometheuses/k8s
>I0929 03:12:55.318090       1 prometheus.go:275] waiting for Prometheus object changes
>I0929 03:13:06.008103       1 tasks.go:37] running task Updating Alertmanager
>I0929 03:13:06.008197       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.116175       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.206704       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.211121       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.263202       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.406698       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.434283       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.506604       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:07.653723       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:17.806692       1 tasks.go:37] running task Updating node-exporter
>I0929 03:13:17.806799       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:18.306761       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:18.506600       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:18.510098       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:18.606531       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:18.629097       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:18.806667       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:20.316096       1 tasks.go:37] running task Updating kube-state-metrics
>I0929 03:13:20.316201       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:20.426936       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:20.506659       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:20.630192       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:20.661769       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:20.706702       1 decoder.go:224] decoding stream as YAML
>I0929 03:13:21.844419       1 tasks.go:37] running task Updating Telemeter client
>I0929 03:13:21.844509       1 decoder.go:224] decoding stream as YAML
>panic: runtime error: invalid memory address or nil pointer dereference
>[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x10d344d]
>
>goroutine 36 [running]:
>github.com/openshift/cluster-monitoring-operator/pkg/manifests.(*Factory).TelemeterClientServiceMonitor(0xc4204ccc60, 0xc4200ae840, 0xc420a51cc0, 0x6ae93a)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/manifests/manifests.go:1441 +0x11d
>github.com/openshift/cluster-monitoring-operator/pkg/tasks.(*TelemeterClientTask).Run(0xc42036a0c0, 0x1, 0x1)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/tasks/telemeter.go:36 +0x33
>github.com/openshift/cluster-monitoring-operator/pkg/tasks.(*TaskRunner).ExecuteTask(0xc420a51eb8, 0xc4204cce40, 0xf, 0xc420a51d60)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/tasks/tasks.go:48 +0x34
>github.com/openshift/cluster-monitoring-operator/pkg/tasks.(*TaskRunner).RunAll(0xc420a51eb8, 0xc420473a80, 0x53f961cb21d1086d)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/tasks/tasks.go:38 +0x141
>github.com/openshift/cluster-monitoring-operator/pkg/operator.(*Operator).sync(0xc42017bf00, 0xc4203b0d50, 0x2e, 0x11a2f40, 0xc42042b790)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/operator/operator.go:251 +0x828
>github.com/openshift/cluster-monitoring-operator/pkg/operator.(*Operator).processNextWorkItem(0xc42017bf00, 0xc420037f00)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/operator/operator.go:201 +0xfb
>github.com/openshift/cluster-monitoring-operator/pkg/operator.(*Operator).worker(0xc42017bf00)
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/operator/operator.go:171 +0x15a
>created by github.com/openshift/cluster-monitoring-operator/pkg/operator.(*Operator).Run
>	/go/src/github.com/openshift/cluster-monitoring-operator/pkg/operator/operator.go:130 +0x1e2
>

Actions: View

Attachments on bug 1634227: 1488287