Bug 1700037
| Summary: | CMO deployments are changing all the time | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Seth Jennings <sjenning> |
| Component: | apiserver-auth | Assignee: | Matt Rogers <mrogers> |
| Status: | CLOSED ERRATA | QA Contact: | Chuan Yu <chuyu> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.1.0 | CC: | anpicker, aos-bugs, erooth, mloibl, mrogers, nagrawal, ncredi, pkrupa, surbania |
| Target Milestone: | --- | ||
| Target Release: | 4.1.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-06-04 10:47:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The CMO itself is strange indeed as we never touch that deployment it's entirely handled by the CVO. The rest is likely because we do perform reconciles every 5mins, and reacting to various configuration changes in the cluster. I understand this might not be perfect, but is priority high appropriate? I can drop it a notch. The prom adapter is a little worse off than the others: $ oc get events | grep "Created pod" 3h5m Normal SuccessfulCreate replicaset/cluster-monitoring-operator-7849456b55 Created pod: cluster-monitoring-operator-7849456b55-xv284 3h4m Normal SuccessfulCreate replicaset/grafana-b87f78c64 Created pod: grafana-b87f78c64-hkddd 3h5m Normal SuccessfulCreate replicaset/kube-state-metrics-7594bf4d85 Created pod: kube-state-metrics-7594bf4d85-t4f6c 3h5m Normal SuccessfulCreate daemonset/node-exporter Created pod: node-exporter-wqkh6 3h5m Normal SuccessfulCreate daemonset/node-exporter Created pod: node-exporter-2frkw 3h5m Normal SuccessfulCreate daemonset/node-exporter Created pod: node-exporter-jvzhd 3h5m Normal SuccessfulCreate daemonset/node-exporter Created pod: node-exporter-tdxdc 3h5m Normal SuccessfulCreate daemonset/node-exporter Created pod: node-exporter-z46zl 56m Normal SuccessfulCreate daemonset/node-exporter Created pod: node-exporter-95cjw 34m Normal SuccessfulCreate replicaset/prometheus-adapter-54b8f8766 Created pod: prometheus-adapter-54b8f8766-wzjld 34m Normal SuccessfulCreate replicaset/prometheus-adapter-54b8f8766 Created pod: prometheus-adapter-54b8f8766-dspz7 100m Normal SuccessfulCreate replicaset/prometheus-adapter-54c4d44786 Created pod: prometheus-adapter-54c4d44786-4t9q7 99m Normal SuccessfulCreate replicaset/prometheus-adapter-54c4d44786 Created pod: prometheus-adapter-54c4d44786-sbr9g 2s Normal SuccessfulCreate replicaset/prometheus-adapter-5555fdcbf4 Created pod: prometheus-adapter-5555fdcbf4-zf5gc 54m Normal SuccessfulCreate replicaset/prometheus-adapter-558f8b4bdc Created pod: prometheus-adapter-558f8b4bdc-wflzr 54m Normal SuccessfulCreate replicaset/prometheus-adapter-558f8b4bdc Created pod: prometheus-adapter-558f8b4bdc-bfj84 9m30s Normal SuccessfulCreate replicaset/prometheus-adapter-5595c65667 Created pod: prometheus-adapter-5595c65667-4xblc 9m21s Normal SuccessfulCreate replicaset/prometheus-adapter-5595c65667 Created pod: prometheus-adapter-5595c65667-crxrs 139m Normal SuccessfulCreate replicaset/prometheus-adapter-5656d4b88c Created pod: prometheus-adapter-5656d4b88c-gnx45 139m Normal SuccessfulCreate replicaset/prometheus-adapter-5656d4b88c Created pod: prometheus-adapter-5656d4b88c-qxhms 64m Normal SuccessfulCreate replicaset/prometheus-adapter-57f8679d7f Created pod: prometheus-adapter-57f8679d7f-x4jtz 64m Normal SuccessfulCreate replicaset/prometheus-adapter-57f8679d7f Created pod: prometheus-adapter-57f8679d7f-l6w99 44m Normal SuccessfulCreate replicaset/prometheus-adapter-594fdb5f86 Created pod: prometheus-adapter-594fdb5f86-sp5g9 44m Normal SuccessfulCreate replicaset/prometheus-adapter-594fdb5f86 Created pod: prometheus-adapter-594fdb5f86-hlzxz 164m Normal SuccessfulCreate replicaset/prometheus-adapter-598d8599d7 Created pod: prometheus-adapter-598d8599d7-tfdtc 164m Normal SuccessfulCreate replicaset/prometheus-adapter-598d8599d7 Created pod: prometheus-adapter-598d8599d7-9nrtt 154m Normal SuccessfulCreate replicaset/prometheus-adapter-5bcfff679d Created pod: prometheus-adapter-5bcfff679d-sc8md 154m Normal SuccessfulCreate replicaset/prometheus-adapter-5bcfff679d Created pod: prometheus-adapter-5bcfff679d-th2c7 130m Normal SuccessfulCreate replicaset/prometheus-adapter-5c5cd8d75d Created pod: prometheus-adapter-5c5cd8d75d-sqq7c 129m Normal SuccessfulCreate replicaset/prometheus-adapter-5c5cd8d75d Created pod: prometheus-adapter-5c5cd8d75d-6b8bp 174m Normal SuccessfulCreate replicaset/prometheus-adapter-5c769fcf97 Created pod: prometheus-adapter-5c769fcf97-qzqc7 174m Normal SuccessfulCreate replicaset/prometheus-adapter-5c769fcf97 Created pod: prometheus-adapter-5c769fcf97-zt7tf 110m Normal SuccessfulCreate replicaset/prometheus-adapter-5c88685d54 Created pod: prometheus-adapter-5c88685d54-x8cf4 109m Normal SuccessfulCreate replicaset/prometheus-adapter-5c88685d54 Created pod: prometheus-adapter-5c88685d54-6vsjv 119m Normal SuccessfulCreate replicaset/prometheus-adapter-5d74d6cbf7 Created pod: prometheus-adapter-5d74d6cbf7-hbc6l 119m Normal SuccessfulCreate replicaset/prometheus-adapter-5d74d6cbf7 Created pod: prometheus-adapter-5d74d6cbf7-t66g8 145m Normal SuccessfulCreate replicaset/prometheus-adapter-5db8866879 Created pod: prometheus-adapter-5db8866879-h94cj 144m Normal SuccessfulCreate replicaset/prometheus-adapter-5db8866879 Created pod: prometheus-adapter-5db8866879-689rn 29m Normal SuccessfulCreate replicaset/prometheus-adapter-5fdd486b9f Created pod: prometheus-adapter-5fdd486b9f-drzzf 29m Normal SuccessfulCreate replicaset/prometheus-adapter-5fdd486b9f Created pod: prometheus-adapter-5fdd486b9f-vhsvz 95m Normal SuccessfulCreate replicaset/prometheus-adapter-64649b86f Created pod: prometheus-adapter-64649b86f-5xj6z 94m Normal SuccessfulCreate replicaset/prometheus-adapter-64649b86f Created pod: prometheus-adapter-64649b86f-7xcjw 15m Normal SuccessfulCreate replicaset/prometheus-adapter-646c8bfc7b Created pod: prometheus-adapter-646c8bfc7b-hlpz5 14m Normal SuccessfulCreate replicaset/prometheus-adapter-646c8bfc7b Created pod: prometheus-adapter-646c8bfc7b-p6jdz 45m Normal SuccessfulCreate replicaset/prometheus-adapter-65bbf89bd4 Created pod: prometheus-adapter-65bbf89bd4-w6cj2 44m Normal SuccessfulCreate replicaset/prometheus-adapter-65bbf89bd4 Created pod: prometheus-adapter-65bbf89bd4-49gjg 140m Normal SuccessfulCreate replicaset/prometheus-adapter-66746859b4 Created pod: prometheus-adapter-66746859b4-4h5df 139m Normal SuccessfulCreate replicaset/prometheus-adapter-66746859b4 Created pod: prometheus-adapter-66746859b4-msq7k 85m Normal SuccessfulCreate replicaset/prometheus-adapter-667f7b64c9 Created pod: prometheus-adapter-667f7b64c9-h54wp 84m Normal SuccessfulCreate replicaset/prometheus-adapter-667f7b64c9 Created pod: prometheus-adapter-667f7b64c9-x6bzr 124m Normal SuccessfulCreate replicaset/prometheus-adapter-674f5cc946 Created pod: prometheus-adapter-674f5cc946-xzvhd 124m Normal SuccessfulCreate replicaset/prometheus-adapter-674f5cc946 Created pod: prometheus-adapter-674f5cc946-tmwgr 165m Normal SuccessfulCreate replicaset/prometheus-adapter-67db5fcd97 Created pod: prometheus-adapter-67db5fcd97-nv8ws 164m Normal SuccessfulCreate replicaset/prometheus-adapter-67db5fcd97 Created pod: prometheus-adapter-67db5fcd97-gcbsj 75m Normal SuccessfulCreate replicaset/prometheus-adapter-67fcd6d495 Created pod: prometheus-adapter-67fcd6d495-2pwnp 74m Normal SuccessfulCreate replicaset/prometheus-adapter-67fcd6d495 Created pod: prometheus-adapter-67fcd6d495-7dg26 59m Normal SuccessfulCreate replicaset/prometheus-adapter-68555ccbbb Created pod: prometheus-adapter-68555ccbbb-pzjcr 59m Normal SuccessfulCreate replicaset/prometheus-adapter-68555ccbbb Created pod: prometheus-adapter-68555ccbbb-njkf8 94m Normal SuccessfulCreate replicaset/prometheus-adapter-68757485f7 Created pod: prometheus-adapter-68757485f7-z4jpm 94m Normal SuccessfulCreate replicaset/prometheus-adapter-68757485f7 Created pod: prometheus-adapter-68757485f7-vkqgj 179m Normal SuccessfulCreate replicaset/prometheus-adapter-6987d4848c Created pod: prometheus-adapter-6987d4848c-s4dkw 179m Normal SuccessfulCreate replicaset/prometheus-adapter-6987d4848c Created pod: prometheus-adapter-6987d4848c-cmql2 55m Normal SuccessfulCreate replicaset/prometheus-adapter-6b5d84b5b8 Created pod: prometheus-adapter-6b5d84b5b8-zlcld 54m Normal SuccessfulCreate replicaset/prometheus-adapter-6b5d84b5b8 Created pod: prometheus-adapter-6b5d84b5b8-8ms79 65m Normal SuccessfulCreate replicaset/prometheus-adapter-6bb6d8d754 Created pod: prometheus-adapter-6bb6d8d754-tgzl5 64m Normal SuccessfulCreate replicaset/prometheus-adapter-6bb6d8d754 Created pod: prometheus-adapter-6bb6d8d754-bd2hc 39m Normal SuccessfulCreate replicaset/prometheus-adapter-6c7548dbc7 Created pod: prometheus-adapter-6c7548dbc7-fw6mv 39m Normal SuccessfulCreate replicaset/prometheus-adapter-6c7548dbc7 Created pod: prometheus-adapter-6c7548dbc7-956qw 155m Normal SuccessfulCreate replicaset/prometheus-adapter-6f87876dff Created pod: prometheus-adapter-6f87876dff-r6d9g 154m Normal SuccessfulCreate replicaset/prometheus-adapter-6f87876dff Created pod: prometheus-adapter-6f87876dff-hghpc 129m Normal SuccessfulCreate replicaset/prometheus-adapter-6fb6c47bcf Created pod: prometheus-adapter-6fb6c47bcf-tcdz4 129m Normal SuccessfulCreate replicaset/prometheus-adapter-6fb6c47bcf Created pod: prometheus-adapter-6fb6c47bcf-fdwj5 50m Normal SuccessfulCreate replicaset/prometheus-adapter-6fd9b56bbc Created pod: prometheus-adapter-6fd9b56bbc-htgd2 49m Normal SuccessfulCreate replicaset/prometheus-adapter-6fd9b56bbc Created pod: prometheus-adapter-6fd9b56bbc-qzvzb 160m Normal SuccessfulCreate replicaset/prometheus-adapter-748587c859 Created pod: prometheus-adapter-748587c859-d669n 159m Normal SuccessfulCreate replicaset/prometheus-adapter-748587c859 Created pod: prometheus-adapter-748587c859-994hq 125m Normal SuccessfulCreate replicaset/prometheus-adapter-75554fb858 Created pod: prometheus-adapter-75554fb858-mvj2l 124m Normal SuccessfulCreate replicaset/prometheus-adapter-75554fb858 Created pod: prometheus-adapter-75554fb858-56ctf 30m Normal SuccessfulCreate replicaset/prometheus-adapter-76dcd75d74 Created pod: prometheus-adapter-76dcd75d74-75pcw 29m Normal SuccessfulCreate replicaset/prometheus-adapter-76dcd75d74 Created pod: prometheus-adapter-76dcd75d74-xvxdh 120m Normal SuccessfulCreate replicaset/prometheus-adapter-774bd5454c Created pod: prometheus-adapter-774bd5454c-j5sbz 119m Normal SuccessfulCreate replicaset/prometheus-adapter-774bd5454c Created pod: prometheus-adapter-774bd5454c-9f4l7 169m Normal SuccessfulCreate replicaset/prometheus-adapter-78bfcc86cf Created pod: prometheus-adapter-78bfcc86cf-gvn5f 169m Normal SuccessfulCreate replicaset/prometheus-adapter-78bfcc86cf Created pod: prometheus-adapter-78bfcc86cf-kxmsb 80m Normal SuccessfulCreate replicaset/prometheus-adapter-78c486c8b5 Created pod: prometheus-adapter-78c486c8b5-smpcq 79m Normal SuccessfulCreate replicaset/prometheus-adapter-78c486c8b5 Created pod: prometheus-adapter-78c486c8b5-scbbw 150m Normal SuccessfulCreate replicaset/prometheus-adapter-78c7cbf8f9 Created pod: prometheus-adapter-78c7cbf8f9-wl28n 149m Normal SuccessfulCreate replicaset/prometheus-adapter-78c7cbf8f9 Created pod: prometheus-adapter-78c7cbf8f9-pj7dv 49m Normal SuccessfulCreate replicaset/prometheus-adapter-78f9f96d8 Created pod: prometheus-adapter-78f9f96d8-r6gnv 49m Normal SuccessfulCreate replicaset/prometheus-adapter-78f9f96d8 Created pod: prometheus-adapter-78f9f96d8-pfqr5 24m Normal SuccessfulCreate replicaset/prometheus-adapter-7944c4c899 Created pod: prometheus-adapter-7944c4c899-pbtt6 24m Normal SuccessfulCreate replicaset/prometheus-adapter-7944c4c899 Created pod: prometheus-adapter-7944c4c899-txd9l 105m Normal SuccessfulCreate replicaset/prometheus-adapter-7bf5dc88cd Created pod: prometheus-adapter-7bf5dc88cd-sldjr 104m Normal SuccessfulCreate replicaset/prometheus-adapter-7bf5dc88cd Created pod: prometheus-adapter-7bf5dc88cd-zvq25 134m Normal SuccessfulCreate replicaset/prometheus-adapter-7c748db4c8 Created pod: prometheus-adapter-7c748db4c8-76qbb 134m Normal SuccessfulCreate replicaset/prometheus-adapter-7c748db4c8 Created pod: prometheus-adapter-7c748db4c8-pgsjq 19m Normal SuccessfulCreate replicaset/prometheus-adapter-7cf87db645 Created pod: prometheus-adapter-7cf87db645-zl9s4 19m Normal SuccessfulCreate replicaset/prometheus-adapter-7cf87db645 Created pod: prometheus-adapter-7cf87db645-8frdf 74m Normal SuccessfulCreate replicaset/prometheus-adapter-7d79864dc5 Created pod: prometheus-adapter-7d79864dc5-4vpl4 74m Normal SuccessfulCreate replicaset/prometheus-adapter-7d79864dc5 Created pod: prometheus-adapter-7d79864dc5-bds9m 69m Normal SuccessfulCreate replicaset/prometheus-adapter-7f4fb579c8 Created pod: prometheus-adapter-7f4fb579c8-v5hww 69m Normal SuccessfulCreate replicaset/prometheus-adapter-7f4fb579c8 Created pod: prometheus-adapter-7f4fb579c8-4ld9q 3h3m Normal SuccessfulCreate replicaset/prometheus-adapter-7f7d4579d6 Created pod: prometheus-adapter-7f7d4579d6-8gg48 3h3m Normal SuccessfulCreate replicaset/prometheus-adapter-7f7d4579d6 Created pod: prometheus-adapter-7f7d4579d6-xpmn2 3h Normal SuccessfulCreate replicaset/prometheus-adapter-8568cd9cb7 Created pod: prometheus-adapter-8568cd9cb7-5572f 179m Normal SuccessfulCreate replicaset/prometheus-adapter-8568cd9cb7 Created pod: prometheus-adapter-8568cd9cb7-89zv7 5m2s Normal SuccessfulCreate replicaset/prometheus-adapter-86b4ccf78f Created pod: prometheus-adapter-86b4ccf78f-bgx6h 4m54s Normal SuccessfulCreate replicaset/prometheus-adapter-86b4ccf78f Created pod: prometheus-adapter-86b4ccf78f-cz4ns 149m Normal SuccessfulCreate replicaset/prometheus-adapter-b57654fb5 Created pod: prometheus-adapter-b57654fb5-fzwlf 149m Normal SuccessfulCreate replicaset/prometheus-adapter-b57654fb5 Created pod: prometheus-adapter-b57654fb5-9vkcj 170m Normal SuccessfulCreate replicaset/prometheus-adapter-c4d8ccd84 Created pod: prometheus-adapter-c4d8ccd84-zqqcp 169m Normal SuccessfulCreate replicaset/prometheus-adapter-c4d8ccd84 Created pod: prometheus-adapter-c4d8ccd84-grd2p 114m Normal SuccessfulCreate replicaset/prometheus-adapter-c8c549648 Created pod: prometheus-adapter-c8c549648-v6l8l 114m Normal SuccessfulCreate replicaset/prometheus-adapter-c8c549648 Created pod: prometheus-adapter-c8c549648-z9d2x 10m Normal SuccessfulCreate replicaset/prometheus-adapter-c8f99fdfc Created pod: prometheus-adapter-c8f99fdfc-m6m7d 9m53s Normal SuccessfulCreate replicaset/prometheus-adapter-c8f99fdfc Created pod: prometheus-adapter-c8f99fdfc-v4vxn 20m Normal SuccessfulCreate replicaset/prometheus-adapter-c9c9c4b6f Created pod: prometheus-adapter-c9c9c4b6f-mzjcn 19m Normal SuccessfulCreate replicaset/prometheus-adapter-c9c9c4b6f Created pod: prometheus-adapter-c9c9c4b6f-tnjp6 89m Normal SuccessfulCreate replicaset/prometheus-adapter-d888cc874 Created pod: prometheus-adapter-d888cc874-jvz42 89m Normal SuccessfulCreate replicaset/prometheus-adapter-d888cc874 Created pod: prometheus-adapter-d888cc874-zdhx5 135m Normal SuccessfulCreate replicaset/prometheus-adapter-d8dc5fbbc Created pod: prometheus-adapter-d8dc5fbbc-2sbfq 134m Normal SuccessfulCreate replicaset/prometheus-adapter-d8dc5fbbc Created pod: prometheus-adapter-d8dc5fbbc-fz2qg 99m Normal SuccessfulCreate replicaset/prometheus-adapter-dd786b994 Created pod: prometheus-adapter-dd786b994-khvp8 99m Normal SuccessfulCreate replicaset/prometheus-adapter-dd786b994 Created pod: prometheus-adapter-dd786b994-b4lqz 3h3m Normal SuccessfulCreate replicaset/prometheus-operator-7b5b64994b Created pod: prometheus-operator-7b5b64994b-hvcb4 3h5m Normal SuccessfulCreate replicaset/prometheus-operator-7f9764d484 Created pod: prometheus-operator-7f9764d484-2896h 3h5m Normal SuccessfulCreate replicaset/telemeter-client-574fbd95d7 Created pod: telemeter-client-574fbd95d7-hbck4 Creating a lot of pods, most of which fail to schedule with events like 36m Warning FailedScheduling pod/prometheus-adapter-54b8f8766-dspz7 Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-dspz7": pod prometheus-adapter-54b8f8766-dspz7 is already assigned to node "worker-0" 36m Warning FailedScheduling pod/prometheus-adapter-54b8f8766-dspz7 Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-dspz7": pod prometheus-adapter-54b8f8766-dspz7 is already assigned to node "worker-0" 36m Warning FailedScheduling pod/prometheus-adapter-54b8f8766-wzjld Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-wzjld": pod prometheus-adapter-54b8f8766-wzjld is already assigned to node "worker-2" 36m Warning FailedScheduling pod/prometheus-adapter-54b8f8766-wzjld Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-wzjld": pod prometheus-adapter-54b8f8766-wzjld is already assigned to node "worker-2" $ oc get events | grep "FailedScheduling" | wc -l 230 It is constantly creating pods that fail to schedule. The ReplicaSets are scaling up and down all the time. Unfortunately, the event aggregation makes them hard to get looking back but: $ oc get events | grep "Scale" 3h17m Normal ScalingReplicaSet deployment/cluster-monitoring-operator Scaled up replica set cluster-monitoring-operator-7849456b55 to 1 3h8m Normal ScalingReplicaSet deployment/grafana Scaled up replica set grafana-b87f78c64 to 1 3h8m Normal ScalingReplicaSet deployment/kube-state-metrics Scaled up replica set kube-state-metrics-7594bf4d85 to 1 3h7m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-7f7d4579d6 to 2 3h3m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-8568cd9cb7 to 1 3h3m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-7f7d4579d6 to 1 3h3m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-8568cd9cb7 to 2 3h3m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-7f7d4579d6 to 0 3h3m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-6987d4848c to 1 3h2m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-8568cd9cb7 to 1 3h2m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-6987d4848c to 2 3h2m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-8568cd9cb7 to 0 178m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-5c769fcf97 to 1 178m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-6987d4848c to 1 178m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-5c769fcf97 to 2 177m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-6987d4848c to 0 173m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-c4d8ccd84 to 1 173m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-5c769fcf97 to 1 173m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-c4d8ccd84 to 2 173m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled down replica set prometheus-adapter-5c769fcf97 to 0 173m Normal ScalingReplicaSet deployment/prometheus-adapter Scaled up replica set prometheus-adapter-78bfcc86cf to 1 3m5s Normal ScalingReplicaSet deployment/prometheus-adapter (combined from similar events): Scaled up replica set prometheus-adapter-855c99888b to 1 3h8m Normal ScalingReplicaSet deployment/prometheus-operator Scaled up replica set prometheus-operator-7f9764d484 to 1 3h7m Normal ScalingReplicaSet deployment/prometheus-operator Scaled up replica set prometheus-operator-7b5b64994b to 1 3h7m Normal ScalingReplicaSet deployment/prometheus-operator Scaled down replica set prometheus-operator-7f9764d484 to 0 3h8m Normal ScalingReplicaSet deployment/telemeter-client Scaled up replica set telemeter-client-574fbd95d7 to 1 $ oc logs service-serving-cert-signer-8596d76f64-695zp | grep prometheus-adapter I0416 16:38:28.248591 1 option.go:47] ServiceServingCertController: handling add openshift-monitoring/prometheus-adapter I0416 16:38:28.249085 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:38:28.552519 1 option.go:47] ServiceServingCertUpdateController: handling add openshift-monitoring/prometheus-adapter-tls I0416 16:38:28.564916 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:40:05.064183 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:40:05.064344 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:40:05.343637 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:40:05.348552 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:40:05.354749 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:43:34.059586 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:43:34.059659 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:43:34.713641 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:43:34.722727 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:43:34.729754 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:43:58.663762 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:43:58.663867 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:43:59.382456 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:43:59.388901 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:43:59.395660 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:48:34.060575 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:48:34.060736 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:48:34.176750 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:48:34.183350 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:48:34.192142 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:49:05.523197 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:49:05.523317 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:49:05.666238 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:49:05.672110 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:49:05.679122 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:50:56.470442 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:53:34.063448 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:53:34.063505 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:53:34.623259 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:53:34.629366 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:53:34.637569 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:53:58.614915 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:53:58.614996 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:53:59.189578 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:53:59.195407 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:53:59.203471 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:58:01.036778 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:58:34.061142 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter I0416 16:58:34.061211 1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter I0416 16:58:34.163834 1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls I0416 16:58:34.169139 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls I0416 16:58:34.177235 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter Appears to be generating a new cert every ~4m but sometimes as frequently as ~30s $ oc logs service-serving-cert-signer-8596d76f64-695zp | grep "generating new cert" | cut -f15- -d' ' | sort | uniq -c
1 openshift-apiserver/api
1 openshift-apiserver-operator/metrics
1 openshift-authentication/openshift-authentication
1 openshift-authentication-operator/metrics
1 openshift-console/console
1 openshift-controller-manager/controller-manager
1 openshift-controller-manager-operator/metrics
1 openshift-image-registry/image-registry
1 openshift-ingress/router-internal-default
1 openshift-kube-apiserver/apiserver
1 openshift-kube-apiserver-operator/metrics
1 openshift-kube-controller-manager/kube-controller-manager
1 openshift-kube-controller-manager-operator/metrics
1 openshift-kube-scheduler-operator/metrics
1 openshift-kube-scheduler/scheduler
42 openshift-monitoring/alertmanager-main
42 openshift-monitoring/grafana
42 openshift-monitoring/kube-state-metrics
42 openshift-monitoring/node-exporter
42 openshift-monitoring/prometheus-adapter
42 openshift-monitoring/prometheus-k8s
42 openshift-monitoring/telemeter-client
1 openshift-operator-lifecycle-manager/catalog-operator-metrics
1 openshift-operator-lifecycle-manager/olm-operator-metrics
1 openshift-service-catalog-apiserver-operator/metrics
1 openshift-service-catalog-controller-manager-operator/metrics
Seems this is happening on a ton of monitoring components
After some further debugging with Sergiusz, we found that since the CMO is rebuilding its services, it is removing the signed-by annotation that the service-ca relies upon for its regeneration check (when it is missing we regenerate). Since the service-ca owns the secrets it creates it does not own the services, so it would be difficult to always rely on this annotation. I have a change that checks the cert directly for the signed-by information instead: https://github.com/openshift/service-ca-operator/pull/44 Verified.
The cluster created about 155m.
A new cert generated every ~20m
$ oc logs service-serving-cert-signer-6599f5b45-krvnt -n openshift-service-ca | grep prometheus-adapter
I0430 03:14:10.660418 1 option.go:47] ServiceServingCertController: handling add openshift-monitoring/prometheus-adapter
I0430 03:14:10.660474 1 secret_creating_controller.go:125] generating new cert for openshift-monitoring/prometheus-adapter
I0430 03:14:10.952047 1 option.go:47] ServiceServingCertUpdateController: handling add openshift-monitoring/prometheus-adapter-tls
I0430 03:14:10.955576 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:14:51.327150 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:28:16.475205 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 03:33:22.787170 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:37:35.345387 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:37:35.559284 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 03:57:35.344966 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:57:35.560222 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 04:17:35.345552 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 04:17:35.560961 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 04:37:35.345722 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 04:37:35.563731 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 04:57:35.346625 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 04:57:35.563744 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 05:17:35.347008 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 05:17:35.565947 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 05:37:35.347656 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 05:37:35.567086 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 05:57:35.347145 1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 05:57:35.574542 1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
$ oc logs service-serving-cert-signer-6599f5b45-krvnt -n openshift-service-ca | grep "generating new cert" | cut -f15- -d' ' | sort | uniq -c
2 metering/reporting-operator
2 metering/reporting-operator-metrics
1 openshift-apiserver/api
1 openshift-apiserver-operator/metrics
1 openshift-authentication/integrated-oauth-server
1 openshift-authentication-operator/metrics
1 openshift-console/console
1 openshift-controller-manager/controller-manager
1 openshift-controller-manager-operator/metrics
1 openshift-image-registry/image-registry
1 openshift-ingress/router-internal-default
1 openshift-kube-apiserver/apiserver
1 openshift-kube-apiserver-operator/metrics
1 openshift-kube-controller-manager/kube-controller-manager
1 openshift-kube-controller-manager-operator/metrics
1 openshift-kube-scheduler-operator/metrics
1 openshift-kube-scheduler/scheduler
1 openshift-logging/elasticsearch-metrics
1 openshift-logging/fluentd
6 openshift-metering/reporting-operator
3 openshift-metering/reporting-operator-metrics
1 openshift-monitoring/alertmanager-main
1 openshift-monitoring/grafana
1 openshift-monitoring/kube-state-metrics
1 openshift-monitoring/node-exporter
1 openshift-monitoring/prometheus-adapter
1 openshift-monitoring/prometheus-k8s
1 openshift-monitoring/telemeter-client
1 openshift-operator-lifecycle-manager/catalog-operator-metrics
1 openshift-operator-lifecycle-manager/olm-operator-metrics
1 openshift-service-catalog-apiserver/api
1 openshift-service-catalog-apiserver-operator/metrics
1 openshift-service-catalog-controller-manager/controller-manager
1 openshift-service-catalog-controller-manager-operator/metrics
1 template-service-broker/apiserver
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.1.0-0.nightly-2019-04-25-121505 True False 157m Cluster version is 4.1.0-0.nightly-2019-04-25-121505
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758 |
I've observed 60 generations of CMO managed deployments in a cluster that was installed 150m ago and has sat idle since installation: $ oc get deployment -oyaml | grep -e "^ name: " -e "observedGeneration: " name: cluster-monitoring-operator observedGeneration: 42 name: grafana observedGeneration: 60 name: kube-state-metrics observedGeneration: 60 name: prometheus-adapter observedGeneration: 59 name: prometheus-operator observedGeneration: 60 name: telemeter-client observedGeneration: 60 $ oc get clusteroperator monitoring NAME VERSION AVAILABLE PROGRESSING FAILING SINCE monitoring 4.0.0-0.ci-2019-04-15-011801 True False False 146m $ ogcv NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.ci-2019-04-15-011801 True False 142m Cluster version is 4.0.0-0.ci-2019-04-15-011801 Even the CMO itself is on generation 42, which is strange.