1700037 – CMO deployments are changing all the time

Bug 1700037 - CMO deployments are changing all the time

Summary: CMO deployments are changing all the time

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	apiserver-auth
Sub Component:
Version:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	4.1.0
Assignee:	Matt Rogers
QA Contact:	Chuan Yu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-04-15 16:12 UTC by Seth Jennings
Modified:	2019-06-04 10:47 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-06-04 10:47:37 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift cluster-monitoring-operator pull 328	'None'	closed	pkg/client: compare Service object before updating it	2020-02-27 19:15:41 UTC
Github	openshift cluster-monitoring-operator pull 329	'None'	closed	pkg/client: compare only parts of Service resource instead of full object	2020-02-27 19:15:41 UTC
Github	openshift service-ca-operator pull 44	'None'	closed	Bug 1700037: Check cert issuer directly	2020-02-27 19:15:41 UTC
Red Hat Product Errata	RHBA-2019:0758	None	None	None	2019-06-04 10:47:43 UTC

Description Seth Jennings 2019-04-15 16:12:15 UTC

I've observed 60 generations of CMO managed deployments in a cluster that was installed 150m ago and has sat idle since installation:

$ oc get deployment -oyaml | grep -e "^    name: " -e "observedGeneration: "
    name: cluster-monitoring-operator
    observedGeneration: 42
    name: grafana
    observedGeneration: 60
    name: kube-state-metrics
    observedGeneration: 60
    name: prometheus-adapter
    observedGeneration: 59
    name: prometheus-operator
    observedGeneration: 60
    name: telemeter-client
    observedGeneration: 60

$ oc get clusteroperator monitoring 
NAME         VERSION                        AVAILABLE   PROGRESSING   FAILING   SINCE
monitoring   4.0.0-0.ci-2019-04-15-011801   True        False         False     146m

$ ogcv
NAME      VERSION                        AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.ci-2019-04-15-011801   True        False         142m    Cluster version is 4.0.0-0.ci-2019-04-15-011801

Even the CMO itself is on generation 42, which is strange.

Comment 1 Frederic Branczyk 2019-04-15 16:42:53 UTC

The CMO itself is strange indeed as we never touch that deployment it's entirely handled by the CVO. The rest is likely because we do perform reconciles every 5mins, and reacting to various configuration changes in the cluster. I understand this might not be perfect, but is priority high appropriate?

Comment 2 Seth Jennings 2019-04-15 16:53:02 UTC

I can drop it a notch.

The prom adapter is a little worse off than the others:

$ oc get events | grep "Created pod"
3h5m        Normal    SuccessfulCreate       replicaset/cluster-monitoring-operator-7849456b55   Created pod: cluster-monitoring-operator-7849456b55-xv284
3h4m        Normal    SuccessfulCreate       replicaset/grafana-b87f78c64                        Created pod: grafana-b87f78c64-hkddd
3h5m        Normal    SuccessfulCreate       replicaset/kube-state-metrics-7594bf4d85            Created pod: kube-state-metrics-7594bf4d85-t4f6c
3h5m        Normal    SuccessfulCreate       daemonset/node-exporter                             Created pod: node-exporter-wqkh6
3h5m        Normal    SuccessfulCreate       daemonset/node-exporter                             Created pod: node-exporter-2frkw
3h5m        Normal    SuccessfulCreate       daemonset/node-exporter                             Created pod: node-exporter-jvzhd
3h5m        Normal    SuccessfulCreate       daemonset/node-exporter                             Created pod: node-exporter-tdxdc
3h5m        Normal    SuccessfulCreate       daemonset/node-exporter                             Created pod: node-exporter-z46zl
56m         Normal    SuccessfulCreate       daemonset/node-exporter                             Created pod: node-exporter-95cjw
34m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-54b8f8766             Created pod: prometheus-adapter-54b8f8766-wzjld
34m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-54b8f8766             Created pod: prometheus-adapter-54b8f8766-dspz7
100m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-54c4d44786            Created pod: prometheus-adapter-54c4d44786-4t9q7
99m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-54c4d44786            Created pod: prometheus-adapter-54c4d44786-sbr9g
2s          Normal    SuccessfulCreate       replicaset/prometheus-adapter-5555fdcbf4            Created pod: prometheus-adapter-5555fdcbf4-zf5gc
54m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-558f8b4bdc            Created pod: prometheus-adapter-558f8b4bdc-wflzr
54m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-558f8b4bdc            Created pod: prometheus-adapter-558f8b4bdc-bfj84
9m30s       Normal    SuccessfulCreate       replicaset/prometheus-adapter-5595c65667            Created pod: prometheus-adapter-5595c65667-4xblc
9m21s       Normal    SuccessfulCreate       replicaset/prometheus-adapter-5595c65667            Created pod: prometheus-adapter-5595c65667-crxrs
139m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5656d4b88c            Created pod: prometheus-adapter-5656d4b88c-gnx45
139m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5656d4b88c            Created pod: prometheus-adapter-5656d4b88c-qxhms
64m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-57f8679d7f            Created pod: prometheus-adapter-57f8679d7f-x4jtz
64m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-57f8679d7f            Created pod: prometheus-adapter-57f8679d7f-l6w99
44m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-594fdb5f86            Created pod: prometheus-adapter-594fdb5f86-sp5g9
44m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-594fdb5f86            Created pod: prometheus-adapter-594fdb5f86-hlzxz
164m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-598d8599d7            Created pod: prometheus-adapter-598d8599d7-tfdtc
164m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-598d8599d7            Created pod: prometheus-adapter-598d8599d7-9nrtt
154m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5bcfff679d            Created pod: prometheus-adapter-5bcfff679d-sc8md
154m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5bcfff679d            Created pod: prometheus-adapter-5bcfff679d-th2c7
130m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5c5cd8d75d            Created pod: prometheus-adapter-5c5cd8d75d-sqq7c
129m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5c5cd8d75d            Created pod: prometheus-adapter-5c5cd8d75d-6b8bp
174m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5c769fcf97            Created pod: prometheus-adapter-5c769fcf97-qzqc7
174m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5c769fcf97            Created pod: prometheus-adapter-5c769fcf97-zt7tf
110m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5c88685d54            Created pod: prometheus-adapter-5c88685d54-x8cf4
109m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5c88685d54            Created pod: prometheus-adapter-5c88685d54-6vsjv
119m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5d74d6cbf7            Created pod: prometheus-adapter-5d74d6cbf7-hbc6l
119m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5d74d6cbf7            Created pod: prometheus-adapter-5d74d6cbf7-t66g8
145m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5db8866879            Created pod: prometheus-adapter-5db8866879-h94cj
144m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-5db8866879            Created pod: prometheus-adapter-5db8866879-689rn
29m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-5fdd486b9f            Created pod: prometheus-adapter-5fdd486b9f-drzzf
29m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-5fdd486b9f            Created pod: prometheus-adapter-5fdd486b9f-vhsvz
95m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-64649b86f             Created pod: prometheus-adapter-64649b86f-5xj6z
94m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-64649b86f             Created pod: prometheus-adapter-64649b86f-7xcjw
15m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-646c8bfc7b            Created pod: prometheus-adapter-646c8bfc7b-hlpz5
14m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-646c8bfc7b            Created pod: prometheus-adapter-646c8bfc7b-p6jdz
45m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-65bbf89bd4            Created pod: prometheus-adapter-65bbf89bd4-w6cj2
44m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-65bbf89bd4            Created pod: prometheus-adapter-65bbf89bd4-49gjg
140m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-66746859b4            Created pod: prometheus-adapter-66746859b4-4h5df
139m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-66746859b4            Created pod: prometheus-adapter-66746859b4-msq7k
85m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-667f7b64c9            Created pod: prometheus-adapter-667f7b64c9-h54wp
84m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-667f7b64c9            Created pod: prometheus-adapter-667f7b64c9-x6bzr
124m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-674f5cc946            Created pod: prometheus-adapter-674f5cc946-xzvhd
124m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-674f5cc946            Created pod: prometheus-adapter-674f5cc946-tmwgr
165m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-67db5fcd97            Created pod: prometheus-adapter-67db5fcd97-nv8ws
164m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-67db5fcd97            Created pod: prometheus-adapter-67db5fcd97-gcbsj
75m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-67fcd6d495            Created pod: prometheus-adapter-67fcd6d495-2pwnp
74m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-67fcd6d495            Created pod: prometheus-adapter-67fcd6d495-7dg26
59m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-68555ccbbb            Created pod: prometheus-adapter-68555ccbbb-pzjcr
59m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-68555ccbbb            Created pod: prometheus-adapter-68555ccbbb-njkf8
94m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-68757485f7            Created pod: prometheus-adapter-68757485f7-z4jpm
94m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-68757485f7            Created pod: prometheus-adapter-68757485f7-vkqgj
179m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-6987d4848c            Created pod: prometheus-adapter-6987d4848c-s4dkw
179m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-6987d4848c            Created pod: prometheus-adapter-6987d4848c-cmql2
55m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6b5d84b5b8            Created pod: prometheus-adapter-6b5d84b5b8-zlcld
54m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6b5d84b5b8            Created pod: prometheus-adapter-6b5d84b5b8-8ms79
65m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6bb6d8d754            Created pod: prometheus-adapter-6bb6d8d754-tgzl5
64m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6bb6d8d754            Created pod: prometheus-adapter-6bb6d8d754-bd2hc
39m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6c7548dbc7            Created pod: prometheus-adapter-6c7548dbc7-fw6mv
39m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6c7548dbc7            Created pod: prometheus-adapter-6c7548dbc7-956qw
155m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-6f87876dff            Created pod: prometheus-adapter-6f87876dff-r6d9g
154m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-6f87876dff            Created pod: prometheus-adapter-6f87876dff-hghpc
129m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-6fb6c47bcf            Created pod: prometheus-adapter-6fb6c47bcf-tcdz4
129m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-6fb6c47bcf            Created pod: prometheus-adapter-6fb6c47bcf-fdwj5
50m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6fd9b56bbc            Created pod: prometheus-adapter-6fd9b56bbc-htgd2
49m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-6fd9b56bbc            Created pod: prometheus-adapter-6fd9b56bbc-qzvzb
160m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-748587c859            Created pod: prometheus-adapter-748587c859-d669n
159m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-748587c859            Created pod: prometheus-adapter-748587c859-994hq
125m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-75554fb858            Created pod: prometheus-adapter-75554fb858-mvj2l
124m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-75554fb858            Created pod: prometheus-adapter-75554fb858-56ctf
30m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-76dcd75d74            Created pod: prometheus-adapter-76dcd75d74-75pcw
29m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-76dcd75d74            Created pod: prometheus-adapter-76dcd75d74-xvxdh
120m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-774bd5454c            Created pod: prometheus-adapter-774bd5454c-j5sbz
119m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-774bd5454c            Created pod: prometheus-adapter-774bd5454c-9f4l7
169m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-78bfcc86cf            Created pod: prometheus-adapter-78bfcc86cf-gvn5f
169m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-78bfcc86cf            Created pod: prometheus-adapter-78bfcc86cf-kxmsb
80m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-78c486c8b5            Created pod: prometheus-adapter-78c486c8b5-smpcq
79m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-78c486c8b5            Created pod: prometheus-adapter-78c486c8b5-scbbw
150m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-78c7cbf8f9            Created pod: prometheus-adapter-78c7cbf8f9-wl28n
149m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-78c7cbf8f9            Created pod: prometheus-adapter-78c7cbf8f9-pj7dv
49m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-78f9f96d8             Created pod: prometheus-adapter-78f9f96d8-r6gnv
49m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-78f9f96d8             Created pod: prometheus-adapter-78f9f96d8-pfqr5
24m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7944c4c899            Created pod: prometheus-adapter-7944c4c899-pbtt6
24m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7944c4c899            Created pod: prometheus-adapter-7944c4c899-txd9l
105m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-7bf5dc88cd            Created pod: prometheus-adapter-7bf5dc88cd-sldjr
104m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-7bf5dc88cd            Created pod: prometheus-adapter-7bf5dc88cd-zvq25
134m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-7c748db4c8            Created pod: prometheus-adapter-7c748db4c8-76qbb
134m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-7c748db4c8            Created pod: prometheus-adapter-7c748db4c8-pgsjq
19m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7cf87db645            Created pod: prometheus-adapter-7cf87db645-zl9s4
19m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7cf87db645            Created pod: prometheus-adapter-7cf87db645-8frdf
74m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7d79864dc5            Created pod: prometheus-adapter-7d79864dc5-4vpl4
74m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7d79864dc5            Created pod: prometheus-adapter-7d79864dc5-bds9m
69m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7f4fb579c8            Created pod: prometheus-adapter-7f4fb579c8-v5hww
69m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-7f4fb579c8            Created pod: prometheus-adapter-7f4fb579c8-4ld9q
3h3m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-7f7d4579d6            Created pod: prometheus-adapter-7f7d4579d6-8gg48
3h3m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-7f7d4579d6            Created pod: prometheus-adapter-7f7d4579d6-xpmn2
3h          Normal    SuccessfulCreate       replicaset/prometheus-adapter-8568cd9cb7            Created pod: prometheus-adapter-8568cd9cb7-5572f
179m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-8568cd9cb7            Created pod: prometheus-adapter-8568cd9cb7-89zv7
5m2s        Normal    SuccessfulCreate       replicaset/prometheus-adapter-86b4ccf78f            Created pod: prometheus-adapter-86b4ccf78f-bgx6h
4m54s       Normal    SuccessfulCreate       replicaset/prometheus-adapter-86b4ccf78f            Created pod: prometheus-adapter-86b4ccf78f-cz4ns
149m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-b57654fb5             Created pod: prometheus-adapter-b57654fb5-fzwlf
149m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-b57654fb5             Created pod: prometheus-adapter-b57654fb5-9vkcj
170m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-c4d8ccd84             Created pod: prometheus-adapter-c4d8ccd84-zqqcp
169m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-c4d8ccd84             Created pod: prometheus-adapter-c4d8ccd84-grd2p
114m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-c8c549648             Created pod: prometheus-adapter-c8c549648-v6l8l
114m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-c8c549648             Created pod: prometheus-adapter-c8c549648-z9d2x
10m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-c8f99fdfc             Created pod: prometheus-adapter-c8f99fdfc-m6m7d
9m53s       Normal    SuccessfulCreate       replicaset/prometheus-adapter-c8f99fdfc             Created pod: prometheus-adapter-c8f99fdfc-v4vxn
20m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-c9c9c4b6f             Created pod: prometheus-adapter-c9c9c4b6f-mzjcn
19m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-c9c9c4b6f             Created pod: prometheus-adapter-c9c9c4b6f-tnjp6
89m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-d888cc874             Created pod: prometheus-adapter-d888cc874-jvz42
89m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-d888cc874             Created pod: prometheus-adapter-d888cc874-zdhx5
135m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-d8dc5fbbc             Created pod: prometheus-adapter-d8dc5fbbc-2sbfq
134m        Normal    SuccessfulCreate       replicaset/prometheus-adapter-d8dc5fbbc             Created pod: prometheus-adapter-d8dc5fbbc-fz2qg
99m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-dd786b994             Created pod: prometheus-adapter-dd786b994-khvp8
99m         Normal    SuccessfulCreate       replicaset/prometheus-adapter-dd786b994             Created pod: prometheus-adapter-dd786b994-b4lqz
3h3m        Normal    SuccessfulCreate       replicaset/prometheus-operator-7b5b64994b           Created pod: prometheus-operator-7b5b64994b-hvcb4
3h5m        Normal    SuccessfulCreate       replicaset/prometheus-operator-7f9764d484           Created pod: prometheus-operator-7f9764d484-2896h
3h5m        Normal    SuccessfulCreate       replicaset/telemeter-client-574fbd95d7              Created pod: telemeter-client-574fbd95d7-hbck4

Creating a lot of pods, most of which fail to schedule with events like
36m         Warning   FailedScheduling       pod/prometheus-adapter-54b8f8766-dspz7              Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-dspz7": pod prometheus-adapter-54b8f8766-dspz7 is already assigned to node "worker-0"
36m         Warning   FailedScheduling       pod/prometheus-adapter-54b8f8766-dspz7              Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-dspz7": pod prometheus-adapter-54b8f8766-dspz7 is already assigned to node "worker-0"
36m         Warning   FailedScheduling       pod/prometheus-adapter-54b8f8766-wzjld              Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-wzjld": pod prometheus-adapter-54b8f8766-wzjld is already assigned to node "worker-2"
36m         Warning   FailedScheduling       pod/prometheus-adapter-54b8f8766-wzjld              Binding rejected: Operation cannot be fulfilled on pods/binding "prometheus-adapter-54b8f8766-wzjld": pod prometheus-adapter-54b8f8766-wzjld is already assigned to node "worker-2"

$ oc get events | grep "FailedScheduling" | wc -l
230

It is constantly creating pods that fail to schedule.  The ReplicaSets are scaling up and down all the time. Unfortunately, the event aggregation makes them hard to get looking back but:

$ oc get events | grep "Scale"
3h17m       Normal    ScalingReplicaSet      deployment/cluster-monitoring-operator              Scaled up replica set cluster-monitoring-operator-7849456b55 to 1
3h8m        Normal    ScalingReplicaSet      deployment/grafana                                  Scaled up replica set grafana-b87f78c64 to 1
3h8m        Normal    ScalingReplicaSet      deployment/kube-state-metrics                       Scaled up replica set kube-state-metrics-7594bf4d85 to 1
3h7m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-7f7d4579d6 to 2
3h3m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-8568cd9cb7 to 1
3h3m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-7f7d4579d6 to 1
3h3m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-8568cd9cb7 to 2
3h3m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-7f7d4579d6 to 0
3h3m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-6987d4848c to 1
3h2m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-8568cd9cb7 to 1
3h2m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-6987d4848c to 2
3h2m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-8568cd9cb7 to 0
178m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-5c769fcf97 to 1
178m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-6987d4848c to 1
178m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-5c769fcf97 to 2
177m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-6987d4848c to 0
173m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-c4d8ccd84 to 1
173m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-5c769fcf97 to 1
173m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-c4d8ccd84 to 2
173m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled down replica set prometheus-adapter-5c769fcf97 to 0
173m        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       Scaled up replica set prometheus-adapter-78bfcc86cf to 1
3m5s        Normal    ScalingReplicaSet      deployment/prometheus-adapter                       (combined from similar events): Scaled up replica set prometheus-adapter-855c99888b to 1
3h8m        Normal    ScalingReplicaSet      deployment/prometheus-operator                      Scaled up replica set prometheus-operator-7f9764d484 to 1
3h7m        Normal    ScalingReplicaSet      deployment/prometheus-operator                      Scaled up replica set prometheus-operator-7b5b64994b to 1
3h7m        Normal    ScalingReplicaSet      deployment/prometheus-operator                      Scaled down replica set prometheus-operator-7f9764d484 to 0
3h8m        Normal    ScalingReplicaSet      deployment/telemeter-client                         Scaled up replica set telemeter-client-574fbd95d7 to 1

Comment 8 Seth Jennings 2019-04-16 18:20:12 UTC

$ oc logs service-serving-cert-signer-8596d76f64-695zp | grep prometheus-adapter
I0416 16:38:28.248591       1 option.go:47] ServiceServingCertController: handling add openshift-monitoring/prometheus-adapter
I0416 16:38:28.249085       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:38:28.552519       1 option.go:47] ServiceServingCertUpdateController: handling add openshift-monitoring/prometheus-adapter-tls
I0416 16:38:28.564916       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:40:05.064183       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:40:05.064344       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:40:05.343637       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:40:05.348552       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:40:05.354749       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:43:34.059586       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:43:34.059659       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:43:34.713641       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:43:34.722727       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:43:34.729754       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:43:58.663762       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:43:58.663867       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:43:59.382456       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:43:59.388901       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:43:59.395660       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:48:34.060575       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:48:34.060736       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:48:34.176750       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:48:34.183350       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:48:34.192142       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:49:05.523197       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:49:05.523317       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:49:05.666238       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:49:05.672110       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:49:05.679122       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:50:56.470442       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:53:34.063448       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:53:34.063505       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:53:34.623259       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:53:34.629366       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:53:34.637569       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:53:58.614915       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:53:58.614996       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:53:59.189578       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:53:59.195407       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:53:59.203471       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:58:01.036778       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:58:34.061142       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0416 16:58:34.061211       1 secret_creating_controller.go:124] generating new cert for openshift-monitoring/prometheus-adapter
I0416 16:58:34.163834       1 secret_creating_controller.go:148] renewing cert in existing secret openshift-monitoring/prometheus-adapter-tls
I0416 16:58:34.169139       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0416 16:58:34.177235       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter

Appears to be generating a new cert every ~4m  but sometimes as frequently as ~30s

Comment 9 Seth Jennings 2019-04-16 18:23:27 UTC

$ oc logs service-serving-cert-signer-8596d76f64-695zp | grep "generating new cert" | cut -f15- -d' ' | sort | uniq -c
      1 openshift-apiserver/api
      1 openshift-apiserver-operator/metrics
      1 openshift-authentication/openshift-authentication
      1 openshift-authentication-operator/metrics
      1 openshift-console/console
      1 openshift-controller-manager/controller-manager
      1 openshift-controller-manager-operator/metrics
      1 openshift-image-registry/image-registry
      1 openshift-ingress/router-internal-default
      1 openshift-kube-apiserver/apiserver
      1 openshift-kube-apiserver-operator/metrics
      1 openshift-kube-controller-manager/kube-controller-manager
      1 openshift-kube-controller-manager-operator/metrics
      1 openshift-kube-scheduler-operator/metrics
      1 openshift-kube-scheduler/scheduler
     42 openshift-monitoring/alertmanager-main
     42 openshift-monitoring/grafana
     42 openshift-monitoring/kube-state-metrics
     42 openshift-monitoring/node-exporter
     42 openshift-monitoring/prometheus-adapter
     42 openshift-monitoring/prometheus-k8s
     42 openshift-monitoring/telemeter-client
      1 openshift-operator-lifecycle-manager/catalog-operator-metrics
      1 openshift-operator-lifecycle-manager/olm-operator-metrics
      1 openshift-service-catalog-apiserver-operator/metrics
      1 openshift-service-catalog-controller-manager-operator/metrics

Seems this is happening on a ton of monitoring components

Comment 12 Matt Rogers 2019-04-17 19:00:15 UTC

After some further debugging with Sergiusz, we found that since the CMO is rebuilding its services, it is removing the signed-by annotation that the service-ca relies upon for its regeneration check (when it is missing we regenerate). Since the service-ca owns the secrets it creates it does not own the services, so it would be difficult to always rely on this annotation. I have a change that checks the cert directly for the signed-by information instead: https://github.com/openshift/service-ca-operator/pull/44

Comment 14 Chuan Yu 2019-04-30 06:14:30 UTC

Verified.

The cluster created about 155m.

A new cert generated every ~20m

$ oc logs service-serving-cert-signer-6599f5b45-krvnt -n openshift-service-ca | grep prometheus-adapter
I0430 03:14:10.660418       1 option.go:47] ServiceServingCertController: handling add openshift-monitoring/prometheus-adapter
I0430 03:14:10.660474       1 secret_creating_controller.go:125] generating new cert for openshift-monitoring/prometheus-adapter
I0430 03:14:10.952047       1 option.go:47] ServiceServingCertUpdateController: handling add openshift-monitoring/prometheus-adapter-tls
I0430 03:14:10.955576       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:14:51.327150       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:28:16.475205       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 03:33:22.787170       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:37:35.345387       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:37:35.559284       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 03:57:35.344966       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 03:57:35.560222       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 04:17:35.345552       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 04:17:35.560961       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 04:37:35.345722       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 04:37:35.563731       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 04:57:35.346625       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 04:57:35.563744       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 05:17:35.347008       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 05:17:35.565947       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 05:37:35.347656       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 05:37:35.567086       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls
I0430 05:57:35.347145       1 option.go:55] ServiceServingCertController: handling update openshift-monitoring/prometheus-adapter
I0430 05:57:35.574542       1 option.go:55] ServiceServingCertUpdateController: handling update openshift-monitoring/prometheus-adapter-tls


$ oc logs service-serving-cert-signer-6599f5b45-krvnt -n openshift-service-ca | grep "generating new cert" | cut -f15- -d' ' | sort | uniq -c
      2 metering/reporting-operator
      2 metering/reporting-operator-metrics
      1 openshift-apiserver/api
      1 openshift-apiserver-operator/metrics
      1 openshift-authentication/integrated-oauth-server
      1 openshift-authentication-operator/metrics
      1 openshift-console/console
      1 openshift-controller-manager/controller-manager
      1 openshift-controller-manager-operator/metrics
      1 openshift-image-registry/image-registry
      1 openshift-ingress/router-internal-default
      1 openshift-kube-apiserver/apiserver
      1 openshift-kube-apiserver-operator/metrics
      1 openshift-kube-controller-manager/kube-controller-manager
      1 openshift-kube-controller-manager-operator/metrics
      1 openshift-kube-scheduler-operator/metrics
      1 openshift-kube-scheduler/scheduler
      1 openshift-logging/elasticsearch-metrics
      1 openshift-logging/fluentd
      6 openshift-metering/reporting-operator
      3 openshift-metering/reporting-operator-metrics
      1 openshift-monitoring/alertmanager-main
      1 openshift-monitoring/grafana
      1 openshift-monitoring/kube-state-metrics
      1 openshift-monitoring/node-exporter
      1 openshift-monitoring/prometheus-adapter
      1 openshift-monitoring/prometheus-k8s
      1 openshift-monitoring/telemeter-client
      1 openshift-operator-lifecycle-manager/catalog-operator-metrics
      1 openshift-operator-lifecycle-manager/olm-operator-metrics
      1 openshift-service-catalog-apiserver/api
      1 openshift-service-catalog-apiserver-operator/metrics
      1 openshift-service-catalog-controller-manager/controller-manager
      1 openshift-service-catalog-controller-manager-operator/metrics
      1 template-service-broker/apiserver


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.1.0-0.nightly-2019-04-25-121505   True        False         157m      Cluster version is 4.1.0-0.nightly-2019-04-25-121505

Comment 16 errata-xmlrpc 2019-06-04 10:47:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758

Note You need to log in before you can comment on or make changes to this bug.