Bug 1702158

Description of problem:
Got the below errors when enabling the Service Catalog:
MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found
MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found

Version-Release number of selected component (if applicable):
Cluster version is 4.1.0-0.nightly-2019-04-22-005054
Service Catalog apiserver operator image:
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a27e0a53ffb0c926f025bea48d2a537e2b0e521ce055a58e0a5ee3af719e8d
version: io.openshift.build.commit.id=0e8e9ffc5e874be13ee99c471216f930575c083b

Service Catalog apiserver image:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c
version: io.openshift.build.commit.id=1dd02c427d9a0b143e5f2a6ef05506a79a311c13

How reproducible:
always

Steps to Reproduce:
1. Install the OCP 4.1 with the payload: 4.1.0-0.nightly-2019-04-22-005054
2. Enable the service catalog apiserver, change `managementState` to `Managed` from `Removed`, like below:
mac:~ jianzhang$ oc edit servicecatalogapiserver
...
  managementState: Managed
...

3. Check the pods in the project "openshift-service-catalog-apiserver".

Actual results:
No pods generated.
mac:~ jianzhang$ oc get pods -n openshift-service-catalog-apiserver
NAME              READY     STATUS              RESTARTS   AGE
apiserver-7kh5s   0/1       ContainerCreating   0          159m
apiserver-w9h4d   0/1       ContainerCreating   0          159m
apiserver-xmw9x   0/1       ContainerCreating   0          159m

mac:~ jianzhang$ oc describe pods apiserver-7kh5s -n openshift-service-catalog-apiserver
...
Events:
  Type     Reason       Age                From                                                   Message
  ----     ------       ----               ----                                                   -------
  Normal   Scheduled    2h                 default-scheduler                                      Successfully assigned openshift-service-catalog-apiserver/apiserver-7kh5s to ip-172-31-131-234.us-west-2.compute.internal
  Warning  FailedMount  2h (x11 over 2h)   kubelet, ip-172-31-131-234.us-west-2.compute.internal  MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found
  Warning  FailedMount  14m (x64 over 2h)  kubelet, ip-172-31-131-234.us-west-2.compute.internal  Unable to mount volumes for pod "apiserver-7kh5s_openshift-service-catalog-apiserver(60d2f2e8-6576-11e9-8e4a-0ad5afc33fa6)": timeout expired waiting for volumes to attach or mount for pod "openshift-service-catalog-apiserver"/"apiserver-7kh5s". list of unmounted volumes=[etcd-serving-ca etcd-client]. list of unattached volumes=[apiserver-ssl etcd-host-cert data-dir etcd-serving-ca etcd-client service-catalog-apiserver-token-qv6ds]
  Warning  FailedMount  3m (x85 over 2h)   kubelet, ip-172-31-131-234.us-west-2.compute.internal  MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found

mac:~ jianzhang$ oc get cm
NAME                   DATA      AGE
aggregator-client-ca   1         3h4m
client-ca              1         3h4m
config                 4         3h3m
mac:~ jianzhang$ oc get secret
NAME                                        TYPE                                  DATA      AGE
builder-dockercfg-6pwpt                     kubernetes.io/dockercfg               1         3h4m
builder-token-l8kcj                         kubernetes.io/service-account-token   4         3h4m
builder-token-lcdhc                         kubernetes.io/service-account-token   4         3h4m
default-dockercfg-w7dgm                     kubernetes.io/dockercfg               1         3h4m
default-token-t77gv                         kubernetes.io/service-account-token   4         3h4m
default-token-ttcsb                         kubernetes.io/service-account-token   4         3h4m
deployer-dockercfg-mv6tg                    kubernetes.io/dockercfg               1         3h4m
deployer-token-m4wjk                        kubernetes.io/service-account-token   4         3h4m
deployer-token-v8ngn                        kubernetes.io/service-account-token   4         3h4m
service-catalog-apiserver-dockercfg-92vj8   kubernetes.io/dockercfg               1         3h4m
service-catalog-apiserver-token-g2jkd       kubernetes.io/service-account-token   4         3h4m
service-catalog-apiserver-token-qv6ds       kubernetes.io/service-account-token   4         3h4m
serving-cert                                kubernetes.io/tls                     2         3h4m
mac:~ jianzhang$ oc get rolebinding
NAME                    AGE
system:deployers        3h
system:image-builders   3h
system:image-pullers    3h


Expected results:
The service catalog works well.

Additional info:
1, No corresponding apiservice generated:
mac:~ jianzhang$ oc get apiservice|grep catalog
mac:~ jianzhang$ 

2, Checked the apiserver operator logs, found the configmap `etcd-serving-ca` and secret `etcd-client` has been deleting. As below:
mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-operator-979f6d66b-m487d |grep etcd-serving-ca -A 3
I0423 03:15:04.955303       1 request.go:530] Throttling request took 391.920593ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:15:04.964860       1 request.go:530] Throttling request took 107.966264ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/service-catalog-namespace-viewer-binding
I0423 03:15:05.155309       1 request.go:530] Throttling request took 197.155469ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:15:05.164835       1 request.go:530] Throttling request took 197.353455ms, request: POST:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings
--
I0423 03:15:05.355311       1 request.go:530] Throttling request took 196.601887ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:15:05.364854       1 request.go:530] Throttling request took 193.27846ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/servicecatalog.k8s.io:service-catalog-readiness
I0423 03:15:05.555301       1 request.go:530] Throttling request took 196.969308ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:15:05.564835       1 request.go:530] Throttling request took 196.946719ms, request: POST:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles
--
I0423 03:15:24.669744       1 apps.go:109] DaemonSet "openshift-service-catalog-apiserver/apiserver" changes: {"metadata":{"annotations":{"operator.openshift.io/force":"093736c2-6576-11e9-b6c3-0a580a810017"}},"spec":{"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"operator.openshift.io/force":"093736c2-6576-11e9-b6c3-0a580a810017"}},"spec":{"containers":[{"args":["apiserver","--storage-type","etcd","--secure-port","6443","--etcd-servers","https://etcd.kube-system.svc.cluster.local:2379","--etcd-cafile","/var/run/configmaps/etcd-serving-ca/ca-bundle.crt","--etcd-certfile","/var/run/secrets/etcd-client/tls.crt","--etcd-keyfile","/var/run/secrets/etcd-client/tls.key","--cors-allowed-origins","localhost","--enable-admission-plugins","NamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck","--feature-gates","OriginatingIdentity=true","--feature-gates","NamespacedServiceBroker=true","-v=3"],"command":["/usr/bin/service-catalog"],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"apiserver","ports":[{"containerPort":6443,"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz/ready","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"requests":{"memory":"200Mi"}},"terminationMessagePath":"/dev/termination-log","volumeMounts":[{"mountPath":"/var/run/kubernetes-service-catalog","name":"apiserver-ssl","readOnly":true},{"mountPath":"/etc/origin/master","name":"etcd-host-cert","readOnly":true},{"mountPath":"/var/run/secrets/etcd-client","name":"etcd-client"},{"mountPath":"/var/run/configmaps/etcd-serving-ca","name":"etcd-serving-ca"}]}],"schedulerName":null,"serviceAccount":null,"volumes":[{"name":"apiserver-ssl","secret":{"defaultMode":420,"items":[{"key":"tls.crt","path":"apiserver.crt"},{"key":"tls.key","path":"apiserver.key"}],"secretName":"serving-cert"}},{"hostPath":{"path":"/etc/origin/master"},"name":"etcd-host-cert"},{"emptyDir":{},"name":"data-dir"},{"configMap":{"name":"etcd-serving-ca"},"name":"etcd-serving-ca"},{"name":"etcd-client","secret":{"secretName":"etcd-client"}}]}}}}
I0423 03:15:24.675281       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DaemonSetUpdated' Updated DaemonSet.apps/apiserver -n openshift-service-catalog-apiserver because it changed
I0423 03:15:24.682516       1 status_controller.go:156] clusteroperator/service-catalog-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-04-23T03:15:04Z","reason":"AsExpected","status":"False","type":"Failing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Progressing: daemonset/apiserver.openshift-operator: observed generation is 1, desired generation is 2.\nProgressing: servicecatalogapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2.","reason":"ProgressingDesiredStateNotYetAchieved","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Available: no openshift-svcat-apiserver daemon pods available on any node.","reason":"AvailableNoAPIServerPod","status":"False","type":"Available"},{"lastTransitionTime":"2019-04-23T03:15:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0423 03:15:24.690685       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator service-catalog-apiserver changed: Progressing changed from False to True ("Progressing: daemonset/apiserver.openshift-operator: observed generation is 1, desired generation is 2.\nProgressing: servicecatalogapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2."),Available changed from True to False ("Available: no openshift-svcat-apiserver daemon pods available on any node.")
--
I0423 03:15:25.134737       1 request.go:530] Throttling request took 197.075386ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:15:25.334739       1 request.go:530] Throttling request took 197.090665ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:15:31.733816       1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - *v1.ServiceAccount total 16 items received
I0423 03:15:43.742838       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock
--
I0423 03:16:24.941745       1 request.go:530] Throttling request took 97.24382ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:16:25.141748       1 request.go:530] Throttling request took 196.95793ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:16:25.341763       1 request.go:530] Throttling request took 196.861396ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:16:25.541776       1 request.go:530] Throttling request took 196.971297ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:16:43.766397       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock
I0423 03:16:44.046285       1 request.go:530] Throttling request took 175.362835ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/servicecatalog-serviceclass-viewer
--
I0423 03:17:26.381744       1 request.go:530] Throttling request took 197.212944ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:26.581728       1 request.go:530] Throttling request took 197.305913ms, request: PUT:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/aggregator-client-ca
I0423 03:17:26.585898       1 core.go:191] ConfigMap "openshift-service-catalog-apiserver/aggregator-client-ca" changes: {"data":{"ca-bundle.crt":"-----BEGIN CERTIFICATE-----\nMIIDPjCCAiagAwIBAgIBATANBgkqhkiG9w0BAQsFADBQMU4wTAYDVQQDDEVvcGVu\nc2hpZnQta3ViZS1hcGlzZXJ2ZXItb3BlcmF0b3JfYWdncmVnYXRvci1jbGllbnQt\nc2lnbmVyQDE1NTU5NzIxNjkwHhcNMTkwNDIyMjIyOTI4WhcNMTkwNTIyMjIyOTI5\nWjBQMU4wTAYDVQQDDEVvcGVuc2hpZnQta3ViZS1hcGlzZXJ2ZXItb3BlcmF0b3Jf\nYWdncmVnYXRvci1jbGllbnQtc2lnbmVyQDE1NTU5NzIxNjkwggEiMA0GCSqGSIb3\nDQEBAQUAA4IBDwAwggEKAoIBAQCsCpq4LdxXETpo5Uf0Ca8pKxcGUg0/1wlHMS3O\n0s9HQPwQNfc9xP+Lw2PLD3xtWuVVaAN4nB3tvK+RVKJ9sxpToZP7H52UYu+1rroQ\n+Csnoh1UPNvIEji1cs91jDtNXJfnRcvZnnwA557pFSBOcMxG+Yo5gwTj7I3WIvMW\nO0oskWGAPPiGZYXcecGZY4FlkD+RI/+c3DMSqarGLSi7maRUsJrXi2T8Yl7LRH66\nVLH49km777jvdHAwZ6xOj8WoWTCyPZdfbMkdgTyyv/hjE3XqL2d771pk34BOYlim\nyXeoypTrKzyplmCvd7lPRm6RP+LsVU45Mk28PnL9Jy3QOz21AgMBAAGjIzAhMA4G\nA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB\nAQCkT4/7HhdBciQDYwpOE+6Kx2J98P8Wt+OP2tuBfuJ4/1xbE9LAers8lwbwf1uH\nichS6kd4KNukqGAdZ2THegohuuvnLg0SSHhhyOWHE8IYIftA+JQ6kwmG7QTjEt/9\nXXcYej53xixnWim9dxew+NEJoxF57MTAL14emxLsJ9kxaD5pZOCB1n5WKNBXBvKG\n1e+uV8It0GdBIEiebDNpPHiegihKvwRS6yUuO0LMX6Bw6r+Ze1ZniXP9DXp+FzFl\nlDC+PPu5lSt4fr4Mq75wFZNQySg7WrC2sO6n3wNJS2H6zT1eNDMKVmk+XfAkPR4J\nbM8wP79h3PnKM5TBdsQbyh/R\n-----END CERTIFICATE-----\n"},"metadata":{"creationTimestamp":"2019-04-22T03:32:58Z","resourceVersion":null,"selfLink":"/api/v1/namespaces/openshift-config-managed/configmaps/kube-apiserver-aggregator-client-ca","uid":"53150f6e-64af-11e9-ad6f-022d39ca3376"}}
I0423 03:17:26.586237       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/aggregator-client-ca -n openshift-service-catalog-apiserver: cause by changes in data.ca-bundle.crt
--
I0423 03:17:26.981733       1 request.go:530] Throttling request took 196.876668ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:27.181731       1 request.go:530] Throttling request took 197.045183ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:27.381731       1 request.go:530] Throttling request took 69.196906ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:27.581738       1 request.go:530] Throttling request took 197.106734ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:27.982066       1 request.go:530] Throttling request took 65.504224ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:28.181737       1 request.go:530] Throttling request took 127.329771ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:28.381668       1 request.go:530] Throttling request took 196.865618ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:28.581729       1 request.go:530] Throttling request took 197.019546ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:28.781730       1 request.go:530] Throttling request took 196.922323ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:28.981726       1 request.go:530] Throttling request took 197.017925ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:29.181706       1 request.go:530] Throttling request took 196.856481ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:30.921173       1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - *v1.ConfigMap total 0 items received
W0423 03:17:30.922497       1 reflector.go:270] k8s.io/client-go/informers/factory.go:131: watch of *v1.ConfigMap ended with: too old resource version: 409719 (411277)
--
I0423 03:17:44.673465       1 apps.go:109] DaemonSet "openshift-service-catalog-apiserver/apiserver" changes: {"metadata":{"annotations":{"operator.openshift.io/force":"5caa6896-6576-11e9-b6c3-0a580a810017"}},"spec":{"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"operator.openshift.io/force":"5caa6896-6576-11e9-b6c3-0a580a810017"}},"spec":{"containers":[{"args":["apiserver","--storage-type","etcd","--secure-port","6443","--etcd-servers","https://etcd.kube-system.svc.cluster.local:2379","--etcd-cafile","/var/run/configmaps/etcd-serving-ca/ca-bundle.crt","--etcd-certfile","/var/run/secrets/etcd-client/tls.crt","--etcd-keyfile","/var/run/secrets/etcd-client/tls.key","--cors-allowed-origins","localhost","--enable-admission-plugins","NamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck","--feature-gates","OriginatingIdentity=true","--feature-gates","NamespacedServiceBroker=true","-v=3"],"command":["/usr/bin/service-catalog"],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"apiserver","ports":[{"containerPort":6443,"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz/ready","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"requests":{"memory":"200Mi"}},"terminationMessagePath":"/dev/termination-log","volumeMounts":[{"mountPath":"/var/run/kubernetes-service-catalog","name":"apiserver-ssl","readOnly":true},{"mountPath":"/etc/origin/master","name":"etcd-host-cert","readOnly":true},{"mountPath":"/var/run/secrets/etcd-client","name":"etcd-client"},{"mountPath":"/var/run/configmaps/etcd-serving-ca","name":"etcd-serving-ca"}]}],"schedulerName":null,"serviceAccount":null,"volumes":[{"name":"apiserver-ssl","secret":{"defaultMode":420,"items":[{"key":"tls.crt","path":"apiserver.crt"},{"key":"tls.key","path":"apiserver.key"}],"secretName":"serving-cert"}},{"hostPath":{"path":"/etc/origin/master"},"name":"etcd-host-cert"},{"emptyDir":{},"name":"data-dir"},{"configMap":{"name":"etcd-serving-ca"},"name":"etcd-serving-ca"},{"name":"etcd-client","secret":{"secretName":"etcd-client"}}]}}}}
I0423 03:17:44.679380       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DaemonSetUpdated' Updated DaemonSet.apps/apiserver -n openshift-service-catalog-apiserver because it changed
I0423 03:17:44.685813       1 status_controller.go:156] clusteroperator/service-catalog-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-04-23T03:15:04Z","reason":"AsExpected","status":"False","type":"Failing"},{"lastTransitionTime":"2019-04-23T03:17:44Z","message":"Progressing: daemonset/apiserver.openshift-operator: observed generation is 2, desired generation is 3.","reason":"ProgressingDesiredStateNotYetAchieved","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Available: no openshift-svcat-apiserver daemon pods available on any node.","reason":"AvailableNoAPIServerPod","status":"False","type":"Available"},{"lastTransitionTime":"2019-04-23T03:15:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0423 03:17:44.686196       1 reflector.go:215] k8s.io/client-go/informers/factory.go:131: forcing resync
--
I0423 03:17:44.789794       1 request.go:530] Throttling request took 122.00941ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:44.989787       1 request.go:530] Throttling request took 311.565648ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver
I0423 03:17:45.189787       1 request.go:530] Throttling request took 396.716236ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:45.389745       1 request.go:530] Throttling request took 396.967347ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver
--
I0423 03:17:45.590085       1 request.go:530] Throttling request took 397.360752ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:45.789760       1 request.go:530] Throttling request took 196.87415ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:46.189771       1 request.go:530] Throttling request took 87.618625ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:18:03.798182       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock
--
...