Bug 1702158 - Fail to enable the Service Catalog on OCP 4.1 due to the configmaps "etcd-serving-ca" not found
Summary: Fail to enable the Service Catalog on OCP 4.1 due to the configmaps "etcd-ser...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.1.0
Assignee: Dan Geoffroy
QA Contact: Jian Zhang
URL:
Whiteboard:
: 1708793 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-23 06:23 UTC by Jian Zhang
Modified: 2019-06-04 10:47 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:47:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:47:58 UTC

Description Jian Zhang 2019-04-23 06:23:04 UTC
Description of problem:
Got the below errors when enabling the Service Catalog:
MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found
MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found

Version-Release number of selected component (if applicable):
Cluster version is 4.1.0-0.nightly-2019-04-22-005054
Service Catalog apiserver operator image:
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a27e0a53ffb0c926f025bea48d2a537e2b0e521ce055a58e0a5ee3af719e8d
version: io.openshift.build.commit.id=0e8e9ffc5e874be13ee99c471216f930575c083b

Service Catalog apiserver image:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c
version: io.openshift.build.commit.id=1dd02c427d9a0b143e5f2a6ef05506a79a311c13

How reproducible:
always

Steps to Reproduce:
1. Install the OCP 4.1 with the payload: 4.1.0-0.nightly-2019-04-22-005054
2. Enable the service catalog apiserver, change `managementState` to `Managed` from `Removed`, like below:
mac:~ jianzhang$ oc edit servicecatalogapiserver
...
  managementState: Managed
...

3. Check the pods in the project "openshift-service-catalog-apiserver".

Actual results:
No pods generated.
mac:~ jianzhang$ oc get pods -n openshift-service-catalog-apiserver
NAME              READY     STATUS              RESTARTS   AGE
apiserver-7kh5s   0/1       ContainerCreating   0          159m
apiserver-w9h4d   0/1       ContainerCreating   0          159m
apiserver-xmw9x   0/1       ContainerCreating   0          159m

mac:~ jianzhang$ oc describe pods apiserver-7kh5s -n openshift-service-catalog-apiserver
...
Events:
  Type     Reason       Age                From                                                   Message
  ----     ------       ----               ----                                                   -------
  Normal   Scheduled    2h                 default-scheduler                                      Successfully assigned openshift-service-catalog-apiserver/apiserver-7kh5s to ip-172-31-131-234.us-west-2.compute.internal
  Warning  FailedMount  2h (x11 over 2h)   kubelet, ip-172-31-131-234.us-west-2.compute.internal  MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found
  Warning  FailedMount  14m (x64 over 2h)  kubelet, ip-172-31-131-234.us-west-2.compute.internal  Unable to mount volumes for pod "apiserver-7kh5s_openshift-service-catalog-apiserver(60d2f2e8-6576-11e9-8e4a-0ad5afc33fa6)": timeout expired waiting for volumes to attach or mount for pod "openshift-service-catalog-apiserver"/"apiserver-7kh5s". list of unmounted volumes=[etcd-serving-ca etcd-client]. list of unattached volumes=[apiserver-ssl etcd-host-cert data-dir etcd-serving-ca etcd-client service-catalog-apiserver-token-qv6ds]
  Warning  FailedMount  3m (x85 over 2h)   kubelet, ip-172-31-131-234.us-west-2.compute.internal  MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found

mac:~ jianzhang$ oc get cm
NAME                   DATA      AGE
aggregator-client-ca   1         3h4m
client-ca              1         3h4m
config                 4         3h3m
mac:~ jianzhang$ oc get secret
NAME                                        TYPE                                  DATA      AGE
builder-dockercfg-6pwpt                     kubernetes.io/dockercfg               1         3h4m
builder-token-l8kcj                         kubernetes.io/service-account-token   4         3h4m
builder-token-lcdhc                         kubernetes.io/service-account-token   4         3h4m
default-dockercfg-w7dgm                     kubernetes.io/dockercfg               1         3h4m
default-token-t77gv                         kubernetes.io/service-account-token   4         3h4m
default-token-ttcsb                         kubernetes.io/service-account-token   4         3h4m
deployer-dockercfg-mv6tg                    kubernetes.io/dockercfg               1         3h4m
deployer-token-m4wjk                        kubernetes.io/service-account-token   4         3h4m
deployer-token-v8ngn                        kubernetes.io/service-account-token   4         3h4m
service-catalog-apiserver-dockercfg-92vj8   kubernetes.io/dockercfg               1         3h4m
service-catalog-apiserver-token-g2jkd       kubernetes.io/service-account-token   4         3h4m
service-catalog-apiserver-token-qv6ds       kubernetes.io/service-account-token   4         3h4m
serving-cert                                kubernetes.io/tls                     2         3h4m
mac:~ jianzhang$ oc get rolebinding
NAME                    AGE
system:deployers        3h
system:image-builders   3h
system:image-pullers    3h


Expected results:
The service catalog works well.

Additional info:
1, No corresponding apiservice generated:
mac:~ jianzhang$ oc get apiservice|grep catalog
mac:~ jianzhang$ 

2, Checked the apiserver operator logs, found the configmap `etcd-serving-ca` and secret `etcd-client` has been deleting. As below:
mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-operator-979f6d66b-m487d |grep etcd-serving-ca -A 3
I0423 03:15:04.955303       1 request.go:530] Throttling request took 391.920593ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:15:04.964860       1 request.go:530] Throttling request took 107.966264ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/service-catalog-namespace-viewer-binding
I0423 03:15:05.155309       1 request.go:530] Throttling request took 197.155469ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:15:05.164835       1 request.go:530] Throttling request took 197.353455ms, request: POST:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings
--
I0423 03:15:05.355311       1 request.go:530] Throttling request took 196.601887ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:15:05.364854       1 request.go:530] Throttling request took 193.27846ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/servicecatalog.k8s.io:service-catalog-readiness
I0423 03:15:05.555301       1 request.go:530] Throttling request took 196.969308ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:15:05.564835       1 request.go:530] Throttling request took 196.946719ms, request: POST:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles
--
I0423 03:15:24.669744       1 apps.go:109] DaemonSet "openshift-service-catalog-apiserver/apiserver" changes: {"metadata":{"annotations":{"operator.openshift.io/force":"093736c2-6576-11e9-b6c3-0a580a810017"}},"spec":{"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"operator.openshift.io/force":"093736c2-6576-11e9-b6c3-0a580a810017"}},"spec":{"containers":[{"args":["apiserver","--storage-type","etcd","--secure-port","6443","--etcd-servers","https://etcd.kube-system.svc.cluster.local:2379","--etcd-cafile","/var/run/configmaps/etcd-serving-ca/ca-bundle.crt","--etcd-certfile","/var/run/secrets/etcd-client/tls.crt","--etcd-keyfile","/var/run/secrets/etcd-client/tls.key","--cors-allowed-origins","localhost","--enable-admission-plugins","NamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck","--feature-gates","OriginatingIdentity=true","--feature-gates","NamespacedServiceBroker=true","-v=3"],"command":["/usr/bin/service-catalog"],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"apiserver","ports":[{"containerPort":6443,"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz/ready","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"requests":{"memory":"200Mi"}},"terminationMessagePath":"/dev/termination-log","volumeMounts":[{"mountPath":"/var/run/kubernetes-service-catalog","name":"apiserver-ssl","readOnly":true},{"mountPath":"/etc/origin/master","name":"etcd-host-cert","readOnly":true},{"mountPath":"/var/run/secrets/etcd-client","name":"etcd-client"},{"mountPath":"/var/run/configmaps/etcd-serving-ca","name":"etcd-serving-ca"}]}],"schedulerName":null,"serviceAccount":null,"volumes":[{"name":"apiserver-ssl","secret":{"defaultMode":420,"items":[{"key":"tls.crt","path":"apiserver.crt"},{"key":"tls.key","path":"apiserver.key"}],"secretName":"serving-cert"}},{"hostPath":{"path":"/etc/origin/master"},"name":"etcd-host-cert"},{"emptyDir":{},"name":"data-dir"},{"configMap":{"name":"etcd-serving-ca"},"name":"etcd-serving-ca"},{"name":"etcd-client","secret":{"secretName":"etcd-client"}}]}}}}
I0423 03:15:24.675281       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DaemonSetUpdated' Updated DaemonSet.apps/apiserver -n openshift-service-catalog-apiserver because it changed
I0423 03:15:24.682516       1 status_controller.go:156] clusteroperator/service-catalog-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-04-23T03:15:04Z","reason":"AsExpected","status":"False","type":"Failing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Progressing: daemonset/apiserver.openshift-operator: observed generation is 1, desired generation is 2.\nProgressing: servicecatalogapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2.","reason":"ProgressingDesiredStateNotYetAchieved","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Available: no openshift-svcat-apiserver daemon pods available on any node.","reason":"AvailableNoAPIServerPod","status":"False","type":"Available"},{"lastTransitionTime":"2019-04-23T03:15:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0423 03:15:24.690685       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator service-catalog-apiserver changed: Progressing changed from False to True ("Progressing: daemonset/apiserver.openshift-operator: observed generation is 1, desired generation is 2.\nProgressing: servicecatalogapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2."),Available changed from True to False ("Available: no openshift-svcat-apiserver daemon pods available on any node.")
--
I0423 03:15:25.134737       1 request.go:530] Throttling request took 197.075386ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:15:25.334739       1 request.go:530] Throttling request took 197.090665ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:15:31.733816       1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - *v1.ServiceAccount total 16 items received
I0423 03:15:43.742838       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock
--
I0423 03:16:24.941745       1 request.go:530] Throttling request took 97.24382ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:16:25.141748       1 request.go:530] Throttling request took 196.95793ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:16:25.341763       1 request.go:530] Throttling request took 196.861396ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:16:25.541776       1 request.go:530] Throttling request took 196.971297ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:16:43.766397       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock
I0423 03:16:44.046285       1 request.go:530] Throttling request took 175.362835ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/servicecatalog-serviceclass-viewer
--
I0423 03:17:26.381744       1 request.go:530] Throttling request took 197.212944ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:26.581728       1 request.go:530] Throttling request took 197.305913ms, request: PUT:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/aggregator-client-ca
I0423 03:17:26.585898       1 core.go:191] ConfigMap "openshift-service-catalog-apiserver/aggregator-client-ca" changes: {"data":{"ca-bundle.crt":"-----BEGIN CERTIFICATE-----\nMIIDPjCCAiagAwIBAgIBATANBgkqhkiG9w0BAQsFADBQMU4wTAYDVQQDDEVvcGVu\nc2hpZnQta3ViZS1hcGlzZXJ2ZXItb3BlcmF0b3JfYWdncmVnYXRvci1jbGllbnQt\nc2lnbmVyQDE1NTU5NzIxNjkwHhcNMTkwNDIyMjIyOTI4WhcNMTkwNTIyMjIyOTI5\nWjBQMU4wTAYDVQQDDEVvcGVuc2hpZnQta3ViZS1hcGlzZXJ2ZXItb3BlcmF0b3Jf\nYWdncmVnYXRvci1jbGllbnQtc2lnbmVyQDE1NTU5NzIxNjkwggEiMA0GCSqGSIb3\nDQEBAQUAA4IBDwAwggEKAoIBAQCsCpq4LdxXETpo5Uf0Ca8pKxcGUg0/1wlHMS3O\n0s9HQPwQNfc9xP+Lw2PLD3xtWuVVaAN4nB3tvK+RVKJ9sxpToZP7H52UYu+1rroQ\n+Csnoh1UPNvIEji1cs91jDtNXJfnRcvZnnwA557pFSBOcMxG+Yo5gwTj7I3WIvMW\nO0oskWGAPPiGZYXcecGZY4FlkD+RI/+c3DMSqarGLSi7maRUsJrXi2T8Yl7LRH66\nVLH49km777jvdHAwZ6xOj8WoWTCyPZdfbMkdgTyyv/hjE3XqL2d771pk34BOYlim\nyXeoypTrKzyplmCvd7lPRm6RP+LsVU45Mk28PnL9Jy3QOz21AgMBAAGjIzAhMA4G\nA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB\nAQCkT4/7HhdBciQDYwpOE+6Kx2J98P8Wt+OP2tuBfuJ4/1xbE9LAers8lwbwf1uH\nichS6kd4KNukqGAdZ2THegohuuvnLg0SSHhhyOWHE8IYIftA+JQ6kwmG7QTjEt/9\nXXcYej53xixnWim9dxew+NEJoxF57MTAL14emxLsJ9kxaD5pZOCB1n5WKNBXBvKG\n1e+uV8It0GdBIEiebDNpPHiegihKvwRS6yUuO0LMX6Bw6r+Ze1ZniXP9DXp+FzFl\nlDC+PPu5lSt4fr4Mq75wFZNQySg7WrC2sO6n3wNJS2H6zT1eNDMKVmk+XfAkPR4J\nbM8wP79h3PnKM5TBdsQbyh/R\n-----END CERTIFICATE-----\n"},"metadata":{"creationTimestamp":"2019-04-22T03:32:58Z","resourceVersion":null,"selfLink":"/api/v1/namespaces/openshift-config-managed/configmaps/kube-apiserver-aggregator-client-ca","uid":"53150f6e-64af-11e9-ad6f-022d39ca3376"}}
I0423 03:17:26.586237       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/aggregator-client-ca -n openshift-service-catalog-apiserver: cause by changes in data.ca-bundle.crt
--
I0423 03:17:26.981733       1 request.go:530] Throttling request took 196.876668ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:27.181731       1 request.go:530] Throttling request took 197.045183ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:27.381731       1 request.go:530] Throttling request took 69.196906ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:27.581738       1 request.go:530] Throttling request took 197.106734ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:27.982066       1 request.go:530] Throttling request took 65.504224ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:28.181737       1 request.go:530] Throttling request took 127.329771ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:28.381668       1 request.go:530] Throttling request took 196.865618ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:28.581729       1 request.go:530] Throttling request took 197.019546ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:28.781730       1 request.go:530] Throttling request took 196.922323ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:28.981726       1 request.go:530] Throttling request took 197.017925ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:29.181706       1 request.go:530] Throttling request took 196.856481ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:30.921173       1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - *v1.ConfigMap total 0 items received
W0423 03:17:30.922497       1 reflector.go:270] k8s.io/client-go/informers/factory.go:131: watch of *v1.ConfigMap ended with: too old resource version: 409719 (411277)
--
I0423 03:17:44.673465       1 apps.go:109] DaemonSet "openshift-service-catalog-apiserver/apiserver" changes: {"metadata":{"annotations":{"operator.openshift.io/force":"5caa6896-6576-11e9-b6c3-0a580a810017"}},"spec":{"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"operator.openshift.io/force":"5caa6896-6576-11e9-b6c3-0a580a810017"}},"spec":{"containers":[{"args":["apiserver","--storage-type","etcd","--secure-port","6443","--etcd-servers","https://etcd.kube-system.svc.cluster.local:2379","--etcd-cafile","/var/run/configmaps/etcd-serving-ca/ca-bundle.crt","--etcd-certfile","/var/run/secrets/etcd-client/tls.crt","--etcd-keyfile","/var/run/secrets/etcd-client/tls.key","--cors-allowed-origins","localhost","--enable-admission-plugins","NamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck","--feature-gates","OriginatingIdentity=true","--feature-gates","NamespacedServiceBroker=true","-v=3"],"command":["/usr/bin/service-catalog"],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"apiserver","ports":[{"containerPort":6443,"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz/ready","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"requests":{"memory":"200Mi"}},"terminationMessagePath":"/dev/termination-log","volumeMounts":[{"mountPath":"/var/run/kubernetes-service-catalog","name":"apiserver-ssl","readOnly":true},{"mountPath":"/etc/origin/master","name":"etcd-host-cert","readOnly":true},{"mountPath":"/var/run/secrets/etcd-client","name":"etcd-client"},{"mountPath":"/var/run/configmaps/etcd-serving-ca","name":"etcd-serving-ca"}]}],"schedulerName":null,"serviceAccount":null,"volumes":[{"name":"apiserver-ssl","secret":{"defaultMode":420,"items":[{"key":"tls.crt","path":"apiserver.crt"},{"key":"tls.key","path":"apiserver.key"}],"secretName":"serving-cert"}},{"hostPath":{"path":"/etc/origin/master"},"name":"etcd-host-cert"},{"emptyDir":{},"name":"data-dir"},{"configMap":{"name":"etcd-serving-ca"},"name":"etcd-serving-ca"},{"name":"etcd-client","secret":{"secretName":"etcd-client"}}]}}}}
I0423 03:17:44.679380       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DaemonSetUpdated' Updated DaemonSet.apps/apiserver -n openshift-service-catalog-apiserver because it changed
I0423 03:17:44.685813       1 status_controller.go:156] clusteroperator/service-catalog-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-04-23T03:15:04Z","reason":"AsExpected","status":"False","type":"Failing"},{"lastTransitionTime":"2019-04-23T03:17:44Z","message":"Progressing: daemonset/apiserver.openshift-operator: observed generation is 2, desired generation is 3.","reason":"ProgressingDesiredStateNotYetAchieved","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Available: no openshift-svcat-apiserver daemon pods available on any node.","reason":"AvailableNoAPIServerPod","status":"False","type":"Available"},{"lastTransitionTime":"2019-04-23T03:15:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0423 03:17:44.686196       1 reflector.go:215] k8s.io/client-go/informers/factory.go:131: forcing resync
--
I0423 03:17:44.789794       1 request.go:530] Throttling request took 122.00941ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:44.989787       1 request.go:530] Throttling request took 311.565648ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver
I0423 03:17:45.189787       1 request.go:530] Throttling request took 396.716236ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:45.389745       1 request.go:530] Throttling request took 396.967347ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver
--
I0423 03:17:45.590085       1 request.go:530] Throttling request took 397.360752ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca
I0423 03:17:45.789760       1 request.go:530] Throttling request took 196.87415ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:17:46.189771       1 request.go:530] Throttling request took 87.618625ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client
I0423 03:18:03.798182       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock
--
...

Comment 2 Jay Boyd 2019-04-23 13:20:24 UTC
Thanks Jian, it appears a recent change moved the etcd configmap and secret from kube-system to openshift-config.

PR https://github.com/openshift/cluster-svcat-apiserver-operator/pull/48  fixes this.

Comment 3 Jay Boyd 2019-04-23 14:35:02 UTC
workaround if necessary:

Prior to setting the Service Catalog CR resources to managed: manually copy the etcd configmap and secret to the location they used to be found in:

$ oc get secret etcd-client -n openshift-config --export -o yaml | oc apply -n kube-system -f -

$ oc get configmap etcd-serving-ca -n openshift-config --export -o yaml | oc apply -n kube-system -f -

Comment 5 Jian Zhang 2019-04-24 03:04:00 UTC
Thanks!@Jay, LGTM, verify it. Details as below:

service-catalog-apiserver-operator version info:
io.openshift.build.commit.id=9d2fffe6e7d10fa88b7534fb07c38e997a6e887f
Cluster version is 4.1.0-0.nightly-2019-04-23-223857

mac:~ jianzhang$ oc edit servicecatalogapiserver
servicecatalogapiserver.operator.openshift.io/cluster edited
mac:~ jianzhang$ oc get pods -n openshift-service-catalog-apiserver
NAME              READY     STATUS    RESTARTS   AGE
apiserver-bgxzb   1/1       Running   0          2m23s
apiserver-dn7ns   1/1       Running   0          2m23s
apiserver-h2m8m   1/1       Running   0          2m23s

mac:~ jianzhang$ oc edit servicecatalogcontrollermanager
servicecatalogcontrollermanager.operator.openshift.io/cluster edited
mac:~ jianzhang$  oc get pods -n openshift-service-catalog-controller-manager
NAME                       READY     STATUS    RESTARTS   AGE
controller-manager-pn6lw   1/1       Running   0          2m15s
controller-manager-srsx2   1/1       Running   0          2m15s
controller-manager-w9pj5   1/1       Running   0          2m15s

mac:~ jianzhang$ oc get apiservice v1beta1.servicecatalog.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    service.alpha.openshift.io/inject-cabundle: "true"
  creationTimestamp: 2019-04-24T02:58:36Z
  name: v1beta1.servicecatalog.k8s.io
  resourceVersion: "29779"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.servicecatalog.k8s.io
  uid: da9ae1d2-663c-11e9-a06a-06bc4f05f9d4
spec:
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDakNDQWZLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREEyTVRRd01nWURWUVFEREN0dmNHVnUKYzJocFpuUXRjMlZ5ZG1salpTMXpaWEoyYVc1bkxYTnBaMjVsY2tBeE5UVTJNRGN4TkRNeE1CNFhEVEU1TURReQpOREF5TURNMU1Wb1hEVEl3TURReU16QXlNRE0xTWxvd05qRTBNRElHQTFVRUF3d3JiM0JsYm5Ob2FXWjBMWE5sCmNuWnBZMlV0YzJWeWRtbHVaeTF6YVdkdVpYSkFNVFUxTmpBM01UUXpNVENDQVNJd0RRWUpLb1pJaHZjTkFRRUIKQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHBDbk0xdVUwT0dZaU1ldTBwQ1B5WDR0eUJ2VWh4cFRJdklnTGRCU2p4YQpXZUpKbElKU3pDdEZUc05PaW0reldnMll1NjN6U1U0Tm0zaGhiNXVOSTdQSjJUUHNCTm9iUTNoU1hXZm0zdFZsCkdsdjhIbFVMUUF1WE9KOW5PMm0xTkNwa2ZYcWZWU1lLQ3YrNFF1ditXMFYwK0ptU1JtdzVYZGtUeVo1dVllNWkKRzMxcW1xU1NRN0haM2VBdnhTUklqSlNjZlV5UmgwdnhpS1NwS3hSNER2a2h5bzkrS2VaODdPNFdOTitEMko0ZQpKNlVlQkRYTkZaYUFKQTBHMG8ySXd4YWdlSVZkY0xDRmVFMVdwS3hXRnZoSUcxSUhHZmZkbHkvN1RPWkNhcUNXCi9leFBaSlRGZTZrejV6NVNlbm1jWWFwRWx6bFlzZVkzTi9FZ0lITkhidDhDQXdFQUFhTWpNQ0V3RGdZRFZSMFAKQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUgvVApnVGxybUhsdSt6SHhwQ3hVM043NS91QWpQeDQwM0p4a2duY0dTTlJXaVhqRzVseWFYSjM3VFRXeWd1TGduTGJICkNUYm50cEhPTlR2aWhiOFZIaG1tdjFJb25pK2g2SXNpdDRxdURFeXRCMUNvT1MvSHl5ZHlVRDc1R0hFWSt5blAKYlgvK0MwQVJPU0EwcXBxOHdTL3lISmhKalowU1Yzb2RvTkhJdmpPNlNGMTN5M3JEL2lUb2Mrd3Vkd1kyL0tTUgpFdEhLQVJPeTlLOWM2KzExNTFGeVRiSEl3cDR3VDUzd2lFbXdmTDBjdDZuYWU5dHFlc2VhNWt2QkJ4eDg0dnVBCisxZXdpVDhhT1hkMzMvTDkyWHREVzN0Ums2ZnkxK3Z0em9jbHdRMXc3alVNcDV6Wi8zRzB5Qjc1a2h6cW9QOVkKVGR3dEJBSEtTOUMvNGFuVW5ZND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  group: servicecatalog.k8s.io
  groupPriorityMinimum: 9900
  service:
    name: api
    namespace: openshift-service-catalog-apiserver
  version: v1beta1
  versionPriority: 15
status:
  conditions:
  - lastTransitionTime: 2019-04-24T02:58:36Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available
mac:~ jianzhang$ oc get svc -n openshift-service-catalog-apiserver
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
api       ClusterIP   172.30.86.86   <none>        443/TCP   4m49s
mac:~ jianzhang$ oc get clusterservicebroker
No resources found.

Comment 6 Jian Zhang 2019-05-13 01:48:09 UTC
*** Bug 1708793 has been marked as a duplicate of this bug. ***

Comment 9 errata-xmlrpc 2019-06-04 10:47:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.