Hide Forgot
Description of problem: Got the below errors when enabling the Service Catalog: MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found Version-Release number of selected component (if applicable): Cluster version is 4.1.0-0.nightly-2019-04-22-005054 Service Catalog apiserver operator image: image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a27e0a53ffb0c926f025bea48d2a537e2b0e521ce055a58e0a5ee3af719e8d version: io.openshift.build.commit.id=0e8e9ffc5e874be13ee99c471216f930575c083b Service Catalog apiserver image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c version: io.openshift.build.commit.id=1dd02c427d9a0b143e5f2a6ef05506a79a311c13 How reproducible: always Steps to Reproduce: 1. Install the OCP 4.1 with the payload: 4.1.0-0.nightly-2019-04-22-005054 2. Enable the service catalog apiserver, change `managementState` to `Managed` from `Removed`, like below: mac:~ jianzhang$ oc edit servicecatalogapiserver ... managementState: Managed ... 3. Check the pods in the project "openshift-service-catalog-apiserver". Actual results: No pods generated. mac:~ jianzhang$ oc get pods -n openshift-service-catalog-apiserver NAME READY STATUS RESTARTS AGE apiserver-7kh5s 0/1 ContainerCreating 0 159m apiserver-w9h4d 0/1 ContainerCreating 0 159m apiserver-xmw9x 0/1 ContainerCreating 0 159m mac:~ jianzhang$ oc describe pods apiserver-7kh5s -n openshift-service-catalog-apiserver ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2h default-scheduler Successfully assigned openshift-service-catalog-apiserver/apiserver-7kh5s to ip-172-31-131-234.us-west-2.compute.internal Warning FailedMount 2h (x11 over 2h) kubelet, ip-172-31-131-234.us-west-2.compute.internal MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found Warning FailedMount 14m (x64 over 2h) kubelet, ip-172-31-131-234.us-west-2.compute.internal Unable to mount volumes for pod "apiserver-7kh5s_openshift-service-catalog-apiserver(60d2f2e8-6576-11e9-8e4a-0ad5afc33fa6)": timeout expired waiting for volumes to attach or mount for pod "openshift-service-catalog-apiserver"/"apiserver-7kh5s". list of unmounted volumes=[etcd-serving-ca etcd-client]. list of unattached volumes=[apiserver-ssl etcd-host-cert data-dir etcd-serving-ca etcd-client service-catalog-apiserver-token-qv6ds] Warning FailedMount 3m (x85 over 2h) kubelet, ip-172-31-131-234.us-west-2.compute.internal MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found mac:~ jianzhang$ oc get cm NAME DATA AGE aggregator-client-ca 1 3h4m client-ca 1 3h4m config 4 3h3m mac:~ jianzhang$ oc get secret NAME TYPE DATA AGE builder-dockercfg-6pwpt kubernetes.io/dockercfg 1 3h4m builder-token-l8kcj kubernetes.io/service-account-token 4 3h4m builder-token-lcdhc kubernetes.io/service-account-token 4 3h4m default-dockercfg-w7dgm kubernetes.io/dockercfg 1 3h4m default-token-t77gv kubernetes.io/service-account-token 4 3h4m default-token-ttcsb kubernetes.io/service-account-token 4 3h4m deployer-dockercfg-mv6tg kubernetes.io/dockercfg 1 3h4m deployer-token-m4wjk kubernetes.io/service-account-token 4 3h4m deployer-token-v8ngn kubernetes.io/service-account-token 4 3h4m service-catalog-apiserver-dockercfg-92vj8 kubernetes.io/dockercfg 1 3h4m service-catalog-apiserver-token-g2jkd kubernetes.io/service-account-token 4 3h4m service-catalog-apiserver-token-qv6ds kubernetes.io/service-account-token 4 3h4m serving-cert kubernetes.io/tls 2 3h4m mac:~ jianzhang$ oc get rolebinding NAME AGE system:deployers 3h system:image-builders 3h system:image-pullers 3h Expected results: The service catalog works well. Additional info: 1, No corresponding apiservice generated: mac:~ jianzhang$ oc get apiservice|grep catalog mac:~ jianzhang$ 2, Checked the apiserver operator logs, found the configmap `etcd-serving-ca` and secret `etcd-client` has been deleting. As below: mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-operator-979f6d66b-m487d |grep etcd-serving-ca -A 3 I0423 03:15:04.955303 1 request.go:530] Throttling request took 391.920593ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:15:04.964860 1 request.go:530] Throttling request took 107.966264ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/service-catalog-namespace-viewer-binding I0423 03:15:05.155309 1 request.go:530] Throttling request took 197.155469ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:15:05.164835 1 request.go:530] Throttling request took 197.353455ms, request: POST:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings -- I0423 03:15:05.355311 1 request.go:530] Throttling request took 196.601887ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:15:05.364854 1 request.go:530] Throttling request took 193.27846ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/servicecatalog.k8s.io:service-catalog-readiness I0423 03:15:05.555301 1 request.go:530] Throttling request took 196.969308ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:15:05.564835 1 request.go:530] Throttling request took 196.946719ms, request: POST:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles -- I0423 03:15:24.669744 1 apps.go:109] DaemonSet "openshift-service-catalog-apiserver/apiserver" changes: {"metadata":{"annotations":{"operator.openshift.io/force":"093736c2-6576-11e9-b6c3-0a580a810017"}},"spec":{"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"operator.openshift.io/force":"093736c2-6576-11e9-b6c3-0a580a810017"}},"spec":{"containers":[{"args":["apiserver","--storage-type","etcd","--secure-port","6443","--etcd-servers","https://etcd.kube-system.svc.cluster.local:2379","--etcd-cafile","/var/run/configmaps/etcd-serving-ca/ca-bundle.crt","--etcd-certfile","/var/run/secrets/etcd-client/tls.crt","--etcd-keyfile","/var/run/secrets/etcd-client/tls.key","--cors-allowed-origins","localhost","--enable-admission-plugins","NamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck","--feature-gates","OriginatingIdentity=true","--feature-gates","NamespacedServiceBroker=true","-v=3"],"command":["/usr/bin/service-catalog"],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"apiserver","ports":[{"containerPort":6443,"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz/ready","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"requests":{"memory":"200Mi"}},"terminationMessagePath":"/dev/termination-log","volumeMounts":[{"mountPath":"/var/run/kubernetes-service-catalog","name":"apiserver-ssl","readOnly":true},{"mountPath":"/etc/origin/master","name":"etcd-host-cert","readOnly":true},{"mountPath":"/var/run/secrets/etcd-client","name":"etcd-client"},{"mountPath":"/var/run/configmaps/etcd-serving-ca","name":"etcd-serving-ca"}]}],"schedulerName":null,"serviceAccount":null,"volumes":[{"name":"apiserver-ssl","secret":{"defaultMode":420,"items":[{"key":"tls.crt","path":"apiserver.crt"},{"key":"tls.key","path":"apiserver.key"}],"secretName":"serving-cert"}},{"hostPath":{"path":"/etc/origin/master"},"name":"etcd-host-cert"},{"emptyDir":{},"name":"data-dir"},{"configMap":{"name":"etcd-serving-ca"},"name":"etcd-serving-ca"},{"name":"etcd-client","secret":{"secretName":"etcd-client"}}]}}}} I0423 03:15:24.675281 1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DaemonSetUpdated' Updated DaemonSet.apps/apiserver -n openshift-service-catalog-apiserver because it changed I0423 03:15:24.682516 1 status_controller.go:156] clusteroperator/service-catalog-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-04-23T03:15:04Z","reason":"AsExpected","status":"False","type":"Failing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Progressing: daemonset/apiserver.openshift-operator: observed generation is 1, desired generation is 2.\nProgressing: servicecatalogapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2.","reason":"ProgressingDesiredStateNotYetAchieved","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Available: no openshift-svcat-apiserver daemon pods available on any node.","reason":"AvailableNoAPIServerPod","status":"False","type":"Available"},{"lastTransitionTime":"2019-04-23T03:15:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} I0423 03:15:24.690685 1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator service-catalog-apiserver changed: Progressing changed from False to True ("Progressing: daemonset/apiserver.openshift-operator: observed generation is 1, desired generation is 2.\nProgressing: servicecatalogapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2."),Available changed from True to False ("Available: no openshift-svcat-apiserver daemon pods available on any node.") -- I0423 03:15:25.134737 1 request.go:530] Throttling request took 197.075386ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:15:25.334739 1 request.go:530] Throttling request took 197.090665ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:15:31.733816 1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - *v1.ServiceAccount total 16 items received I0423 03:15:43.742838 1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock -- I0423 03:16:24.941745 1 request.go:530] Throttling request took 97.24382ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:16:25.141748 1 request.go:530] Throttling request took 196.95793ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:16:25.341763 1 request.go:530] Throttling request took 196.861396ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:16:25.541776 1 request.go:530] Throttling request took 196.971297ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:16:43.766397 1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock I0423 03:16:44.046285 1 request.go:530] Throttling request took 175.362835ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/servicecatalog-serviceclass-viewer -- I0423 03:17:26.381744 1 request.go:530] Throttling request took 197.212944ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:26.581728 1 request.go:530] Throttling request took 197.305913ms, request: PUT:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/aggregator-client-ca I0423 03:17:26.585898 1 core.go:191] ConfigMap "openshift-service-catalog-apiserver/aggregator-client-ca" changes: {"data":{"ca-bundle.crt":"-----BEGIN CERTIFICATE-----\nMIIDPjCCAiagAwIBAgIBATANBgkqhkiG9w0BAQsFADBQMU4wTAYDVQQDDEVvcGVu\nc2hpZnQta3ViZS1hcGlzZXJ2ZXItb3BlcmF0b3JfYWdncmVnYXRvci1jbGllbnQt\nc2lnbmVyQDE1NTU5NzIxNjkwHhcNMTkwNDIyMjIyOTI4WhcNMTkwNTIyMjIyOTI5\nWjBQMU4wTAYDVQQDDEVvcGVuc2hpZnQta3ViZS1hcGlzZXJ2ZXItb3BlcmF0b3Jf\nYWdncmVnYXRvci1jbGllbnQtc2lnbmVyQDE1NTU5NzIxNjkwggEiMA0GCSqGSIb3\nDQEBAQUAA4IBDwAwggEKAoIBAQCsCpq4LdxXETpo5Uf0Ca8pKxcGUg0/1wlHMS3O\n0s9HQPwQNfc9xP+Lw2PLD3xtWuVVaAN4nB3tvK+RVKJ9sxpToZP7H52UYu+1rroQ\n+Csnoh1UPNvIEji1cs91jDtNXJfnRcvZnnwA557pFSBOcMxG+Yo5gwTj7I3WIvMW\nO0oskWGAPPiGZYXcecGZY4FlkD+RI/+c3DMSqarGLSi7maRUsJrXi2T8Yl7LRH66\nVLH49km777jvdHAwZ6xOj8WoWTCyPZdfbMkdgTyyv/hjE3XqL2d771pk34BOYlim\nyXeoypTrKzyplmCvd7lPRm6RP+LsVU45Mk28PnL9Jy3QOz21AgMBAAGjIzAhMA4G\nA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB\nAQCkT4/7HhdBciQDYwpOE+6Kx2J98P8Wt+OP2tuBfuJ4/1xbE9LAers8lwbwf1uH\nichS6kd4KNukqGAdZ2THegohuuvnLg0SSHhhyOWHE8IYIftA+JQ6kwmG7QTjEt/9\nXXcYej53xixnWim9dxew+NEJoxF57MTAL14emxLsJ9kxaD5pZOCB1n5WKNBXBvKG\n1e+uV8It0GdBIEiebDNpPHiegihKvwRS6yUuO0LMX6Bw6r+Ze1ZniXP9DXp+FzFl\nlDC+PPu5lSt4fr4Mq75wFZNQySg7WrC2sO6n3wNJS2H6zT1eNDMKVmk+XfAkPR4J\nbM8wP79h3PnKM5TBdsQbyh/R\n-----END CERTIFICATE-----\n"},"metadata":{"creationTimestamp":"2019-04-22T03:32:58Z","resourceVersion":null,"selfLink":"/api/v1/namespaces/openshift-config-managed/configmaps/kube-apiserver-aggregator-client-ca","uid":"53150f6e-64af-11e9-ad6f-022d39ca3376"}} I0423 03:17:26.586237 1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/aggregator-client-ca -n openshift-service-catalog-apiserver: cause by changes in data.ca-bundle.crt -- I0423 03:17:26.981733 1 request.go:530] Throttling request took 196.876668ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:27.181731 1 request.go:530] Throttling request took 197.045183ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:27.381731 1 request.go:530] Throttling request took 69.196906ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:27.581738 1 request.go:530] Throttling request took 197.106734ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:27.982066 1 request.go:530] Throttling request took 65.504224ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:28.181737 1 request.go:530] Throttling request took 127.329771ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:28.381668 1 request.go:530] Throttling request took 196.865618ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:28.581729 1 request.go:530] Throttling request took 197.019546ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:28.781730 1 request.go:530] Throttling request took 196.922323ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:28.981726 1 request.go:530] Throttling request took 197.017925ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:29.181706 1 request.go:530] Throttling request took 196.856481ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:30.921173 1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - *v1.ConfigMap total 0 items received W0423 03:17:30.922497 1 reflector.go:270] k8s.io/client-go/informers/factory.go:131: watch of *v1.ConfigMap ended with: too old resource version: 409719 (411277) -- I0423 03:17:44.673465 1 apps.go:109] DaemonSet "openshift-service-catalog-apiserver/apiserver" changes: {"metadata":{"annotations":{"operator.openshift.io/force":"5caa6896-6576-11e9-b6c3-0a580a810017"}},"spec":{"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"operator.openshift.io/force":"5caa6896-6576-11e9-b6c3-0a580a810017"}},"spec":{"containers":[{"args":["apiserver","--storage-type","etcd","--secure-port","6443","--etcd-servers","https://etcd.kube-system.svc.cluster.local:2379","--etcd-cafile","/var/run/configmaps/etcd-serving-ca/ca-bundle.crt","--etcd-certfile","/var/run/secrets/etcd-client/tls.crt","--etcd-keyfile","/var/run/secrets/etcd-client/tls.key","--cors-allowed-origins","localhost","--enable-admission-plugins","NamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck","--feature-gates","OriginatingIdentity=true","--feature-gates","NamespacedServiceBroker=true","-v=3"],"command":["/usr/bin/service-catalog"],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:faccd1e392179fc7dc7b3dfb5d71c7b4884f5e5361ee12b0871ac1d68adcbd7c","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"apiserver","ports":[{"containerPort":6443,"protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz/ready","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"requests":{"memory":"200Mi"}},"terminationMessagePath":"/dev/termination-log","volumeMounts":[{"mountPath":"/var/run/kubernetes-service-catalog","name":"apiserver-ssl","readOnly":true},{"mountPath":"/etc/origin/master","name":"etcd-host-cert","readOnly":true},{"mountPath":"/var/run/secrets/etcd-client","name":"etcd-client"},{"mountPath":"/var/run/configmaps/etcd-serving-ca","name":"etcd-serving-ca"}]}],"schedulerName":null,"serviceAccount":null,"volumes":[{"name":"apiserver-ssl","secret":{"defaultMode":420,"items":[{"key":"tls.crt","path":"apiserver.crt"},{"key":"tls.key","path":"apiserver.key"}],"secretName":"serving-cert"}},{"hostPath":{"path":"/etc/origin/master"},"name":"etcd-host-cert"},{"emptyDir":{},"name":"data-dir"},{"configMap":{"name":"etcd-serving-ca"},"name":"etcd-serving-ca"},{"name":"etcd-client","secret":{"secretName":"etcd-client"}}]}}}} I0423 03:17:44.679380 1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"2464c25c-64af-11e9-ad6f-022d39ca3376", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DaemonSetUpdated' Updated DaemonSet.apps/apiserver -n openshift-service-catalog-apiserver because it changed I0423 03:17:44.685813 1 status_controller.go:156] clusteroperator/service-catalog-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-04-23T03:15:04Z","reason":"AsExpected","status":"False","type":"Failing"},{"lastTransitionTime":"2019-04-23T03:17:44Z","message":"Progressing: daemonset/apiserver.openshift-operator: observed generation is 2, desired generation is 3.","reason":"ProgressingDesiredStateNotYetAchieved","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-04-23T03:15:24Z","message":"Available: no openshift-svcat-apiserver daemon pods available on any node.","reason":"AvailableNoAPIServerPod","status":"False","type":"Available"},{"lastTransitionTime":"2019-04-23T03:15:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} I0423 03:17:44.686196 1 reflector.go:215] k8s.io/client-go/informers/factory.go:131: forcing resync -- I0423 03:17:44.789794 1 request.go:530] Throttling request took 122.00941ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:44.989787 1 request.go:530] Throttling request took 311.565648ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver I0423 03:17:45.189787 1 request.go:530] Throttling request took 396.716236ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:45.389745 1 request.go:530] Throttling request took 396.967347ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver -- I0423 03:17:45.590085 1 request.go:530] Throttling request took 397.360752ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/configmaps/etcd-serving-ca I0423 03:17:45.789760 1 request.go:530] Throttling request took 196.87415ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:17:46.189771 1 request.go:530] Throttling request took 87.618625ms, request: DELETE:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver/secrets/etcd-client I0423 03:18:03.798182 1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock -- ...
Thanks Jian, it appears a recent change moved the etcd configmap and secret from kube-system to openshift-config. PR https://github.com/openshift/cluster-svcat-apiserver-operator/pull/48 fixes this.
workaround if necessary: Prior to setting the Service Catalog CR resources to managed: manually copy the etcd configmap and secret to the location they used to be found in: $ oc get secret etcd-client -n openshift-config --export -o yaml | oc apply -n kube-system -f - $ oc get configmap etcd-serving-ca -n openshift-config --export -o yaml | oc apply -n kube-system -f -
Thanks!@Jay, LGTM, verify it. Details as below: service-catalog-apiserver-operator version info: io.openshift.build.commit.id=9d2fffe6e7d10fa88b7534fb07c38e997a6e887f Cluster version is 4.1.0-0.nightly-2019-04-23-223857 mac:~ jianzhang$ oc edit servicecatalogapiserver servicecatalogapiserver.operator.openshift.io/cluster edited mac:~ jianzhang$ oc get pods -n openshift-service-catalog-apiserver NAME READY STATUS RESTARTS AGE apiserver-bgxzb 1/1 Running 0 2m23s apiserver-dn7ns 1/1 Running 0 2m23s apiserver-h2m8m 1/1 Running 0 2m23s mac:~ jianzhang$ oc edit servicecatalogcontrollermanager servicecatalogcontrollermanager.operator.openshift.io/cluster edited mac:~ jianzhang$ oc get pods -n openshift-service-catalog-controller-manager NAME READY STATUS RESTARTS AGE controller-manager-pn6lw 1/1 Running 0 2m15s controller-manager-srsx2 1/1 Running 0 2m15s controller-manager-w9pj5 1/1 Running 0 2m15s mac:~ jianzhang$ oc get apiservice v1beta1.servicecatalog.k8s.io -o yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: annotations: service.alpha.openshift.io/inject-cabundle: "true" creationTimestamp: 2019-04-24T02:58:36Z name: v1beta1.servicecatalog.k8s.io resourceVersion: "29779" selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.servicecatalog.k8s.io uid: da9ae1d2-663c-11e9-a06a-06bc4f05f9d4 spec: caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDakNDQWZLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREEyTVRRd01nWURWUVFEREN0dmNHVnUKYzJocFpuUXRjMlZ5ZG1salpTMXpaWEoyYVc1bkxYTnBaMjVsY2tBeE5UVTJNRGN4TkRNeE1CNFhEVEU1TURReQpOREF5TURNMU1Wb1hEVEl3TURReU16QXlNRE0xTWxvd05qRTBNRElHQTFVRUF3d3JiM0JsYm5Ob2FXWjBMWE5sCmNuWnBZMlV0YzJWeWRtbHVaeTF6YVdkdVpYSkFNVFUxTmpBM01UUXpNVENDQVNJd0RRWUpLb1pJaHZjTkFRRUIKQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHBDbk0xdVUwT0dZaU1ldTBwQ1B5WDR0eUJ2VWh4cFRJdklnTGRCU2p4YQpXZUpKbElKU3pDdEZUc05PaW0reldnMll1NjN6U1U0Tm0zaGhiNXVOSTdQSjJUUHNCTm9iUTNoU1hXZm0zdFZsCkdsdjhIbFVMUUF1WE9KOW5PMm0xTkNwa2ZYcWZWU1lLQ3YrNFF1ditXMFYwK0ptU1JtdzVYZGtUeVo1dVllNWkKRzMxcW1xU1NRN0haM2VBdnhTUklqSlNjZlV5UmgwdnhpS1NwS3hSNER2a2h5bzkrS2VaODdPNFdOTitEMko0ZQpKNlVlQkRYTkZaYUFKQTBHMG8ySXd4YWdlSVZkY0xDRmVFMVdwS3hXRnZoSUcxSUhHZmZkbHkvN1RPWkNhcUNXCi9leFBaSlRGZTZrejV6NVNlbm1jWWFwRWx6bFlzZVkzTi9FZ0lITkhidDhDQXdFQUFhTWpNQ0V3RGdZRFZSMFAKQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUgvVApnVGxybUhsdSt6SHhwQ3hVM043NS91QWpQeDQwM0p4a2duY0dTTlJXaVhqRzVseWFYSjM3VFRXeWd1TGduTGJICkNUYm50cEhPTlR2aWhiOFZIaG1tdjFJb25pK2g2SXNpdDRxdURFeXRCMUNvT1MvSHl5ZHlVRDc1R0hFWSt5blAKYlgvK0MwQVJPU0EwcXBxOHdTL3lISmhKalowU1Yzb2RvTkhJdmpPNlNGMTN5M3JEL2lUb2Mrd3Vkd1kyL0tTUgpFdEhLQVJPeTlLOWM2KzExNTFGeVRiSEl3cDR3VDUzd2lFbXdmTDBjdDZuYWU5dHFlc2VhNWt2QkJ4eDg0dnVBCisxZXdpVDhhT1hkMzMvTDkyWHREVzN0Ums2ZnkxK3Z0em9jbHdRMXc3alVNcDV6Wi8zRzB5Qjc1a2h6cW9QOVkKVGR3dEJBSEtTOUMvNGFuVW5ZND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= group: servicecatalog.k8s.io groupPriorityMinimum: 9900 service: name: api namespace: openshift-service-catalog-apiserver version: v1beta1 versionPriority: 15 status: conditions: - lastTransitionTime: 2019-04-24T02:58:36Z message: all checks passed reason: Passed status: "True" type: Available mac:~ jianzhang$ oc get svc -n openshift-service-catalog-apiserver NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE api ClusterIP 172.30.86.86 <none> 443/TCP 4m49s mac:~ jianzhang$ oc get clusterservicebroker No resources found.
*** Bug 1708793 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758