Created attachment 1566799 [details]
openshift-service-catalog-apiserver-operator pod log

Description of problem:
Set servicecatalogapiservers/cluster and servicecatalogcontrollermanagers/cluster to Managed in HTB4 on several clusters. On one cluster, the operators entered an odd state.

# apiserver pods cannot start
[ec2-user us-east-2 ~]$ oc get pods -n openshift-service-catalog-apiserver
NAME              READY   STATUS              RESTARTS   AGE
apiserver-bxhxx   0/1     ContainerCreating   0          21h
apiserver-cjdn2   0/1     ContainerCreating   0          21h
apiserver-q98gr   0/1     ContainerCreating   0          21h


# Events show missing configmaps and secrets
Events:
  Type     Reason       Age                    From                                                Message
  ----     ------       ----                   ----                                                -------
  Warning  FailedMount  16m (x563 over 21h)    kubelet, ip-10-0-144-46.us-east-2.compute.internal  Unable to mount volumes for pod "apiserver-bxhxx_openshift-service-catalog-apiserver(6a7d8e1d-72b2-11e9-ac48-0aa717a400a0)": timeout expired waiting for volumes to attach or mount for pod "openshift-service-catalog-apiserver"/"apiserver-bxhxx". list of unmounted volumes=[etcd-serving-ca etcd-client]. list of unattached volumes=[apiserver-ssl etcd-host-cert data-dir etcd-serving-ca etcd-client service-catalog-apiserver-token-gbqxx]
  Warning  FailedMount  6m31s (x639 over 21h)  kubelet, ip-10-0-144-46.us-east-2.compute.internal  MountVolume.SetUp failed for volume "etcd-serving-ca" : configmaps "etcd-serving-ca" not found
  Warning  FailedMount  2m7s (x641 over 21h)   kubelet, ip-10-0-144-46.us-east-2.compute.internal  MountVolume.SetUp failed for volume "etcd-client" : secrets "etcd-client" not found


# Secrets and configmaps which are currently in the namespace
[ec2-user us-east-2 ~]$ oc get cm -n openshift-service-catalog-apiserver 
NAME                   DATA   AGE
aggregator-client-ca   1      21h
client-ca              1      21h
config                 4      21h
[ec2-user us-east-2 ~]$ oc get secret -n openshift-service-catalog-apiserver 
NAME                                        TYPE                                  DATA   AGE
builder-dockercfg-jp2qg                     kubernetes.io/dockercfg               1      21h
builder-token-pmkvf                         kubernetes.io/service-account-token   4      21h
builder-token-stvps                         kubernetes.io/service-account-token   4      21h
default-dockercfg-bzpfs                     kubernetes.io/dockercfg               1      21h
default-token-mzw7b                         kubernetes.io/service-account-token   4      21h
default-token-zv964                         kubernetes.io/service-account-token   4      21h
deployer-dockercfg-xfj6l                    kubernetes.io/dockercfg               1      21h
deployer-token-gxldf                        kubernetes.io/service-account-token   4      21h
deployer-token-k7w9f                        kubernetes.io/service-account-token   4      21h
service-catalog-apiserver-dockercfg-9xnzk   kubernetes.io/dockercfg               1      21h
service-catalog-apiserver-token-gbqxx       kubernetes.io/service-account-token   4      21h
service-catalog-apiserver-token-l6qsr       kubernetes.io/service-account-token   4      21h
serving-cert                                kubernetes.io/tls                     2      21h


# Areas of the operator log indicate that various creates failed (full log will be attached)
I0509 23:29:36.269693       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-service-catalog-apiserver-operator", Name:"openshift-service-catalog-apiserver-operator", UID:"5d22bacd-67b2-11e9-b8f4-02fd9e2da322", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ConfigMapCreateFailed' Failed to create ConfigMap/aggregator-client-ca -n openshift-service-catalog-apiserver: namespaces "openshift-service-catalog-apiserver" not found



Version-Release number of selected component (if applicable):
4.1.0-rc.0 (HTB4)

How reproducible:
Unknown


Steps to Reproduce:
1. Set servicecatalogapiservers and servicecatalogcontrollermanagers to Managed
2. Wait for operators to become ready
3.

Actual results:
- The operators never become available


Additional info:
See attachments and listings.