Bug 1889663 - Cluster operator image-registry not created after cluster deployment on OpenStack (disconnected)
Summary: Cluster operator image-registry not created after cluster deployment on OpenS...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.7.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-20 10:45 UTC by Dhruv Gautam
Modified: 2020-11-25 16:48 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-12 12:29:38 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Dhruv Gautam 2020-10-20 10:45:53 UTC
Description of problem:
Cluster operator image-registry can't create when launch

Version-Release number of selected component (if applicable):
OCP 4.5.6

How reproducible:


Steps to Reproduce:
Create a OCP 4.5 cluster on OpenStack disconnect environment via UPI installation.
Login to the cluster and check operators status. 

Actual results:
$ oc get clusteroperator
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.6     True        False         False      38d
cloud-credential                           4.5.6     True        False         False      38d
cluster-autoscaler                         4.5.6     True        False         False      38d
config-operator                            4.5.6     True        False         False      38d
console                                    4.5.6     True        False         False      35d
csi-snapshot-controller                    4.5.6     True        False         False      35d
dns                                        4.5.6     True        False         False      38d
etcd                                       4.5.6     True        False         False      38d
image-registry
ingress                                    4.5.6     True        False         False      38d
insights                                   4.5.6     True        False         False      38d
kube-apiserver                             4.5.6     True        False         False      38d
kube-controller-manager                    4.5.6     True        False         False      38d
kube-scheduler                             4.5.6     True        False         False      38d
kube-storage-version-migrator              4.5.6     True        False         False      35d
machine-api                                4.5.6     True        False         False      38d
machine-approver                           4.5.6     True        False         False      38d
machine-config                             4.5.6     True        False         False      35d
marketplace                                4.5.6     True        False         False      35d
monitoring                                 4.5.6     True        False         False      38d
network                                    4.5.6     True        False         False      38d
node-tuning                                4.5.6     True        False         False      38d
openshift-apiserver                        4.5.6     True        False         False      38d
openshift-controller-manager               4.5.6     True        False         False      38d
openshift-samples                          4.5.6     True        False         False      38d
operator-lifecycle-manager                 4.5.6     True        False         False      38d
operator-lifecycle-manager-catalog         4.5.6     True        False         False      38d
operator-lifecycle-manager-packageserver   4.5.6     True        False         False      35d
service-ca                                 4.5.6     True        False         False      38d
storage                                    4.5.6     True        False         False      38d

Expected results:
All operators in available state

Additional info:

Errors observed:
E1020 06:54:06.631909      13 nodecadaemon.go:82] NodeCADaemonController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.631932      13 clusteroperator.go:96] unable to sync ClusterOperatorStatusController: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
I1020 06:54:06.631978      13 caconfig.go:75] unable to get the service name to add service-ca.crt
E1020 06:54:06.632063      13 imageregistrycertificates.go:94] ImageRegistryCertificatesController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.636007      13 imageconfig.go:109] ImageConfigController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.637069      13 nodecadaemon.go:82] NodeCADaemonController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.637072      13 clusteroperator.go:96] unable to sync ClusterOperatorStatusController: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
I1020 06:54:06.637156      13 caconfig.go:75] unable to get the service name to add service-ca.crt
E1020 06:54:06.637200      13 imageregistrycertificates.go:94] ImageRegistryCertificatesController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.639683      13 controllerimagepruner.go:306] (image pruner) unable to sync: unable to apply objects: failed to create object *v1beta1.CronJob, Namespace=openshift-image-registry, Name=image-pruner: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.644046      13 imageconfig.go:109] ImageConfigController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.647779      13 clusteroperator.go:96] unable to sync ClusterOperatorStatusController: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
I1020 06:54:06.647968      13 caconfig.go:75] unable to get the service name to add service-ca.crt
E1020 06:54:06.648041      13 nodecadaemon.go:82] NodeCADaemonController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.648093      13 imageregistrycertificates.go:94] ImageRegistryCertificatesController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.657326      13 imageconfig.go:109] ImageConfigController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.660002      13 controllerimagepruner.go:306] (image pruner) unable to sync: unable to apply objects: failed to create object *v1beta1.CronJob, Namespace=openshift-image-registry, Name=image-pruner: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.660391      13 controller.go:311] unable to sync: Failed to authenticate provider client: Post https://X.X.X.X:YYYY//v3/auth/tokens: x509: certificate signed by unknown authority, requeuing
E1020 06:54:06.667932      13 clusteroperator.go:96] unable to sync ClusterOperatorStatusController: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
E1020 06:54:06.668198      13 nodecadaemon.go:82] NodeCADaemonController: unable to sync: config.imageregistry.operator.openshift.io "cluster" not found, requeuing
I1020 06:54:06.668231      13 caconfig.go:75] unable to get the service name to add service-ca.crt

Comment 1 Oleg Bulatov 2020-10-20 11:42:36 UTC
Dhruv, can you check the configmap cloud-provider-config in the openshift-config namespace?

It should contain ca-bundle.pem with CA for Swift.

Comment 3 Dhruv Gautam 2020-10-20 11:58:56 UTC
Hello Oleg

Find details below.

$ oc get cm cloud-provider-config -n openshift-config -o yaml
apiVersion: v1
data:
  config: |
    [Global]
    secret-name = openstack-credentials
    secret-namespace = kube-system
    region = regionOne
kind: ConfigMap
metadata:
  creationTimestamp: "2020-09-09T11:51:56Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:config: {}
    manager: cluster-bootstrap
    operation: Update
    time: "2020-09-09T11:51:56Z"
  name: cloud-provider-config
  namespace: openshift-config
  resourceVersion: "1098"
  selfLink: /api/v1/namespaces/openshift-config/configmaps/cloud-provider-config
  uid: ca5c3c71-f83b-40d3-8d42-e614bbd0e4ab

$ oc get cm -n openshift-config
NAME                               DATA   AGE
admin-kubeconfig-client-ca         1      41d
cloud-provider-config              1      41d
custom-ca                          1      40d
etcd-ca-bundle                     1      41d
etcd-metric-serving-ca             1      41d
etcd-serving-ca                    1      41d
initial-etcd-ca                    1      41d
initial-kube-apiserver-server-ca   1      41d
openshift-install-manifests        2      41d
user-ca-bundle                     1      41d

Comment 4 Oleg Bulatov 2020-10-20 12:28:41 UTC
Ok, so they don't have it there.

Do they have `cacert` in their `clouds.yaml` file? It's the recommended way to provide the CA bundle for OpenStack if it uses self-signed certificates [1].

[1]: https://github.com/openshift/installer/tree/master/docs/user/openstack#self-signed-openstack-ca-certificates

Comment 6 Oleg Bulatov 2020-10-21 12:49:49 UTC
Edit cloud-provider-config manually and add something like this:

data:
  ca-bundle.pem: |
    -----BEGIN CERTIFICATE -----
    ...

The operator should notice this field and continue with it.

Please let me know if it helps.


Note You need to log in before you can comment on or make changes to this bug.