Description of problem: MariaDB Platform Operator can not be installed on OCP4.1 Version-Release number of selected component (if applicable): Cluster version is 4.1.0-0.nightly-2019-05-24-040103 MariaDB Platform Operator is 0.1.3-3 provided by MariaDB Corporation How reproducible: Always Steps to Reproduce: 1.Install the MariaDB Platform Operator, select the "A specific namespace on the cluster" choose namespace "test-chuo" Actual results: Webconle result: Failed: install failed: Timeout: deployment mariadb-platform-operator not ready before timeout: deployment "mariadb-platform-operator" exceeded its progress deadline Backend result: oc get csv -n test-chuo | grep mariadb mariadb-platform-operator-0.1.3-3 MariaDB Platform Operator 0.1.3-3 Failed oc describe csv mariadb-platform-operator-0.1.3-3 -n test-chuo Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 27m operator-lifecycle-manager requirements not yet checked Warning InstallCheckFailed 22m operator-lifecycle-manager install timeout Normal AllRequirementsMet 22m (x2 over 27m) operator-lifecycle-manager all requirements found, attempting install Normal NeedsReinstall 22m operator-lifecycle-manager installing: Waiting: waiting for deployment mariadb-platform-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available... Normal InstallSucceeded 22m (x4 over 27m) operator-lifecycle-manager waiting for install components to report healthy Normal InstallWaiting 22m (x4 over 27m) operator-lifecycle-manager installing: Waiting: waiting for deployment mariadb-platform-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available... Warning InstallCheckFailed 17m (x2 over 17m) operator-lifecycle-manager install failed: Timeout: deployment mariadb-platform-operator not ready before timeout: deployment "mariadb-platform-operator" exceeded its progress deadline Expected results: MariaDB Platform Operator should be installed successfully Additional info: oc get sub -n test-chuo -o yaml apiVersion: v1 items: - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: creationTimestamp: "2019-05-27T10:31:59Z" generation: 1 labels: csc-owner-name: installed-certified-test-chuo csc-owner-namespace: openshift-marketplace name: mariadb namespace: test-chuo resourceVersion: "166921" selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/test-chuo/subscriptions/mariadb uid: a8a774fd-806a-11e9-a074-0203904d834c spec: channel: alpha installPlanApproval: Automatic name: mariadb source: installed-certified-test-chuo sourceNamespace: test-chuo startingCSV: mariadb-platform-operator-0.1.3-3 status: currentCSV: mariadb-platform-operator-0.1.3-3 installPlanRef: apiVersion: operators.coreos.com/v1alpha1 kind: InstallPlan name: install-7bshg namespace: test-chuo resourceVersion: "166826" uid: bbaa6dd6-806a-11e9-a074-0203904d834c installedCSV: mariadb-platform-operator-0.1.3-3 installplan: apiVersion: operators.coreos.com/v1alpha1 kind: InstallPlan name: install-7bshg uuid: bbaa6dd6-806a-11e9-a074-0203904d834c lastUpdated: "2019-05-27T10:32:39Z" state: AtLatestKnown kind: List metadata: resourceVersion: "" selfLink: "" oc get ip -n test-chuo | grep mariadb install-7bshg mariadb-platform-operator-0.1.3-3 Automatic true oc logs catalog-operator-9474f495-q6vfc -n openshift-operator-lifecycle-manager | grep mariadb E0527 10:31:59.069544 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:31:59.079841 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:31:59.100106 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:31:59.140396 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:09.446339 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:09.456590 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:09.477018 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:09.517310 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:19.847150 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:19.857479 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:19.877854 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found E0527 10:32:19.918168 1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found oc get csc -n openshift-marketplace | grep test-chuo installed-certified-test-chuo Succeeded The object has been successfully reconciled 32m oc get catsrc installed-certified-test-chuo -n test-chuo -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: creationTimestamp: "2019-05-27T10:31:58Z" generation: 1 labels: csc-owner-name: installed-certified-test-chuo csc-owner-namespace: openshift-marketplace name: installed-certified-test-chuo namespace: test-chuo resourceVersion: "184081" selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/test-chuo/catalogsources/installed-certified-test-chuo uid: a84ac837-806a-11e9-a074-0203904d834c spec: address: 172.30.200.143:50051 displayName: Certified Operators icon: base64data: "" mediatype: "" publisher: Certified sourceType: grpc status: lastSync: "2019-05-27T11:00:05Z" registryService: createdAt: "2019-05-27T11:00:05Z" protocol: grpc oc logs installed-certified-test-chuo-f9959dc75-2fg5c -n openshift-marketplace time="2019-05-27T10:32:06Z" level=info msg="Using in-cluster kube client config" port=50051 type=appregistry time="2019-05-27T10:32:06Z" level=info msg="operator source(s) specified are - [https://quay.io/cnr|certified-operators]" port=50051 type=appregistry time="2019-05-27T10:32:06Z" level=info msg="package(s) specified are - mariadb" port=50051 type=appregistry time="2019-05-27T10:32:06Z" level=info msg="input has been sanitized" port=50051 type=appregistry time="2019-05-27T10:32:06Z" level=info msg="sources: [https://quay.io/cnr/certified-operators]" port=50051 type=appregistry time="2019-05-27T10:32:06Z" level=info msg="packages: [mariadb]" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="resolved the following packages: [certified-operators/mariadb:3.0.0]" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="downloading repository: certified-operators/mariadb:3.0.0 from https://quay.io/cnr" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="download complete - 1 repositories have been downloaded" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="decoding the downloaded operator manifest(s)" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="manifest format is - flattened" port=50051 repository="certified-operators/mariadb:3.0.0" type=appregistry time="2019-05-27T10:32:07Z" level=info msg="decoded successfully" port=50051 repository="certified-operators/mariadb:3.0.0" type=appregistry time="2019-05-27T10:32:07Z" level=info msg="merging all flattened manifests into a single configmap 'data' section" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="decoded 1 flattened and 0 nested operator manifest(s)" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="loading flattened operator manifest(s) into sqlite" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="using configmap loader to build sqlite database" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="loading CRDs" port=50051 type=appregistry time="2019-05-27T10:32:07Z" level=info msg="loading Bundles" port=50051 type=appregistry time="2019-05-27T10:32:08Z" level=info msg="loading Packages" port=50051 type=appregistry time="2019-05-27T10:32:08Z" level=info msg="extracting provided API information" port=50051 type=appregistry time="2019-05-27T10:32:08Z" level=info msg="serving registry" port=50051 type=appregistry
This is a 3rd party operator provided by MariaDB Corporation More logs: oc describe po mariadb-platform-operator-c9c9986fd-5ngwh Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 86s default-scheduler Successfully assigned chuo/mariadb-platform-operator-c9c9986fd-5ngwh to dell-r730-067.dsal.lab.eng.rdu2.redhat.com Normal Pulling 40s (x3 over 79s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3" Warning Failed 40s (x3 over 77s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Failed to pull image "registry.connect.redhat.com/mariadb/operator:0.1.3-3": rpc error: code = Unknown desc = Error reading manifest 0.1.3-3 in registry.connect.redhat.com/mariadb/operator: manifest unknown: manifest unknown Warning Failed 40s (x3 over 77s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Error: ErrImagePull Normal BackOff 2s (x5 over 76s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Back-off pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3" Warning Failed 2s (x5 over 76s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Error: ImagePullBackOff sudo docker pull registry.connect.redhat.com/mariadb/operator:0.1.3-3 Trying to pull repository registry.connect.redhat.com/mariadb/operator ... manifest for registry.connect.redhat.com/mariadb/operator:0.1.3-3 not found [root@dhcp-140-36 jzhang]# docker pull registry.connect.redhat.com/mariadb/operator:latest Trying to pull repository registry.connect.redhat.com/mariadb/operator ... sha256:51e07a840b389475f15ae020f488fbad555834c0165cd7289d810858fa2a9b09: Pulling from registry.connect.redhat.com/mariadb/operator d69140bdce18: Already exists a82dd37af30d: Already exists c640798d1b54: Pull complete fc3fc23c158d: Pull complete 97ddebb26e36: Pull complete ad718a26d272: Pull complete a7fb88628f85: Pull complete 00598404fc54: Pull complete 97aa797cbcb5: Pull complete 659ea22b0133: Pull complete 18153de7c1bd: Pull complete cb68538ebf7b: Pull complete 9fe62901bb0f: Pull complete Digest: sha256:51e07a840b389475f15ae020f488fbad555834c0165cd7289d810858fa2a9b09 Status: Downloaded newer image for registry.connect.redhat.com/mariadb/operator:latest
This is operator itself issue. Move it to OperatorHub first. @Aravindh Do you know where's it from? I couldn't find it in https://github.com/operator-framework/community-operators/tree/release-4.1
Looks like it is a certified operator. Brian Cook, how should we report such issues?
This issue should now be resolved. I verified yesterday that the image tag 0.1.3-3 does not exist in RHCC. It appears that the partner (MariaDB) published a new image tag (0.1.3-4a) to RHCC, and unpublished the previous image tag (0.1.3-3). I've republished the image to RHCC and can verify that the image is pullable.
Verified failed Cluster version is 4.1.0-0.nightly-2019-05-24-040103 with error: "Failed to pull image "registry.connect.redhat.com/mariadb/operator:0.1.3-3": rpc error: code = Unknown desc = Error reading manifest 0.1.3-3 in registry.connect.redhat.com/mariadb/operator: manifest unknown: manifest unknown" @Josh Manning, it seems that there is no image tag with 0.1.3-3 here in https://access.redhat.com/containers/?tab=tags#/registry.connect.redhat.com/mariadb/operator [chuo@dhcp-140-165 .kube]$ oc get sub -n chuo NAME PACKAGE SOURCE CHANNEL mariadb mariadb installed-certified-chuo alpha [chuo@dhcp-140-165 .kube]$ oc get ip -n chuo NAME CSV SOURCE APPROVAL APPROVED install-jfhp6 mariadb-platform-operator-0.1.3-3 Automatic true [chuo@dhcp-140-165 .kube]$ oc get csv -n chuo NAME DISPLAY VERSION REPLACES PHASE anchore-engine-operator.v0.0.1 Anchore Engine Operator 0.0.1 Succeeded descheduler.v0.0.4 Descheduler 0.0.4 descheduler.v0.0.3 Succeeded elasticsearch-operator.v4.3.0 Elasticsearch Operator 4.3.0 elasticsearch-operator.v4.1.0 Succeeded mariadb-platform-operator-0.1.3-3 MariaDB Platform Operator 0.1.3-3 Installing mongodboperator.v0.3.2 MongoDB 0.3.2 [chuo@dhcp-140-165 .kube]$ oc get po -n chuo NAME READY STATUS RESTARTS AGE mariadb-platform-operator-c9c9986fd-mr5cn 0/1 ContainerCreating 0 9s [chuo@dhcp-140-165 .kube]$ oc get po -n chuo NAME READY STATUS RESTARTS AGE mariadb-platform-operator-c9c9986fd-mr5cn 0/1 ErrImagePull 0 16s [chuo@dhcp-140-165 .kube]$ oc describe po mariadb-platform-operator-c9c9986fd-mr5cn -n chuo Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 32s default-scheduler Successfully assigned chuo/mariadb-platform-operator-c9c9986fd-mr5cn to dell-r730-067.dsal.lab.eng.rdu2.redhat.com Warning FailedScheduling 32s default-scheduler Binding rejected: Operation cannot be fulfilled on pods/binding "mariadb-platform-operator-c9c9986fd-mr5cn": pod mariadb-platform-operator-c9c9986fd-mr5cn is already assigned to node "dell-r730-067.dsal.lab.eng.rdu2.redhat.com" Warning FailedScheduling 32s default-scheduler Binding rejected: Operation cannot be fulfilled on pods/binding "mariadb-platform-operator-c9c9986fd-mr5cn": pod mariadb-platform-operator-c9c9986fd-mr5cn is already assigned to node "dell-r730-067.dsal.lab.eng.rdu2.redhat.com" Normal BackOff 21s kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Back-off pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3" Warning Failed 21s kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Error: ImagePullBackOff Normal Pulling 6s (x2 over 24s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3" Warning Failed 5s (x2 over 22s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Failed to pull image "registry.connect.redhat.com/mariadb/operator:0.1.3-3": rpc error: code = Unknown desc = Error reading manifest 0.1.3-3 in registry.connect.redhat.com/mariadb/operator: manifest unknown: manifest unknown Warning Failed 5s (x2 over 22s) kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com Error: ErrImagePull
Hmm... I published this yesterday (and verified by pulling the image afterward) and it appears that it was unpublished since then. I've published the image again, and will inform the partner so that they don't unpublish the image.
Verified Cluster version is 4.1.0-rc.7 [chuo@dhcp-140-165 .kube]$ oc get sub -n chuo NAME PACKAGE SOURCE CHANNEL mariadb mariadb installed-certified-chuo alpha opsmx-spinnaker-operator opsmx-spinnaker-operator installed-community-chuo alpha [chuo@dhcp-140-165 .kube]$ oc get ip -n chuo NAME CSV SOURCE APPROVAL APPROVED install-fjjj6 mariadb-platform-operator-0.1.3-3 Automatic true [chuo@dhcp-140-165 .kube]$ oc get csv -n chuo NAME DISPLAY VERSION REPLACES PHASE amqstreams.v1.1.0 AMQ Streams 1.1.0 amqstreams.v1.0.0 Succeeded appdynamics-operator.0.1.4 AppDynamics ClusterAgent 0.1.4 Succeeded camel-k-operator.v0.2.0 Camel-K Operator 0.2.0 Succeeded descheduler.v0.0.4 Descheduler 0.0.4 descheduler.v0.0.3 Succeeded elasticsearch-operator.v4.1.0 Elasticsearch Operator 4.1.0 Succeeded federatorai.v0.1.0 Federator.ai 0.1.0 federatorai.v0.0.1 Succeeded mariadb-platform-operator-0.1.3-3 MariaDB Platform Operator 0.1.3-3 Installing mongodboperator.v0.3.2 MongoDB 0.3.2 Installing opsmx-spinnaker-operator.v1.13.1 Opsmx Spinnaker Operator 1.13.1 Succeeded [chuo@dhcp-140-165 .kube]$ oc get po -n chuo NAME READY STATUS RESTARTS AGE mariadb-platform-operator-86ddfbb7c8-6vmlz 1/1 Running 0 21s [chuo@dhcp-140-165 .kube]$ oc describe po mariadb-platform-operator-86ddfbb7c8-6vmlz -n chuo Name: mariadb-platform-operator-86ddfbb7c8-6vmlz Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 41s default-scheduler Successfully assigned chuo/mariadb-platform-operator-86ddfbb7c8-6vmlz to dell-r730-007.dsal.lab.eng.rdu2.redhat.com Warning FailedScheduling 41s default-scheduler Binding rejected: Operation cannot be fulfilled on pods/binding "mariadb-platform-operator-86ddfbb7c8-6vmlz": pod mariadb-platform-operator-86ddfbb7c8-6vmlz is already assigned to node "dell-r730-007.dsal.lab.eng.rdu2.redhat.com" Normal Pulling 33s kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com Pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3" Normal Pulled 22s kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com Successfully pulled image "registry.connect.redhat.com/mariadb/operator:0.1.3-3" Normal Created 22s kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com Created container controller Normal Started 22s kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com Started container controller
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1382
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days