Bug 1714200

Summary: mariadb-platform-operator can not be installed for OCP4.1 due to "ErrImagePull"
Product: OpenShift Container Platform Reporter: Cuiping HUO <chuo>
Component: OLMAssignee: tonyc
OLM sub component: OperatorHub QA Contact: Cuiping HUO <chuo>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aravindh, bandrade, bcook, chezhang, dyan, jfan, jiazhang, jiazha, jmanning, scolange, sponnaga, zitang
Version: 4.1.0Keywords: OSE41z_next
Target Milestone: ---Flags: aravindh: needinfo? (bcook)
Target Release: 4.1.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: 4.1.2
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-19 06:45:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1718956    

Description Cuiping HUO 2019-05-27 11:15:25 UTC
Description of problem:
MariaDB Platform Operator can not be installed on OCP4.1

Version-Release number of selected component (if applicable):
Cluster version is 4.1.0-0.nightly-2019-05-24-040103
MariaDB Platform Operator is 0.1.3-3 provided by MariaDB Corporation

How reproducible:
Always

Steps to Reproduce:
1.Install the MariaDB Platform Operator, select the "A specific namespace on the cluster" choose namespace "test-chuo"


Actual results:

Webconle result:
Failed: install failed: Timeout: deployment mariadb-platform-operator not ready before timeout: deployment "mariadb-platform-operator" exceeded its progress deadline

Backend result:
oc get csv -n test-chuo | grep mariadb
mariadb-platform-operator-0.1.3-3   MariaDB Platform Operator   0.1.3-3                        Failed

oc describe csv mariadb-platform-operator-0.1.3-3 -n test-chuo
Events:
  Type     Reason               Age                From                        Message
  ----     ------               ----               ----                        -------
  Normal   RequirementsUnknown  27m                operator-lifecycle-manager  requirements not yet checked
  Warning  InstallCheckFailed   22m                operator-lifecycle-manager  install timeout
  Normal   AllRequirementsMet   22m (x2 over 27m)  operator-lifecycle-manager  all requirements found, attempting install
  Normal   NeedsReinstall       22m                operator-lifecycle-manager  installing: Waiting: waiting for deployment mariadb-platform-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...
  Normal   InstallSucceeded     22m (x4 over 27m)  operator-lifecycle-manager  waiting for install components to report healthy
  Normal   InstallWaiting       22m (x4 over 27m)  operator-lifecycle-manager  installing: Waiting: waiting for deployment mariadb-platform-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...
  Warning  InstallCheckFailed   17m (x2 over 17m)  operator-lifecycle-manager  install failed: Timeout: deployment mariadb-platform-operator not ready before timeout: deployment "mariadb-platform-operator" exceeded its progress deadline



Expected results:
MariaDB Platform Operator should be installed successfully


Additional info:
oc get sub -n test-chuo -o yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    creationTimestamp: "2019-05-27T10:31:59Z"
    generation: 1
    labels:
      csc-owner-name: installed-certified-test-chuo
      csc-owner-namespace: openshift-marketplace
    name: mariadb
    namespace: test-chuo
    resourceVersion: "166921"
    selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/test-chuo/subscriptions/mariadb
    uid: a8a774fd-806a-11e9-a074-0203904d834c
  spec:
    channel: alpha
    installPlanApproval: Automatic
    name: mariadb
    source: installed-certified-test-chuo
    sourceNamespace: test-chuo
    startingCSV: mariadb-platform-operator-0.1.3-3
  status:
    currentCSV: mariadb-platform-operator-0.1.3-3
    installPlanRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: InstallPlan
      name: install-7bshg
      namespace: test-chuo
      resourceVersion: "166826"
      uid: bbaa6dd6-806a-11e9-a074-0203904d834c
    installedCSV: mariadb-platform-operator-0.1.3-3
    installplan:
      apiVersion: operators.coreos.com/v1alpha1
      kind: InstallPlan
      name: install-7bshg
      uuid: bbaa6dd6-806a-11e9-a074-0203904d834c
    lastUpdated: "2019-05-27T10:32:39Z"
    state: AtLatestKnown
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

oc get ip -n test-chuo | grep mariadb
install-7bshg   mariadb-platform-operator-0.1.3-3            Automatic   true

oc logs catalog-operator-9474f495-q6vfc -n openshift-operator-lifecycle-manager | grep mariadb
E0527 10:31:59.069544       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:31:59.079841       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:31:59.100106       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:31:59.140396       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:09.446339       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:09.456590       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:09.477018       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:09.517310       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:19.847150       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:19.857479       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:19.877854       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found
E0527 10:32:19.918168       1 queueinformer_operator.go:186] Sync "test-chuo" failed: {mariadb alpha mariadb-platform-operator-0.1.3-3 {installed-certified-test-chuo test-chuo}} not found: CatalogSource {installed-certified-test-chuo test-chuo} not found

oc get csc -n openshift-marketplace | grep test-chuo
installed-certified-test-chuo                       Succeeded   The object has been successfully reconciled   32m

oc get catsrc installed-certified-test-chuo -n test-chuo -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  creationTimestamp: "2019-05-27T10:31:58Z"
  generation: 1
  labels:
    csc-owner-name: installed-certified-test-chuo
    csc-owner-namespace: openshift-marketplace
  name: installed-certified-test-chuo
  namespace: test-chuo
  resourceVersion: "184081"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/test-chuo/catalogsources/installed-certified-test-chuo
  uid: a84ac837-806a-11e9-a074-0203904d834c
spec:
  address: 172.30.200.143:50051
  displayName: Certified Operators
  icon:
    base64data: ""
    mediatype: ""
  publisher: Certified
  sourceType: grpc
status:
  lastSync: "2019-05-27T11:00:05Z"
  registryService:
    createdAt: "2019-05-27T11:00:05Z"
    protocol: grpc


oc logs installed-certified-test-chuo-f9959dc75-2fg5c  -n openshift-marketplace
time="2019-05-27T10:32:06Z" level=info msg="Using in-cluster kube client config" port=50051 type=appregistry
time="2019-05-27T10:32:06Z" level=info msg="operator source(s) specified are - [https://quay.io/cnr|certified-operators]" port=50051 type=appregistry
time="2019-05-27T10:32:06Z" level=info msg="package(s) specified are - mariadb" port=50051 type=appregistry
time="2019-05-27T10:32:06Z" level=info msg="input has been sanitized" port=50051 type=appregistry
time="2019-05-27T10:32:06Z" level=info msg="sources: [https://quay.io/cnr/certified-operators]" port=50051 type=appregistry
time="2019-05-27T10:32:06Z" level=info msg="packages: [mariadb]" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="resolved the following packages: [certified-operators/mariadb:3.0.0]" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="downloading repository: certified-operators/mariadb:3.0.0 from https://quay.io/cnr" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="download complete - 1 repositories have been downloaded" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="decoding the downloaded operator manifest(s)" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="manifest format is - flattened" port=50051 repository="certified-operators/mariadb:3.0.0" type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="decoded successfully" port=50051 repository="certified-operators/mariadb:3.0.0" type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="merging all flattened manifests into a single configmap 'data' section" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="decoded 1 flattened and 0 nested operator manifest(s)" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="loading flattened operator manifest(s) into sqlite" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="using configmap loader to build sqlite database" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="loading CRDs" port=50051 type=appregistry
time="2019-05-27T10:32:07Z" level=info msg="loading Bundles" port=50051 type=appregistry
time="2019-05-27T10:32:08Z" level=info msg="loading Packages" port=50051 type=appregistry
time="2019-05-27T10:32:08Z" level=info msg="extracting provided API information" port=50051 type=appregistry
time="2019-05-27T10:32:08Z" level=info msg="serving registry" port=50051 type=appregistry

Comment 1 Cuiping HUO 2019-05-28 03:12:52 UTC
This is a 3rd party operator provided by MariaDB Corporation

More logs:
oc describe po mariadb-platform-operator-c9c9986fd-5ngwh
Events:
  Type     Reason     Age                From                                                 Message
  ----     ------     ----               ----                                                 -------
  Normal   Scheduled  86s                default-scheduler                                    Successfully assigned chuo/mariadb-platform-operator-c9c9986fd-5ngwh to dell-r730-067.dsal.lab.eng.rdu2.redhat.com
  Normal   Pulling    40s (x3 over 79s)  kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3"
  Warning  Failed     40s (x3 over 77s)  kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Failed to pull image "registry.connect.redhat.com/mariadb/operator:0.1.3-3": rpc error: code = Unknown desc = Error reading manifest 0.1.3-3 in registry.connect.redhat.com/mariadb/operator: manifest unknown: manifest unknown
  Warning  Failed     40s (x3 over 77s)  kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Error: ErrImagePull
  Normal   BackOff    2s (x5 over 76s)   kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Back-off pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3"
  Warning  Failed     2s (x5 over 76s)   kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Error: ImagePullBackOff


sudo docker  pull registry.connect.redhat.com/mariadb/operator:0.1.3-3
Trying to pull repository registry.connect.redhat.com/mariadb/operator ... 
manifest for registry.connect.redhat.com/mariadb/operator:0.1.3-3 not found
[root@dhcp-140-36 jzhang]# docker  pull registry.connect.redhat.com/mariadb/operator:latest
Trying to pull repository registry.connect.redhat.com/mariadb/operator ... 
sha256:51e07a840b389475f15ae020f488fbad555834c0165cd7289d810858fa2a9b09: Pulling from registry.connect.redhat.com/mariadb/operator
d69140bdce18: Already exists 
a82dd37af30d: Already exists 
c640798d1b54: Pull complete 
fc3fc23c158d: Pull complete 
97ddebb26e36: Pull complete 
ad718a26d272: Pull complete 
a7fb88628f85: Pull complete 
00598404fc54: Pull complete 
97aa797cbcb5: Pull complete 
659ea22b0133: Pull complete 
18153de7c1bd: Pull complete 
cb68538ebf7b: Pull complete 
9fe62901bb0f: Pull complete 
Digest: sha256:51e07a840b389475f15ae020f488fbad555834c0165cd7289d810858fa2a9b09
Status: Downloaded newer image for registry.connect.redhat.com/mariadb/operator:latest

Comment 2 Jian Zhang 2019-05-28 03:34:07 UTC
This is operator itself issue. Move it to OperatorHub first.

@Aravindh
Do you know where's it from? I couldn't find it in https://github.com/operator-framework/community-operators/tree/release-4.1

Comment 3 Aravindh Puthiyaparambil 2019-05-28 13:58:47 UTC
Looks like it is a certified operator. 

Brian Cook, how should we report such issues?

Comment 4 Josh Manning 2019-05-29 13:45:36 UTC
This issue should now be resolved. I verified yesterday that the image tag 0.1.3-3 does not exist in RHCC. It appears that the partner (MariaDB) published a new image tag (0.1.3-4a) to RHCC, and unpublished the previous image tag (0.1.3-3). I've republished the image to RHCC and can verify that the image is pullable.

Comment 5 Cuiping HUO 2019-05-30 02:24:39 UTC
Verified failed 
Cluster version is 4.1.0-0.nightly-2019-05-24-040103

with error: "Failed to pull image "registry.connect.redhat.com/mariadb/operator:0.1.3-3": rpc error: code = Unknown desc = Error reading manifest 0.1.3-3 in registry.connect.redhat.com/mariadb/operator: manifest unknown: manifest unknown"

@Josh Manning, it seems that there is no image tag with 0.1.3-3 here in https://access.redhat.com/containers/?tab=tags#/registry.connect.redhat.com/mariadb/operator

[chuo@dhcp-140-165 .kube]$ oc get sub -n chuo
NAME      PACKAGE   SOURCE                     CHANNEL
mariadb   mariadb   installed-certified-chuo   alpha
[chuo@dhcp-140-165 .kube]$ oc get ip -n chuo
NAME            CSV                                 SOURCE   APPROVAL    APPROVED
install-jfhp6   mariadb-platform-operator-0.1.3-3            Automatic   true
[chuo@dhcp-140-165 .kube]$ oc  get csv -n chuo
NAME                                DISPLAY                     VERSION   REPLACES                        PHASE
anchore-engine-operator.v0.0.1      Anchore Engine Operator     0.0.1                                     Succeeded
descheduler.v0.0.4                  Descheduler                 0.0.4     descheduler.v0.0.3              Succeeded
elasticsearch-operator.v4.3.0       Elasticsearch Operator      4.3.0     elasticsearch-operator.v4.1.0   Succeeded
mariadb-platform-operator-0.1.3-3   MariaDB Platform Operator   0.1.3-3                                   Installing
mongodboperator.v0.3.2              MongoDB                     0.3.2                                     
[chuo@dhcp-140-165 .kube]$ oc get po -n chuo
NAME                                        READY   STATUS              RESTARTS   AGE
mariadb-platform-operator-c9c9986fd-mr5cn   0/1     ContainerCreating   0          9s
[chuo@dhcp-140-165 .kube]$ oc get po -n chuo
NAME                                        READY   STATUS         RESTARTS   AGE
mariadb-platform-operator-c9c9986fd-mr5cn   0/1     ErrImagePull   0          16s
[chuo@dhcp-140-165 .kube]$ oc describe po mariadb-platform-operator-c9c9986fd-mr5cn -n chuo
Events:
  Type     Reason            Age               From                                                 Message
  ----     ------            ----              ----                                                 -------
  Normal   Scheduled         32s               default-scheduler                                    Successfully assigned chuo/mariadb-platform-operator-c9c9986fd-mr5cn to dell-r730-067.dsal.lab.eng.rdu2.redhat.com
  Warning  FailedScheduling  32s               default-scheduler                                    Binding rejected: Operation cannot be fulfilled on pods/binding "mariadb-platform-operator-c9c9986fd-mr5cn": pod mariadb-platform-operator-c9c9986fd-mr5cn is already assigned to node "dell-r730-067.dsal.lab.eng.rdu2.redhat.com"
  Warning  FailedScheduling  32s               default-scheduler                                    Binding rejected: Operation cannot be fulfilled on pods/binding "mariadb-platform-operator-c9c9986fd-mr5cn": pod mariadb-platform-operator-c9c9986fd-mr5cn is already assigned to node "dell-r730-067.dsal.lab.eng.rdu2.redhat.com"
  Normal   BackOff           21s               kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Back-off pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3"
  Warning  Failed            21s               kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Error: ImagePullBackOff
  Normal   Pulling           6s (x2 over 24s)  kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3"
  Warning  Failed            5s (x2 over 22s)  kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Failed to pull image "registry.connect.redhat.com/mariadb/operator:0.1.3-3": rpc error: code = Unknown desc = Error reading manifest 0.1.3-3 in registry.connect.redhat.com/mariadb/operator: manifest unknown: manifest unknown
  Warning  Failed            5s (x2 over 22s)  kubelet, dell-r730-067.dsal.lab.eng.rdu2.redhat.com  Error: ErrImagePull

Comment 6 Josh Manning 2019-05-30 17:15:52 UTC
Hmm... I published this yesterday (and verified by pulling the image afterward) and it appears that it was unpublished since then. I've published the image again, and will inform the partner so that they don't unpublish the image.

Comment 7 Cuiping HUO 2019-05-31 02:49:38 UTC
Verified 
Cluster version is 4.1.0-rc.7


[chuo@dhcp-140-165 .kube]$ oc get sub -n chuo
NAME                       PACKAGE                    SOURCE                     CHANNEL
mariadb                    mariadb                    installed-certified-chuo   alpha
opsmx-spinnaker-operator   opsmx-spinnaker-operator   installed-community-chuo   alpha
[chuo@dhcp-140-165 .kube]$ oc get ip -n chuo
NAME            CSV                                 SOURCE   APPROVAL    APPROVED
install-fjjj6   mariadb-platform-operator-0.1.3-3            Automatic   true
[chuo@dhcp-140-165 .kube]$ oc get csv -n chuo
NAME                                DISPLAY                     VERSION   REPLACES             PHASE
amqstreams.v1.1.0                   AMQ Streams                 1.1.0     amqstreams.v1.0.0    Succeeded
appdynamics-operator.0.1.4          AppDynamics ClusterAgent    0.1.4                          Succeeded
camel-k-operator.v0.2.0             Camel-K Operator            0.2.0                          Succeeded
descheduler.v0.0.4                  Descheduler                 0.0.4     descheduler.v0.0.3   Succeeded
elasticsearch-operator.v4.1.0       Elasticsearch Operator      4.1.0                          Succeeded
federatorai.v0.1.0                  Federator.ai                0.1.0     federatorai.v0.0.1   Succeeded
mariadb-platform-operator-0.1.3-3   MariaDB Platform Operator   0.1.3-3                        Installing
mongodboperator.v0.3.2              MongoDB                     0.3.2                          Installing
opsmx-spinnaker-operator.v1.13.1    Opsmx Spinnaker Operator    1.13.1                         Succeeded
[chuo@dhcp-140-165 .kube]$ oc get po -n chuo
NAME                                         READY   STATUS    RESTARTS   AGE
mariadb-platform-operator-86ddfbb7c8-6vmlz   1/1     Running   0          21s

[chuo@dhcp-140-165 .kube]$ oc describe po mariadb-platform-operator-86ddfbb7c8-6vmlz -n chuo
Name:               mariadb-platform-operator-86ddfbb7c8-6vmlz
Events:
  Type     Reason            Age   From                                                 Message
  ----     ------            ----  ----                                                 -------
  Normal   Scheduled         41s   default-scheduler                                    Successfully assigned chuo/mariadb-platform-operator-86ddfbb7c8-6vmlz to dell-r730-007.dsal.lab.eng.rdu2.redhat.com
  Warning  FailedScheduling  41s   default-scheduler                                    Binding rejected: Operation cannot be fulfilled on pods/binding "mariadb-platform-operator-86ddfbb7c8-6vmlz": pod mariadb-platform-operator-86ddfbb7c8-6vmlz is already assigned to node "dell-r730-007.dsal.lab.eng.rdu2.redhat.com"
  Normal   Pulling           33s   kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com  Pulling image "registry.connect.redhat.com/mariadb/operator:0.1.3-3"
  Normal   Pulled            22s   kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com  Successfully pulled image "registry.connect.redhat.com/mariadb/operator:0.1.3-3"
  Normal   Created           22s   kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com  Created container controller
  Normal   Started           22s   kubelet, dell-r730-007.dsal.lab.eng.rdu2.redhat.com  Started container controller

Comment 9 errata-xmlrpc 2019-06-19 06:45:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1382