Bug 1802212 - MCG/NooBaa noobaa-core and PV Pool statefulset pods are created using an upstream image/tag
Summary: MCG/NooBaa noobaa-core and PV Pool statefulset pods are created using an upst...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: ocs-operator
Version: 4.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: OCS 4.3.0
Assignee: Jacky Albo
QA Contact: Ben Eli
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-12 15:45 UTC by Jason Montleon
Modified: 2020-09-23 09:04 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-14 09:45:56 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ocs-operator pull 381 0 None closed Adding noobaa's core and db images to a new environment variables for the noobaa operator 2020-09-28 08:12:06 UTC
Red Hat Product Errata RHBA-2020:1437 0 None None None 2020-04-14 09:48:16 UTC

Description Jason Montleon 2020-02-12 15:45:39 UTC
Description of problem (please be detailed as possible and provide log
snippests):

When installing OpenShift Container Storage from OLM on OpenShift 4.3 the noobaa-core-0 and noobaa statefulset pods for PV Pools are created using an upstream image.


Version of all relevant components (if applicable):
4.2.1

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
It works, but our requirement to work in disconnected/restricted network environments is limited due to this issue.

Is there any workaround available to the best of your knowledge?
No.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes.

Can this issue reproduce from the UI?
No.

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install OCS operator
2. Create a NooBaa System
3. Create a PV Pool Backing Store

apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  labels:
    app: noobaa
  name: mcg-pv-pool-bs
  namespace: openshift-storage
spec:
  pvPool:
    numVolumes: 3
    resources:
      requests:
        storage: 50Gi
    storageClass: gp2
  type: pv-pool

Actual results:
The downstream operator:
$ oc describe po noobaa-operator-59894b6668-l6l6s | grep Image
    Image:          registry.redhat.io/ocs4/mcg-rhel8-operator@sha256:ecb1c4e486aca77bda000fa71d190f90e06173447cff1bed983fb6e246975863
    Image ID:       registry.redhat.io/ocs4/mcg-rhel8-operator@sha256:ecb1c4e486aca77bda000fa71d190f90e06173447cff1bed983fb6e246975863

is creating the noobaa core and statefulset pods using upstream images:
$ oc describe po mcg-pv-pool-bs-noobaa-noobaa-2 | grep Image
    Image:         noobaa/noobaa-core:5.2.11
    Image ID:      docker.io/noobaa/noobaa-core@sha256:4389a1bb98083946b7837837613e7e64829333b166a2acd664093ced3e034fa6

$ oc describe po mcg-pv-pool-bs-noobaa-noobaa-2 | grep Image
    Image:         noobaa/noobaa-core:5.2.11
    Image ID:      docker.io/noobaa/noobaa-core@sha256:4389a1bb98083946b7837837613e7e64829333b166a2acd664093ced3e034fa6
[jmontleo@jmontleo mig-operator]$ oc describe po noobaa-core-0 | grep Image
    Image:         noobaa/noobaa-core:5.2.11
    Image ID:      docker.io/noobaa/noobaa-core@sha256:4389a1bb98083946b7837837613e7e64829333b166a2acd664093ced3e034fa6
    Image:          noobaa/noobaa-core:5.2.11
    Image ID:       docker.io/noobaa/noobaa-core@sha256:4389a1bb98083946b7837837613e7e64829333b166a2acd664093ced3e034fa6
    Image:         centos/mongodb-36-centos7
    Image ID:      docker.io/centos/mongodb-36-centos7@sha256:48a9fc021f16138cda94dc36914610df78caac36d751218ae499cc9c6363194b

Expected results:
The noobaa core and statefulset pods are created using the downstream images referenced in the CSV:
registry.redhat.io/ocs4/mcg-core-rhel8@sha256:1906a018df2774f1d0338483255c5f314fe707afcf12271f53a346790de9fdcf
registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:506e4daa19721199e35249f2e0e67b02cbcbee1fed653f7eb91d5595b74c3bf8

Additional info:
Tying into https://bugzilla.redhat.com/show_bug.cgi?id=1798052 tags cannot be used to mirror images in OpenShift 4 because mirror by tag is turned off in OpenShift 4 cri-o, making it impossible to use noobaa in a disconnected environment even if you do determine the list of images to mirror and do so manually.

Comment 2 Jason Montleon 2020-02-12 15:48:06 UTC
ocs-operator has env vars for these images but they are not being used:

$ oc describe po ocs-operator-5694b5bc77-mnk7l | grep -e mcg -e mongo
      NOOBAA_CORE_IMAGE:   registry.redhat.io/ocs4/mcg-core-rhel8@sha256:1906a018df2774f1d0338483255c5f314fe707afcf12271f53a346790de9fdcf
      NOOBAA_DB_IMAGE:     registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:506e4daa19721199e35249f2e0e67b02cbcbee1fed653f7eb91d5595b74c3bf8

Comment 3 Nimrod Becker 2020-02-13 10:32:47 UTC
For 4.3, we provided the ability to set ENV on the ocs-operator and provide the desired images (attached the PR).

Comment 6 Jason Montleon 2020-02-14 19:55:45 UTC
Nimrod I see these Env vars on the ocs-operator per my comment https://bugzilla.redhat.com/show_bug.cgi?id=1802212#c2

Is there a bug in 4.2.1 where they're not being used that's fixed or getting fixed in OpenShift Container Storage 4.3?

Comment 7 Nimrod Becker 2020-02-17 06:21:56 UTC
@Jason

Exactly. In 4.2.x we did not use them, now we added them and we should use (pass the correct) value which is targeted for 4.3

Comment 8 Michael Adam 2020-03-04 14:15:09 UTC
The feature freeze build has this Patch:
https://ceph-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/OCS%20Build%20Pipeline%204.3/82/

Comment 10 Ben Eli 2020-03-12 17:32:11 UTC
Seems to have been fixed.

> oc describe po mcg-pv-pool-bs-noobaa-noobaa-2 | grep Image
    Image:         quay.io/rhceph-dev/mcg-core@sha256:9fdf00038cfb64f044ea71e4aaa76980c5e0407b462eb326e051377d9669a3b6
    Image ID:      quay.io/rhceph-dev/mcg-core@sha256:9fdf00038cfb64f044ea71e4aaa76980c5e0407b462eb326e051377d9669a3b6
> oc describe po noobaa-core-0 | grep Image
    Image:          quay.io/rhceph-dev/mcg-core@sha256:9fdf00038cfb64f044ea71e4aaa76980c5e0407b462eb326e051377d9669a3b6
    Image ID:       quay.io/rhceph-dev/mcg-core@sha256:9fdf00038cfb64f044ea71e4aaa76980c5e0407b462eb326e051377d9669a3b6

Verified.
OCS 4.3.0-373.ci

Comment 12 errata-xmlrpc 2020-04-14 09:45:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1437


Note You need to log in before you can comment on or make changes to this bug.