Description of problem (please be detailed as possible and provide log
snippests):
After upgrading from ocs-operator.v4.16.1-rhodf to ocs-registry:4.17.0-78, image "rhceph-7-rhel9" is not upgraded
Version of all relevant components (if applicable):
ocs-registry:4.17.0-78
Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes
Is there any workaround available to the best of your knowledge?
Not aware
Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1
Can this issue reproducible?
2/2
Can this issue reproduce from the UI?
Not tried
If this is a regression, please provide more details to justify this:
yes
Steps to Reproduce:
1. install odf ocs-operator.v4.16.1-rhodf ( GA'ed )
2. upgrade to 4.17.0-78
3. check all images are upgraded
Actual results:
2024-08-20 12:36:16 03:06:15 - MainThread - ocs_ci.ocs.resources.pod - WARNING - Images: {'registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3'} weren't upgraded in: rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-5cfd559kt8f2!
Expected results:
image should be upgraded to rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580
Additional info:
from csv, image should be rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580
$ oc get csv ocs-operator.v4.17.0-78.stable -o yaml | grep -i rhceph
value: registry.redhat.io/rhceph/rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580
- image: registry.redhat.io/rhceph/rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580
but from pod image is "rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3"
$ oc get pod rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-5cfd559kt8f2 -o yaml | grep -i rhceph
value: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
> rook-ceph-operator-7694fff5fc-665cm.log logs
2024-08-20T10:05:56.916+0000 7f92efd7e800 0 ERROR: failed to decode obj from .rgw.root:periods.126ed078-86bd-4aa4-85bf-dc94896c9f20.1
2024-08-20T10:05:56.916+0000 7f92efd7e800 0 failed to load current period: (5) Input/output error. : exit status 1
2024-08-20 10:05:57.407564 I | ceph-spec: parsing mon endpoints: a=172.30.49.190:3300,b=172.30.102.23:3300,c=172.30.164.34:3300
2024-08-20 10:05:57.407642 I | ceph-object-store-user-controller: CephObjectStore "ocs-storagecluster-cephobjectstore" found
2024-08-20 10:05:57.407800 I | ceph-object-store-user-controller: CephObjectStore "ocs-storagecluster-cephobjectstore" found
2024-08-20 10:05:57.489822 E | ceph-object-store-user-controller: failed to reconcile CephObjectStoreUser "openshift-storage/noobaa-ceph-objectstore-user". failed to initialized rgw admin ops client api: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "ocs-storagecluster-cephobjectstore": failed to create s3 user. Unable to initialize site config.2024-08-20T10:05:57.473+0000 7f1ef172f800 0 ERROR: failed to decode obj from .rgw.root:periods.126ed078-86bd-4aa4-85bf-dc94896c9f20.1
job: https://url.corp.redhat.com/e23d2b4
must gather: https://url.corp.redhat.com/0a9f93d
Description of problem (please be detailed as possible and provide log snippests): After upgrading from ocs-operator.v4.16.1-rhodf to ocs-registry:4.17.0-78, image "rhceph-7-rhel9" is not upgraded Version of all relevant components (if applicable): ocs-registry:4.17.0-78 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes Is there any workaround available to the best of your knowledge? Not aware Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? 2/2 Can this issue reproduce from the UI? Not tried If this is a regression, please provide more details to justify this: yes Steps to Reproduce: 1. install odf ocs-operator.v4.16.1-rhodf ( GA'ed ) 2. upgrade to 4.17.0-78 3. check all images are upgraded Actual results: 2024-08-20 12:36:16 03:06:15 - MainThread - ocs_ci.ocs.resources.pod - WARNING - Images: {'registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3'} weren't upgraded in: rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-5cfd559kt8f2! Expected results: image should be upgraded to rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580 Additional info: from csv, image should be rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580 $ oc get csv ocs-operator.v4.17.0-78.stable -o yaml | grep -i rhceph value: registry.redhat.io/rhceph/rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580 - image: registry.redhat.io/rhceph/rhceph-8-rhel9@sha256:6647eb400b2c02d2c41fc1a20d2d7b96b11703688f6a4529e9e1410b9250a580 but from pod image is "rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3" $ oc get pod rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-5cfd559kt8f2 -o yaml | grep -i rhceph value: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3 imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad > rook-ceph-operator-7694fff5fc-665cm.log logs 2024-08-20T10:05:56.916+0000 7f92efd7e800 0 ERROR: failed to decode obj from .rgw.root:periods.126ed078-86bd-4aa4-85bf-dc94896c9f20.1 2024-08-20T10:05:56.916+0000 7f92efd7e800 0 failed to load current period: (5) Input/output error. : exit status 1 2024-08-20 10:05:57.407564 I | ceph-spec: parsing mon endpoints: a=172.30.49.190:3300,b=172.30.102.23:3300,c=172.30.164.34:3300 2024-08-20 10:05:57.407642 I | ceph-object-store-user-controller: CephObjectStore "ocs-storagecluster-cephobjectstore" found 2024-08-20 10:05:57.407800 I | ceph-object-store-user-controller: CephObjectStore "ocs-storagecluster-cephobjectstore" found 2024-08-20 10:05:57.489822 E | ceph-object-store-user-controller: failed to reconcile CephObjectStoreUser "openshift-storage/noobaa-ceph-objectstore-user". failed to initialized rgw admin ops client api: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "ocs-storagecluster-cephobjectstore": failed to create s3 user. Unable to initialize site config.2024-08-20T10:05:57.473+0000 7f1ef172f800 0 ERROR: failed to decode obj from .rgw.root:periods.126ed078-86bd-4aa4-85bf-dc94896c9f20.1 job: https://url.corp.redhat.com/e23d2b4 must gather: https://url.corp.redhat.com/0a9f93d