Bug 2039723
| Summary: | [IBM Z] patching noobaa minCount have no effect | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Abdul Kandathil (IBM) <akandath> | ||||
| Component: | Multi-Cloud Object Gateway | Assignee: | Nimrod Becker <nbecker> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Elad <ebenahar> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.10 | CC: | etamir, ocs-bugs, odf-bz-bot | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | s390x | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-01-17 14:45:18 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Thanks Elad for noticing this,
Why is the noobaa CR being patched with the minCount? It is being reconciled by the odf-operator, and in fact the storagecluster should be updated.
From the QE run (again, thanks Elad!)
[2022-01-10T19:52:48.613Z] 19:52:48 - MainThread - tests.conftest - ESC[32mINFOESC[0m - Changing minimum Noobaa endpoints to 2
[2022-01-10T19:52:48.613Z] 19:52:48 - MainThread - ocs_ci.ocs.ocp - ESC[32mINFOESC[0m - Command: patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"minCount":2}}}}' --type merge
[2022-01-10T19:52:48.613Z] 19:52:48 - MainThread - ocs_ci.utility.utils - ESC[32mINFOESC[0m - Executing command: oc -n openshift-storage patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"minCount":2}}}}' --type merge
[2022-01-10T19:52:48.868Z] 19:52:48 - MainThread - tests.conftest - ESC[32mINFOESC[0m - Changing maximum Noobaa endpoints to 2
[2022-01-10T19:52:48.868Z] 19:52:48 - MainThread - ocs_ci.ocs.ocp - ESC[32mINFOESC[0m - Command: patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"maxCount":2}}}}' --type merge
[2022-01-10T19:52:48.868Z] 19:52:48 - MainThread - ocs_ci.utility.utils - ESC[32mINFOESC[0m - Executing command: oc -n openshift-storage patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"maxCount":2}}}}' --type merge
[2022-01-10T19:52:48.868Z] 19:52:48 - MainThread - ocs_ci.utility.utils - ESC[32mINFOESC[0m - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml
[2022-01-10T19:52:49.123Z] 19:52:48 - MainThread - tests.conftest - ESC[32mINFOESC[0m - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1
[2022-01-10T19:52:49.123Z] 19:52:48 - MainThread - ocs_ci.utility.utils - ESC[32mINFOESC[0m - Going to sleep for 30 seconds before next iteration
[2022-01-10T19:53:21.130Z] 19:53:18 - MainThread - ocs_ci.utility.utils - ESC[32mINFOESC[0m - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml
[2022-01-10T19:53:21.130Z] 19:53:19 - MainThread - tests.conftest - ESC[32mINFOESC[0m - NooBaa endpoints stabilized. Ready endpoints: 2
Please notice in the snippet in the BZ description
oc patch noobaa noobaa ...
While in the QE runs
oc -n openshift-storage patch storagecluster ...
Can you please verify that it works as expected with patching the storagecluster?
Seems like there was also an update to the CI in one of the conditions testing if the version tested is higher or lower than 4.6 With the fix, the detection should be correct, can you please rebase the tests code? Pulled latest changes from stable branch. Issue looks to be fixed now. Thank you.
14:44:21 - MainThread - tests.conftest - INFO - Changing minimum Noobaa endpoints to 2
14:44:21 - MainThread - ocs_ci.ocs.ocp - INFO - Command: patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"minCount":2}}}}' --type merge
14:44:21 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"minCount":2}}}}' --type merge
14:44:22 - MainThread - tests.conftest - INFO - Changing maximum Noobaa endpoints to 2
14:44:22 - MainThread - ocs_ci.ocs.ocp - INFO - Command: patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"maxCount":2}}}}' --type merge
14:44:22 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec":{"multiCloudGateway":{"endpoints":{"maxCount":2}}}}' --type merge
14:44:22 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml
14:44:22 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1
14:44:22 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration
14:44:52 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml
14:44:52 - MainThread - tests.conftest - INFO - NooBaa endpoints stabilized. Ready endpoints: 2
|
Created attachment 1850335 [details] noobaa-error must-gather Description of problem (please be detailed as possible and provide log snippets): Patching noobaa to increase minimum replica has no effect. All mcg related tests in ocs-ci tier1 are erroring due to this error. Sample test: "tests/manage/mcg/test_bucket_creation.py::TestBucketCreation::test_bucket_creation[3-S3-DEFAULT-BACKINGSTORE]" ocs-ci Logs: 09:22:21 - MainThread - tests.conftest - INFO - Changing minimum Noobaa endpoints to 2 09:22:21 - MainThread - ocs_ci.ocs.ocp - INFO - Command: oc patch noobaa noobaa -n openshift-storage -p '{"spec":{"endpoints":{"minCount":2}}}' --type merge 09:22:21 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage patch noobaa noobaa -n openshift-storage -p '{"spec":{"endpoints":{"minCount":2}}}' --type merge 09:22:21 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:22:21 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:22:21 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:22:51 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:22:51 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:22:51 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:23:21 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:23:21 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:23:21 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:23:51 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:23:52 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:23:52 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:24:22 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:24:22 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:24:22 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:24:52 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:24:52 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:24:52 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:25:22 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:25:22 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:25:22 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:25:52 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:25:52 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:25:52 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:26:22 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:26:23 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:26:23 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:26:53 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get Pod -n openshift-storage --selector=noobaa-s3=noobaa -o yaml 09:26:53 - MainThread - tests.conftest - INFO - Waiting for the NooBaa endpoints to stabilize. Current ready count: 1 09:26:53 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 30 seconds before next iteration 09:27:23 - MainThread - ocs_ci.ocs.utils - INFO - Must gather image: quay.io/rhceph-dev/ocs-must-gather:latest-4.10 will be used. 09:27:23 - MainThread - ocs_ci.ocs.utils - INFO - OCS logs will be placed in location /opt/ocs-ci-logs/failed_testcase_ocs_logs_1641979308/test_bucket_creation[3-S3-DEFAULT-BACKINGSTORE]_ocs_logs/ocs_must_gather 09:27:23 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.10 --dest-dir=/opt/ocs-ci-logs/failed_testcase_ocs_logs_1641979308/test_bucket_creation[3-S3-DEFAULT-BACKINGSTORE]_ocs_logs/ocs_must_gather Version of all relevant components (if applicable): OCP : 4.10.0-0.nightly-s390x-2022-01-10-013938 ODF : 4.10.0-79 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Patching noobaa doesn't work. Is there any workaround available to the best of your knowledge? NO Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? yes Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Deploy ODF 4.10 with LSO 4.10 on OCP 4.10 2. execute command "oc -n openshift-storage patch noobaa noobaa -n openshift-storage -p '{"spec":{"endpoints":{"minCount":2}}}' --type merge" 3. check number of noobaa endpoint pods. Actual results: only one noobaa endpoint pod running Expected results: Two noobaa endpoint pods running Additional info: