Bug 2119660

Created attachment 1906480 [details]
Test case output log

Description of problem (please be detailed as possible and provide log
snippests):

ocs-ci testcase - test_scale_obc_creation_noobaa_pod_respin[noobaa-core-openshift-storage.noobaa.io] fails as all OBC's do not move to bound state after  node reset. We see the following error :

```
       # Reset node which noobaa pods is running on
        # And validate noobaa pods are re-spinned and in running state
        scale_noobaa_lib.noobaa_running_node_restart(pod_name=pod_name)



       # Verify all OBCs are in Bound state after node restart
        log.info("Verify all OBCs are in Bound state after node restart.....")
        obc_status_list = scale_noobaa_lib.check_all_obcs_status(
            namespace=self.namespace
        )
        log.info(
            f"Number of OBCs in Bound state after node reset: "
            f"{len(obc_status_list[0])}"
        )
>       assert (
            len(obc_status_list[0]) == self.scale_obc_count
        ), "Not all OBCs in Bound state"
E       AssertionError: Not all OBCs in Bound state
E       assert 675 == 1000
E         +675
E         -1000



tests/e2e/scale/noobaa/test_scale_obc_creation_repsin_noobaa_pods.py:124: AssertionError
```


Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? No


Is there any workaround available to the best of your knowledge? We tried to increase the timeout of the test case and re-run but it still fails.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible? 


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deploy OCP 4.11 and ODF 4.11 
2. Try to execute ocs-ci test case test_scale_obc_creation_noobaa_pod_respin[noobaa-core-openshift-storage.noobaa.io]



Actual results: Only 675 OBCs move to Bound state after node reset


Expected results: All 1000 OBC's should move to Bound state after node reset


Additional info: