Bug 1913771

Description of problem (please be detailed as possible and provide log
snippests):
If user constantly pushes data during upgrade into MCG bucket via S3 interface, then at one point the request to push data fails with response: HTTP/1.1 403 Forbidden.

Version of all relevant components (if applicable):
Upgrade from OCS 4.5 to OCS 4.6

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
4

Can this issue reproducible?
yes

Steps to Reproduce:
1. Create MCG bucket.
2. Start pushing data into bucket and check response codes from S3 api.
3. Upgrade OCS from 4.6 to 4.7.

Or use automated tests "test_start_upgrade_mcg_io" and "test_upgrade_mcg_io" from ocs-ci framework (https://github.com/red-hat-storage/ocs-ci/blob/master/tests/ecosystem/upgrade/test_noobaa.py)

Actual results:
For most of the upgrade the bucket is utilized correctly but at one point the response from request to upload data is 403 Forbidden. The bucket recovers after few moments and upload can continue.

Expected results:
There are 2 problems with the response:
1. The bucket should be available for the whole upgrade but this problem may be in our configuration because of insufficient number of replicas:
noobaa-core: 1 replica
noobaa-db: 1 replica
noobaa-endpoint: 2 replicas
noobaa-operator: 1 replica
I am not sure if any of those deployments have to have higher number of replicas for workload of mcg bucket to survive during upgrade.
2. When the service is unavailable then the response should not be 403 Forbidden but 503 Service Unavailable.

Additional info: