Bug 2225250

Description of problem (please be detailed as possible and provide log
snippests):
AFter ODF got auto updated, noobaa-backing store was spitting out the following message:

Jul-22 1:21:05.901 [Endpoint/13] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv redirector_api.register_to_cluster reqid 257@wss://noobaa-mgmt.openshift-storage.svc:443(1idvy56) connid wss://noobaa-mgmt.openshift-storage.svc:443(1idvy56) params undefined took [0.6+1.1=1.6] [RpcError: account not found 641c895851bc0e0029263b75] { rpc_code: 'UNAUTHORIZED' }
Jul-22 1:21:05.901 [Endpoint/13] [ERROR] core.server.system_services.system_store:: SystemStore: load failed [RpcError: account not found 641c895851bc0e0029263b75] { rpc_code: 'UNAUTHORIZED' }
Jul-22 1:21:05.901 [Endpoint/13] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.add_endpoint_report reqid 156@fcall://fcall(7pkqyyn0) connid fcall://fcall(7pkqyyn0) [RpcError: account not found 641c895851bc0e0029263b75] { rpc_code: 'UNAUTHORIZED' }
Jul-22 1:21:05.902 [Endpoint/13] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.add_endpoint_report reqid 156@fcall://fcall(7pkqyyn0) connid fcall://fcall(7pkqyyn0) params { timestamp: 1689988865898, group_name: 'f54b2f7a-34e0-4a1f-a7b3-96e2346bcc09', hostname: 'noobaa-endpoint-5d7fd99889-qw4ms', cpu: { count: 1, usage: 0.004439137133078674 }, memory: { total: 2147483648, used: 181059584 }, s3_ops: { usage: { total_calls: 0 }, errors: { total_errors: 8, AccessDenied: 8 } }, bandwidth: [ [length]: 0 ] } took [2.3+0.5=2.7] [RpcError: account not found 641c895851bc0e0029263b75] { rpc_code: 'UNAUTHORIZED' }
Jul-22 1:21:05.902 [Endpoint/13] [ERROR] core.endpoint.endpoint:: Could not submit endpoint monitor report, got: [RpcError: account not found 641c895851bc0e0029263b75] { rpc_code: 'UNAUTHORIZED' }
Jul-22 1:21:06.038 [Endpoint/13]    [L0] core.server.bg_services.namespace_monitor:: namespace_monitor: system_store did not finish initial load

Version of all relevant components (if applicable):
ODF 4.11.9

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, quay is unable to be used, production is down

Is there any workaround available to the best of your knowledge?
No


Additional info:
Files are on supportshell:

$  pwd
/cases/03568874
$  ls -lt
total 0
drwxrwxrwx. 3 yank yank 37 Jul 22 01:35 0010-ocs-must-gather.tar.gz

We were thinking maybe about regenerating the s3 creds following https://www.ibm.com/docs/en/storage-fusion/2.5?topic=gateway-regenerating-s3-credentials-accounts but not sure if this would do anything? Since this is production, we need to make sure non data loss occurs with whatever procedure we follow.