Bug 2268959

Summary: After upgrade, noobaa is in initialization state due system.read_system() Response Error: Code=UNAUTHORIZED Message=account not found 65ec27f52e541700269b2156
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Vijay Avuthu <vavuthu>
Component: Multi-Cloud Object GatewayAssignee: Danny <dzaken>
Status: CLOSED ERRATA QA Contact: krishnaram Karthick <kramdoss>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.15CC: dzaken, ebenahar, kramdoss, muagarwa, nbecker, odf-bz-bot, sheggodu
Target Milestone: ---Keywords: Automation, Regression
Target Release: ODF 4.15.0Flags: kramdoss: needinfo+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-03-19 15:33:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vijay Avuthu 2024-03-11 07:06:38 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

platform: VSPHERE IPI 1AZ RHCOS VSAN 3M 3W 


Version of all relevant components (if applicable):
upgraded from 4.14.5 to 4.15.0-157


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
1/1

Can this issue reproduce from the UI?
not tried

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. upgrade odf from 4.14.5 to 4.15.0-157
2. check storagecluster satus
3.


Actual results:

Status:
  Conditions:
    Last Heartbeat Time:   2024-03-09T09:03:52Z
    Last Transition Time:  2024-03-09T09:03:52Z
    Message:               Version check successful
    Reason:                VersionMatched
    Status:                False
    Type:                  VersionMismatch
    Last Heartbeat Time:   2024-03-09T14:20:13Z
    Last Transition Time:  2024-03-09T13:38:36Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  ReconcileComplete
    Last Heartbeat Time:   2024-03-09T13:29:41Z
    Last Transition Time:  2024-03-09T09:12:33Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  Available
    Last Heartbeat Time:   2024-03-09T14:20:13Z
    Last Transition Time:  2024-03-09T13:30:28Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True
    Type:                  Progressing
    Last Heartbeat Time:   2024-03-09T13:29:41Z
    Last Transition Time:  2024-03-09T11:47:32Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                False
    Type:                  Degraded
    Last Heartbeat Time:   2024-03-09T13:39:07Z
    Last Transition Time:  2024-03-09T13:30:38Z
    Message:               CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-0-data-0w4lzl"
    Reason:                ClusterStateCreating
    Status:                False
    Type:                  Upgradeable
  Current Mon Count:       3
  Failure Domain:          rack
 
.
.
.

  Phase:  Progressing


Expected results:

storagecluster should be in Ready state


Additional info:

> operator log

time="2024-03-09T14:23:14Z" level=error msg="âš ï¸  RPC: system.read_system() Response Error: Code=UNAUTHORIZED Message=account not found 65ec27f52e541700269b2156"
time="2024-03-09T14:23:14Z" level=info msg="SetPhase: temporary error during phase \"Connecting\"" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-03-09T14:23:14Z" level=warning msg="â³ Temporary Error: account not found 65ec27f52e541700269b2156" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-03-09T14:23:14Z" level=info msg="Update event detected for noobaa-default-backing-store (openshift-storage), queuing Reconcile"

> noobaa-core logs

2024-03-09T14:19:37.401306082Z [32mMar-9 14:19:37.401[35m [WebServer/38] [31m[ERROR][39m CONSOLE:: RPC._on_request: ERROR srv system_api.read_system reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-5043 connid ws://[::ffff:10.128.4.12]:52282/(59puv4u.zzxm) Error: account not found 65ec27f52e541700269b2156
2024-03-09T14:19:37.401306082Z     at req.check_auth (/root/node_modules/noobaa-core/src/server/common_services/auth_server.js:544:23)
2024-03-09T14:19:37.401306082Z     at authorize (/root/node_modules/noobaa-core/src/server/common_services/auth_server.js:434:17)
2024-03-09T14:19:37.401306082Z     at RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:323:27)
2024-03-09T14:19:37.401306082Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)


job: https://url.corp.redhat.com/1248f21
must gather: https://url.corp.redhat.com/94a5771
Comment 14 errata-xmlrpc 2024-03-19 15:33:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:1383