Description of problem: Hi Folks, I am trying to setup a simple MCG for Red Hat Quay registry using the following guide: https://access.redhat.com/documentation/en-us/red_hat_quay/3.5/html/deploy_red_hat_quay_on_openshift_with_the_quay_operator/installing_the_quay_operator OpenShift Container Storage Operator has been deployed successfully. After the creation of noobaa instance and while trying to find the status using the command, I see that it is stuck on Configuring phase oc get -n openshift-storage noobaas noobaa -w NAME MGMT-ENDPOINTS S3-ENDPOINTS IMAGE PHASE AGE noobaa registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4 Creating 5s noobaa registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4 Connecting 7s noobaa ["https://10.150.8.166:31268"] registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4 Connecting 11s noobaa ["https://10.150.8.166:31268"] registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4 Connecting 14s noobaa ["https://10.150.8.166:31268"] registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4 Configuring 22s Upon checking the logs on the noobaa-core pod I find that there is an Error Credentials not found oc logs noobaa-core-0 | tail -100 Jul-8 5:18:38.624 [WebServer/36] [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process update_master_change { is_master: true } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ] Jul-8 5:18:39.395 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:39.424 [BGWorkers/32] [L0] core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN Jul-8 5:18:39.503 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:39.504 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1646 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:18:42.919 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:43.019 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:43.019 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1649 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:18:46.200 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:46.299 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:46.300 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1652 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:18:47.324 [BGWorkers/32] [L0] core.server.bg_services.mirror_writer:: no buckets with mirror policy. nothing to do Jul-8 5:18:47.325 [BGWorkers/32] [L0] core.server.bg_services.buckets_reclaimer:: no buckets in "deleting" state. nothing to do Jul-8 5:18:47.326 [BGWorkers/32] [LOG] CONSOLE:: TieringTTFWorker: start running Jul-8 5:18:47.326 [BGWorkers/32] [L0] core.server.bg_services.tier_ttf_worker:: no buckets with more than one tier. nothing to do Jul-8 5:18:47.381 [BGWorkers/32] [LOG] CONSOLE:: TieringSpillbackWorker: start running Jul-8 5:18:48.628 [BGWorkers/32] [L0] core.server.bg_services.cluster_master:: no local cluster info or server is not part of a cluster. therefore will be cluster master Jul-8 5:18:48.632 [WebServer/36] [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process update_master_change { is_master: true } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ] Jul-8 5:18:49.132 [BGWorkers/32] [L0] core.server.bg_services.scrubber:: SCRUBBER: BEGIN Jul-8 5:18:49.136 [BGWorkers/32] [L0] core.server.bg_services.scrubber:: SCRUBBER: END Jul-8 5:18:49.439 [BGWorkers/32] [L0] core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN Jul-8 5:18:49.453 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:49.552 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:49.553 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1655 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:18:50.502 [BGWorkers/32] [L0] core.server.bg_services.objects_reclaimer:: no objects in "unreclaimed" state. nothing to do Jul-8 5:18:52.721 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:52.819 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:52.819 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1658 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:18:56.203 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:56.247 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: MD aggregator start running Jul-8 5:18:56.248 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: find_next_range: from_time 1625721412532 till_time* 33715 target_now* 33715 global_last_update 1625721412532 Jul-8 5:18:56.248 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: find_next_range:: next work from_time 1625721412532 till_time* 30000 target_now* 33715 Jul-8 5:18:56.302 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:56.302 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1661 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:18:57.256 [BGWorkers/32] [L0] core.server.bg_services.cluster_hb:: no local cluster info. HB is not written Jul-8 5:18:57.381 [BGWorkers/32] [LOG] CONSOLE:: TieringSpillbackWorker: start running Jul-8 5:18:58.635 [BGWorkers/32] [L0] core.server.bg_services.cluster_master:: no local cluster info or server is not part of a cluster. therefore will be cluster master Jul-8 5:18:58.638 [WebServer/36] [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process update_master_change { is_master: true } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ] Jul-8 5:18:59.444 [BGWorkers/32] [L0] core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN Jul-8 5:18:59.453 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:59.551 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:18:59.552 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1664 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:19:01.698 [BGWorkers/32] [L0] core.server.system_services.system_store:: SystemStore.make_changes: { update: { buckets: [ { _id: 60e52c5863d9a8002a621eec, storage_stats: { chunks_capacity: 0, objects_size: 0, objects_count: 0, stats_by_content_type: [], blocks_size: 0, pools: {}, objects_hist: [], last_update: 1625721442532 }, dont_change_last_update: true } ], pools: [ { _id: 60e52c5863d9a8002a621ee6, storage_stats: { blocks_size: 0, last_update: 1625721442532 }, dont_change_last_update: true } ] } } Jul-8 5:19:01.706 [BGWorkers/32] [L0] core.server.system_services.system_store:: SystemStore.make_changes: { update: { systems: [ { _id: 60e52c5863d9a8002a621ee4, global_last_update: 1625721442532 } ] } } Jul-8 5:19:01.713 [WebServer/36] [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process load_system_store { since: 1625721541706 } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ] Jul-8 5:19:02.718 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:19:02.816 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:19:02.816 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1667 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Jul-8 5:19:03.273 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: find_next_range: from_time 1625721442532 till_time* 3715 target_now* 3715 global_last_update 1625721412532 Jul-8 5:19:03.273 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: find_next_range:: next work from_time 1625721442532 till_time* 3715 target_now* 3715 Jul-8 5:19:03.284 [BGWorkers/32] [L0] core.server.system_services.system_store:: SystemStore.make_changes: { update: { buckets: [ { _id: 60e52c5863d9a8002a621eec, storage_stats: { chunks_capacity: 0, objects_size: 0, objects_count: 0, stats_by_content_type: [], blocks_size: 0, pools: {}, objects_hist: [], last_update: 1625721446247 }, dont_change_last_update: true } ], pools: [ { _id: 60e52c5863d9a8002a621ee6, storage_stats: { blocks_size: 0, last_update: 1625721446247 }, dont_change_last_update: true } ] } } Jul-8 5:19:03.293 [BGWorkers/32] [L0] core.server.system_services.system_store:: SystemStore.make_changes: { update: { systems: [ { _id: 60e52c5863d9a8002a621ee4, global_last_update: 1625721446247 } ] } } Jul-8 5:19:03.299 [WebServer/36] [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process load_system_store { since: 1625721543293 } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ] Jul-8 5:19:04.325 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: find_next_range: from_time 1625721446247 till_time* 0 target_now* 0 global_last_update 1625721412532 Jul-8 5:19:04.325 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: find_next_range:: no more work from_time 1625721446247 till_time* 0 target_now* 0 Jul-8 5:19:04.325 [BGWorkers/32] [L0] core.server.bg_services.md_aggregator:: MD aggregator done Jul-8 5:19:05.982 [WebServer/36] [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:19:06.083 [WebServer/36] [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa Jul-8 5:19:06.083 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1670 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39 at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) On the console status show cannot create an auth token for admin, error: credentials not found The status on the yaml shows: status: accounts: admin: secretRef: name: noobaa-admin namespace: openshift-storage actualImage: >- registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4 conditions: - lastHeartbeatTime: '2021-07-08T04:47:41Z' lastTransitionTime: '2021-07-08T04:47:41Z' message: 'cannot create an auth token for admin, error: credentials not found' reason: TemporaryError status: 'False' type: Available - lastHeartbeatTime: '2021-07-08T04:47:41Z' lastTransitionTime: '2021-07-08T04:47:41Z' message: 'cannot create an auth token for admin, error: credentials not found' reason: TemporaryError status: 'True' type: Progressing - lastHeartbeatTime: '2021-07-08T04:47:41Z' lastTransitionTime: '2021-07-08T04:47:41Z' message: 'cannot create an auth token for admin, error: credentials not found' reason: TemporaryError status: 'False' type: Degraded - lastHeartbeatTime: '2021-07-08T04:47:41Z' lastTransitionTime: '2021-07-08T04:47:41Z' message: 'cannot create an auth token for admin, error: credentials not found' reason: TemporaryError status: 'False' type: Upgradeable observedGeneration: 2 phase: Configuring readme: "\n\n\tNooBaa operator is still working to reconcile this system.\n\tCheck out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl -n openshift-storage describe noobaa\n\t\tkubectl -n openshift-storage get noobaa -o yaml\n\t\tkubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp\n\n\tYou can wait for a specific condition with:\n\n\t\tkubectl -n openshift-storage wait noobaa/noobaa --for condition=available --timeout -1s\n\n\tNooBaa Core Version: 5.7.0\n\tNooBaa Operator Version: 5.7.0\n" I can see the noobaa-admin secret created in the namespace/project. Need help in finding out how can this problem be fixed. Version-Release number of selected component (if applicable): OpenShift Container Storage 4.7 How reproducible: Deploy OpenShift Cluster and Install OpenShift Container Storage Operator. Steps to Reproduce: 1. Install OpenShift Container Storage Operator from the Operator Hub 2. Deploy the NooBaa Instance using the yaml file specified in the link https://access.redhat.com/documentation/en-us/red_hat_quay/3.5/html/deploy_red_hat_quay_on_openshift_with_the_quay_operator/installing_the_quay_operator 3. Upon checking the status NooBaa instance will show error. Actual results: NooBaa Instance stuck on configuring phase with error 'cannot create an auth token for admin, error: credentials not found' Expected results: Successful NooBaa Deployment
@ray Please let me know the what additional info is required and I'll be happy to share. Thanks