Bug 1981028

Summary: NooBaa Simple MCG Deployment stuck in configuring state
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Muhammad Usman <m.usmanayub>
Component: Multi-Cloud Object GatewayAssignee: Romy Ayalon <rayalon>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Raz Tamir <ratamir>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.7CC: etamir, muagarwa, nbecker, ocs-bugs, odf-bz-bot
Target Milestone: ---Flags: m.usmanayub: needinfo-
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-01 07:45:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Muhammad Usman 2021-07-10 16:32:14 UTC 
Description of problem:


Hi Folks,

I am trying to setup a simple MCG for Red Hat Quay registry using the following guide:

https://access.redhat.com/documentation/en-us/red_hat_quay/3.5/html/deploy_red_hat_quay_on_openshift_with_the_quay_operator/installing_the_quay_operator

OpenShift Container Storage Operator has been deployed successfully. 

After the creation of noobaa instance and while trying to find the status using the command, I see that it is stuck on Configuring phase

oc get -n openshift-storage noobaas noobaa -w
NAME     MGMT-ENDPOINTS   S3-ENDPOINTS   IMAGE                                                                                                            PHASE      AGE
noobaa                                   registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4   Creating   5s
noobaa                                   registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4   Connecting   7s
noobaa   ["https://10.150.8.166:31268"]                  registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4   Connecting   11s
noobaa   ["https://10.150.8.166:31268"]                  registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4   Connecting   14s
noobaa   ["https://10.150.8.166:31268"]                  registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4   Configuring   22s


Upon checking the logs on the noobaa-core pod I find that there is an Error Credentials not found

oc logs noobaa-core-0 | tail -100
Jul-8 5:18:38.624 [WebServer/36]    [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process update_master_change { is_master: true } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ]
Jul-8 5:18:39.395 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:39.424 [BGWorkers/32]    [L0] core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN
Jul-8 5:18:39.503 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:39.504 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1646 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:18:42.919 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:43.019 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:43.019 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1649 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:18:46.200 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:46.299 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:46.300 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1652 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:18:47.324 [BGWorkers/32]    [L0] core.server.bg_services.mirror_writer:: no buckets with mirror policy. nothing to do
Jul-8 5:18:47.325 [BGWorkers/32]    [L0] core.server.bg_services.buckets_reclaimer:: no buckets in "deleting" state. nothing to do
Jul-8 5:18:47.326 [BGWorkers/32]   [LOG] CONSOLE:: TieringTTFWorker: start running
Jul-8 5:18:47.326 [BGWorkers/32]    [L0] core.server.bg_services.tier_ttf_worker:: no buckets with more than one tier. nothing to do
Jul-8 5:18:47.381 [BGWorkers/32]   [LOG] CONSOLE:: TieringSpillbackWorker: start running
Jul-8 5:18:48.628 [BGWorkers/32]    [L0] core.server.bg_services.cluster_master:: no local cluster info or server is not part of a cluster. therefore will be cluster master
Jul-8 5:18:48.632 [WebServer/36]    [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process update_master_change { is_master: true } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ]
Jul-8 5:18:49.132 [BGWorkers/32]    [L0] core.server.bg_services.scrubber:: SCRUBBER: BEGIN
Jul-8 5:18:49.136 [BGWorkers/32]    [L0] core.server.bg_services.scrubber:: SCRUBBER: END
Jul-8 5:18:49.439 [BGWorkers/32]    [L0] core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN
Jul-8 5:18:49.453 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:49.552 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:49.553 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1655 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:18:50.502 [BGWorkers/32]    [L0] core.server.bg_services.objects_reclaimer:: no objects in "unreclaimed" state. nothing to do
Jul-8 5:18:52.721 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:52.819 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:52.819 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1658 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:18:56.203 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:56.247 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: MD aggregator start running
Jul-8 5:18:56.248 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: find_next_range: from_time 1625721412532 till_time* 33715 target_now* 33715 global_last_update 1625721412532
Jul-8 5:18:56.248 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: find_next_range:: next work from_time 1625721412532 till_time* 30000 target_now* 33715
Jul-8 5:18:56.302 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:56.302 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1661 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:18:57.256 [BGWorkers/32]    [L0] core.server.bg_services.cluster_hb:: no local cluster info. HB is not written
Jul-8 5:18:57.381 [BGWorkers/32]   [LOG] CONSOLE:: TieringSpillbackWorker: start running
Jul-8 5:18:58.635 [BGWorkers/32]    [L0] core.server.bg_services.cluster_master:: no local cluster info or server is not part of a cluster. therefore will be cluster master
Jul-8 5:18:58.638 [WebServer/36]    [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process update_master_change { is_master: true } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ]
Jul-8 5:18:59.444 [BGWorkers/32]    [L0] core.server.bg_services.agent_blocks_reclaimer:: AGENT_BLOCKS_RECLAIMER: BEGIN
Jul-8 5:18:59.453 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:59.551 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:18:59.552 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1664 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:19:01.698 [BGWorkers/32]    [L0] core.server.system_services.system_store:: SystemStore.make_changes: {
  update: {
    buckets: [ { _id: 60e52c5863d9a8002a621eec, storage_stats: { chunks_capacity: 0, objects_size: 0, objects_count: 0, stats_by_content_type: [], blocks_size: 0, pools: {}, objects_hist: [], last_update: 1625721442532 }, dont_change_last_update: true } ],
    pools: [ { _id: 60e52c5863d9a8002a621ee6, storage_stats: { blocks_size: 0, last_update: 1625721442532 }, dont_change_last_update: true } ]
  }
}
Jul-8 5:19:01.706 [BGWorkers/32]    [L0] core.server.system_services.system_store:: SystemStore.make_changes: {
  update: { systems: [ { _id: 60e52c5863d9a8002a621ee4, global_last_update: 1625721442532 } ] }
}
Jul-8 5:19:01.713 [WebServer/36]    [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process load_system_store { since: 1625721541706 } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ]
Jul-8 5:19:02.718 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:19:02.816 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:19:02.816 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1667 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)
Jul-8 5:19:03.273 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: find_next_range: from_time 1625721442532 till_time* 3715 target_now* 3715 global_last_update 1625721412532
Jul-8 5:19:03.273 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: find_next_range:: next work from_time 1625721442532 till_time* 3715 target_now* 3715
Jul-8 5:19:03.284 [BGWorkers/32]    [L0] core.server.system_services.system_store:: SystemStore.make_changes: {
  update: {
    buckets: [ { _id: 60e52c5863d9a8002a621eec, storage_stats: { chunks_capacity: 0, objects_size: 0, objects_count: 0, stats_by_content_type: [], blocks_size: 0, pools: {}, objects_hist: [], last_update: 1625721446247 }, dont_change_last_update: true } ],
    pools: [ { _id: 60e52c5863d9a8002a621ee6, storage_stats: { blocks_size: 0, last_update: 1625721446247 }, dont_change_last_update: true } ]
  }
}
Jul-8 5:19:03.293 [BGWorkers/32]    [L0] core.server.system_services.system_store:: SystemStore.make_changes: {
  update: { systems: [ { _id: 60e52c5863d9a8002a621ee4, global_last_update: 1625721446247 } ] }
}
Jul-8 5:19:03.299 [WebServer/36]    [L0] core.server.system_services.redirector:: publish_to_cluster: server_inter_process load_system_store { since: 1625721543293 } [ 'fcall://fcall(2lq5x8u)', 'ws://[::ffff:127.0.0.1]:44874/(4dep0fq)', 'ws://[::ffff:127.0.0.1]:44962/(4ouj6fn)' ]
Jul-8 5:19:04.325 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: find_next_range: from_time 1625721446247 till_time* 0 target_now* 0 global_last_update 1625721412532
Jul-8 5:19:04.325 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: find_next_range:: no more work from_time 1625721446247 till_time* 0 target_now* 0
Jul-8 5:19:04.325 [BGWorkers/32]    [L0] core.server.bg_services.md_aggregator:: MD aggregator done
Jul-8 5:19:05.982 [WebServer/36]    [L0] core.server.common_services.auth_server:: credentials account not found SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:19:06.083 [WebServer/36]    [L0] core.server.common_services.auth_server:: password mismatch SENSITIVE-9cf0fd89409efef8 noobaa
Jul-8 5:19:06.083 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv auth_api.create_auth reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-1670 connid ws://[::ffff:10.130.0.148]:39970/(2lkyyis) Error: credentials not found
    at /root/node_modules/noobaa-core/src/server/common_services/auth_server.js:71:39
    at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32
    at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25)

On the console status show cannot create an auth token for admin, error: credentials not found
The status on the yaml shows:

status:
  accounts:
    admin:
      secretRef:
        name: noobaa-admin
        namespace: openshift-storage
  actualImage: >-
    registry.redhat.io/ocs4/mcg-core-rhel8@sha256:6ff8645efdde95fa97d496084d3555b7680895f0b79c147f2a880b43742af3a4
  conditions:
    - lastHeartbeatTime: '2021-07-08T04:47:41Z'
      lastTransitionTime: '2021-07-08T04:47:41Z'
      message: 'cannot create an auth token for admin, error: credentials not found'
      reason: TemporaryError
      status: 'False'
      type: Available
    - lastHeartbeatTime: '2021-07-08T04:47:41Z'
      lastTransitionTime: '2021-07-08T04:47:41Z'
      message: 'cannot create an auth token for admin, error: credentials not found'
      reason: TemporaryError
      status: 'True'
      type: Progressing
    - lastHeartbeatTime: '2021-07-08T04:47:41Z'
      lastTransitionTime: '2021-07-08T04:47:41Z'
      message: 'cannot create an auth token for admin, error: credentials not found'
      reason: TemporaryError
      status: 'False'
      type: Degraded
    - lastHeartbeatTime: '2021-07-08T04:47:41Z'
      lastTransitionTime: '2021-07-08T04:47:41Z'
      message: 'cannot create an auth token for admin, error: credentials not found'
      reason: TemporaryError
      status: 'False'
      type: Upgradeable
  observedGeneration: 2
  phase: Configuring
  readme: "\n\n\tNooBaa operator is still working to reconcile this system.\n\tCheck out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl -n openshift-storage describe noobaa\n\t\tkubectl -n openshift-storage get noobaa -o yaml\n\t\tkubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp\n\n\tYou can wait for a specific condition with:\n\n\t\tkubectl -n openshift-storage wait noobaa/noobaa --for condition=available --timeout -1s\n\n\tNooBaa Core Version:     5.7.0\n\tNooBaa Operator Version: 5.7.0\n"

I can see the noobaa-admin secret created in the namespace/project.

Need help in finding out how can this problem be fixed.

Version-Release number of selected component (if applicable):
OpenShift Container Storage 4.7

How reproducible:
Deploy OpenShift Cluster and Install OpenShift Container Storage Operator.

Steps to Reproduce:
1. Install OpenShift Container Storage Operator from the Operator Hub
2. Deploy the NooBaa Instance using the yaml file specified in the link https://access.redhat.com/documentation/en-us/red_hat_quay/3.5/html/deploy_red_hat_quay_on_openshift_with_the_quay_operator/installing_the_quay_operator
3. Upon checking the status NooBaa instance will show error.

Actual results:
NooBaa Instance stuck on configuring phase with error 'cannot create an auth token for admin, error: credentials not found'

Expected results:
Successful NooBaa Deployment
Comment 3 Muhammad Usman 2021-07-12 03:19:01 UTC 
@ray Please let me know the what additional info is required and I'll be happy to share. Thanks