Bug 2272084

Created bucket class with spread policy in test cluster. Provisioned obc and added some data. Modified placement policy to replica. The data got replicaed to new backingtore, however, bucketclass is stuck in configuring phase with syntax invalid error. Can't create new obc from it.

To verify syntax and spec of edited bucket class, created one more bucket class from GUI with mirror policy. It was created successfully. The spec section of both bucketclasses is same.
Not sure why noobaa-operator is throwing invalid syntax message.

---------------

TEST 1:

1. Created a backingstore

~~~
test-bs-1                      pv-pool         Ready   10m
~~~

2. Created a bucket class with spread placement policy

~~~
test-bucket-class             {"tiers":[{"backingStores":["test-bs-1"],"placement":"Spread"}]}                             Ready   0s
~~~

3. Created an obc and put some data

~~~
oc get obc|grep test-sona
default             test-sona     openshift-storage.noobaa.io   Bound     56s
~~~

~~~
 AWS_ACCESS_KEY_ID="CSOk3cOuxPlcUUFnF1Vv" AWS_SECRET_ACCESS_KEY="65KblHQok8JCKtAtd95DHjXSNSmUU0pMqGhQYRPr" aws --endpoint=https://s3-openshift-storage.apps.emea-odf414.psi.redhat.com --no-verify-ssl  s3 ls s3://test-sona-3b6bf67b-c666-4db8-bd47-55da2f0db55d
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-openshift-storage.apps.emea-odf414.psi.redhat.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2024-03-28 17:26:24        981 pvc.yaml
2024-03-28 17:25:46          5 test
~~~

4. Created one more backingstore 

~~~
test-bs-2                      pv-pool         Ready   6m24s
~~~

5. Changed placement policy to mirror in bucketclass

oc get bucketclass test-bucket-class -o yaml
~~~
spec:
  placementPolicy:
    tiers:
    - backingStores:
      - test-bs-1
      - test-bs-2
      placement: Mirror
  conditions:
  - lastHeartbeatTime: "2024-03-28T16:34:53Z"
    lastTransitionTime: "2024-03-28T16:34:04Z"
    message: 'Failed to update bucket class "test-bucket-class" with error: error:
      invalid input syntax for type json - Can''t revert changes'
    reason: TemporaryError
    status: "False"
    type: Available
~~~

In noobaa-operator logs, observed "invalid input syntax" message:

~~~
time="2024-03-28T16:34:10Z" level=info msg="✈️  RPC: tiering_policy.update_bucket_class() Request: {Name:test-bucket-class Policy:{Name:TEMP Tiers:[{Order:0 Tier:TEMP Mode:}] ChunkSplitConfig:<nil> DataCapacity:<nil> StorageCapacity:<nil> Mode:} Tiers:[{Name:TEMP DataPlacement:MIRROR AttachedPools:[test-bs-1 test-bs-2] ChunkCoderConfig:<nil> DataCapacity:<nil> StorageCapacity:<nil>}]}"
time="2024-03-28T16:34:10Z" level=error msg="⚠️  RPC: tiering_policy.update_bucket_class() Response Error: Code=INTERNAL Message=error: invalid input syntax for type json"
time="2024-03-28T16:34:10Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" bucketclass=openshift-storage/test-bucket-class
time="2024-03-28T16:34:10Z" level=warning msg="⏳ Temporary Error: Failed to update bucket class \"test-bucket-class\" with error: error: invalid input syntax for type json - Can't revert changes" bucketclass=openshift-storage/test-bucket-class
~~~

6. The data got replicated in new backingstore.

7. Created a obc. It got stuck in pending state:

~~~
oc get obc|grep test-sona-2

test-sona-2   openshift-storage.noobaa.io   Pending   12m
~~~

obc is stuck in pending state as bucketclass is not ready:
~~~
  Warning  BucketClassNotReady  2m37s (x17 over 13m)  noobaa-operator  BucketClass "test-bucket-class" is not ready
~~~

--------

TEST 2:

1. Created a new bucketclass from GUI. 

~~~
oc get bucketclass|grep replica

replica-bucket-class          {"tiers":[{"backingStores":["test-bs-3","test-bs-4"],"placement":"Mirror"}]}                             Ready         22m
~~~

2. The bucketclass is provisioned successfully. The syntax of spec section is exaclty same as that of the older bucketclass which was modified:
~~~
oc get bucketclass  replica-bucket-class -o yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
[..]
spec:
  placementPolicy:
    tiers:
    - backingStores:
      - test-bs-3
      - test-bs-4
      placement: Mirror
[..]      
  phase: Ready
~~~

Moving the bug to 4.14.8 as we had had exceeded the fixes to be taken in 4.14.7.

Please sync with Bipin, this is a customer case

Hi Danny,

Customer upgraded ODF to fixed version (4.15.2).

The bucketclass is still stuck in configuring phase.

~~~
  spec:
    placementPolicy:
      tiers:
      - backingStores:
        - noobaa-default-backing-store
        - aws-glacier
        placement: Mirror
  status:
    conditions:
    - lastHeartbeatTime: "2024-06-11T12:36:39Z"
      lastTransitionTime: "2024-06-11T12:36:38Z"
      message: 'Failed to update bucket class "noobaa-default-bucket-class" with error:
        Tier quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718109399.0 already
        exists - Can''t revert changes' <---
      reason: TemporaryError
      status: "False"
      type: Available
~~~

It is stuck at the mirroring quay bucket as tiers in quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2 already exists on both backing stores.

~~~
 32mJun-11 10:06:23.575 35m [WebServer/38]  36m   [L0] 39m core.server.system_services.system_store:: SystemStore.make_changes: { 
  insert: { tiers: [], chunk_configs: [] }, 
  update: { 
    tieringpolicies: [], 
    tiers: [ 
      { _id: 64a676278650a5002ae10a8f, name:  32m'quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718100384.0' 39m, system: 62aafde87257d700299f1346, chunk_config: 62aafde87257d700299f1349, data_placement:  32m'MIRROR' 39m, mirrors: [ { _id: 6668219f7fe5ea00265356ca, spread_pools:  36m[Array] 39m }, { _id: 6668219f7fe5ea00265356cb, spread_pools:  36m[Array] 39m } ] },
[..]
 32mJun-11 10:06:23.590 35m [WebServer/38]  36m   [L0] 39m core.server.system_services.redirector:: publish_to_cluster: server_inter_process load_system_store { since:  33m1718100383575 39m } [  32m'fcall://fcall(4dltykv)' 39m,  32m'ws://[::1]:56762/(4eifg5k)' 39m,  32m'ws://[::1]:56776/(4nlqfld)' 39m,  32m'ws://[::ffff:10.164.4.16]:42842/(5q58hxc)' 39m,  32m'ws://[::ffff:10.164.12.35]:33218/(2nwjthtra6)' 39m ]
 32mJun-11 10:06:24.222 35m [WebServer/38]  31m[ERROR] 39m CONSOLE:: RPC._on_request: ERROR srv tiering_policy_api.update_bucket_class reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-3307911 connid ws://[::ffff:10.164.4.16]:42842/(5q58hxc) Error: Tier quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718100384.0 already exists
    at check_tier_exists (/root/node_modules/noobaa-core/src/server/system_services/tier_server.js:868:75)
    at Object.update_bucket_class (/root/node_modules/noobaa-core/src/server/system_services/tier_server.js:373:17)
    at Object.server_func (/root/node_modules/noobaa-core/src/rpc/rpc.js:113:48)
    at /root/node_modules/noobaa-core/src/rpc/rpc.js:340:46
    at RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:345:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
~~~


~~~
 grep "already exists" noobaa-core-0-core.log |awk -F 'Tier' '{ print $2 }'|awk '{ print $1 }'
quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718100384.0
quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718100386.0
quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718100386.0
[..]
quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718109395.0
quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718109397.0
quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2.1718109399.0
~~~


~~~
grep "already exists" noobaa-core-0-core.log |awk -F 'Tier' '{ print $2 }'|awk '{ print $1 }'|sort|uniq|wc -l
5948
~~~

How can we proceed here? Is there any way to skip mirroring the tiers which are already mirrored?

Regards,
Sonal Arora

Hi Danny,

Did you get a chance to check this?


Regards,
Sonal Arora

Hello Danny,

Can we continue on this bug for troubleshooting the issue as bug was primarily opened for this case.


1. Created a bucket for Quay to use with the default bucket class. (non-replicated)
2. Modified the bucket class to enable mirroring the data to AWS S3. At this time, everything was working fine and the bucket class state was not Configuring. The bucket class configuration was then stored in Git so it can be monitored by OpenShift GitOps
3. After some time, the bucket class went into Configuring state and never went out again.

Backingstore
~~~
NAME                           TYPE            TARGET-BUCKET                                                    PHASE   AGE              
aws-glacier                    aws-s3          resurs-glacier-bucket-openshift-prod                             Ready   192d23h53m12s    
noobaa-default-backing-store   s3-compatible   nb.1655373326030.apps.rbp-openshift-cluster-011.ocp.resurs.com   Ready   1y361d2h41m12s    <---
~~~

Bucket class

~~~
noobaa-default-bucket-class   {"tiers":[{"placement":"Mirror","backingStores":["noobaa-default-backing-store","aws-glacier"]}]}   null               null    Configuring   1y361d2h41m12s    <---- 
replicated-glacier            {"tiers":[{"placement":"Mirror","backingStores":["aws-glacier","noobaa-default-backing-store"]}]}   null               null    Ready         4d2h22m17s       
~~~

After upgrading to ODF 4.15.2 the issue persists. Bucket class is still in configuring phase:

noobaa-default-bucket-class   {"tiers":[{"placement":"Mirror","backingStores":["noobaa-default-backing-store","aws-glacier"]}]}   null               null    Configuring   1y361d2h41m12s   

It is stuck at the mirroring quay bucket as tiers of bucket quay-datastore-62864698-3ac5-4ae1-bae1-c87271963ec2 already exists on both backing stores.

Data : ODF MG: 0050-odf-must-gather-rbp011-415.tgz

Hi Danny,

Any updates?

Please backport the fix to ODF-4.14 and update the RDT flag/text appropriately.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.14.10 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:6398