Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use Jira Cloud for all bug tracking management.

Bug 2139694

Summary: RGW cloud Transition. Found Errors during transition when using MCG Azure Namespacestore with a pre-created bucket
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: daniel parkes <dparkes>
Component: RGWAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Tejas <tchandra>
Severity: high Docs Contact: Akash Raj <akraj>
Priority: unspecified    
Version: 6.0CC: akraj, cbodley, ceph-eng-bugs, cephqe-warriors, kbader, kdreyer, kkeithle, mbenjamin, skoduri, sostapov
Target Milestone: ---   
Target Release: 6.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Cause: RGW supports object transition to only other S3 cloud endpoint as per the set lifecycle rules. Consequence: If using MCG/Azure cloud store, transition can fail if MCG cannot map the S3 objects including xattrs to Azure format. Fix: In order to allow some cloud services detect the source and map the user-defined ‘x-amz-meta-’ attributes, below two additional new attributes are added to the objects being transitioned x-rgw-cloud : true/false (set to "true", by default, if the object is being transitioned from RGW) x-rgw-cloud-keep-attrs : true/false (if set to default value "true", the cloud service should map and store all the x-amz-meta-* attributes. If it cannot, then the operation should fail. if set to "false", the cloud service can ignore such attributes and just store the object data being sent.) Result: Using these new headers, MCG can detect transitioned objects from RGW and map accordingly
 Story Points: ---
Clone Of:
: 2141020 (view as bug list) Environment:
Last Closed: 2023-06-15 09:16:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2142941, 2173926    
Bug Blocks: 2141020    

Description daniel parkes 2022-11-03 09:31:42 UTC 
Description of problem:

Found Errors during cloud transition when using MCG Azure Namespacestore with a pre-created bucket.

If I use pre-created buckets with AWS S3, or with an MCG AWS S3 Namespacestore, the transitions work fine, but when using an MCG Azure Namespacestore I get the following errors, and no objects are transitioned:

022-11-03T08:42:29.675+0000 7fa1bffe7640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.675+0000 7fa1bffe7640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1111 , ret = -13
2022-11-03T08:42:29.675+0000 7fa1bffe7640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to create target bucket: rgwtransition, ret:-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to create target bucket on the cloud endpoint ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to transfer object(azns/aznss1115) to the cloud endpoint(https://s3-openshift-storage.apps.ocp410.077dazopenshift.com) ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to transition obj(key:azns/aznss1115) to cloud (r=-39)
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: remove_expired_obj :mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1115 (39) Directory not empty wp_thrd: 2, 2
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 20 lifecycle: ERROR: orule.process() returned ret=-39thread:wp_thrd: 2, 2
2022-11-03T08:42:29.707+0000 7fa1c07e8640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.707+0000 7fa1c07e8640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1113 , ret = -13
2022-11-03T08:42:29.707+0000 7fa1c07e8640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1211 , ret = -13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.756+0000 7fa1bf7e6640  0 lifecycle: ERROR: fail to complete request, ret = -22



Version-Release number of selected component (if applicable):


How reproducible:

Every time


Steps to Reproduce:
1.Create in ODF a Azure namespace store
2.Create an OBC, and use the credentials of the OBC and name of the bucket in the RGW SC configuration
3.Add a LC to the bucket that uses the SC
4. Check RGW logs for failure.

Actual results:


Expected results:

For MCG Azure namespace buckets to work with RGW cloud transitions


Additional info:
Comment 1 RHEL Program Management 2022-11-03 09:31:53 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 11 Scott Ostapovicz 2022-12-14 15:02:48 UTC 
Moved to 6.1 as per the Dec 7th program meeting.
Comment 33 errata-xmlrpc 2023-06-15 09:16:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3623