Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2139694

Summary: RGW cloud Transition. Found Errors during transition when using MCG Azure Namespacestore with a pre-created bucket
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: daniel parkes <dparkes>
Component: RGWAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Tejas <tchandra>
Severity: high Docs Contact: Akash Raj <akraj>
Priority: unspecified    
Version: 6.0CC: akraj, cbodley, ceph-eng-bugs, cephqe-warriors, kbader, kdreyer, kkeithle, mbenjamin, skoduri, sostapov
Target Milestone: ---   
Target Release: 6.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Cause: RGW supports object transition to only other S3 cloud endpoint as per the set lifecycle rules. Consequence: If using MCG/Azure cloud store, transition can fail if MCG cannot map the S3 objects including xattrs to Azure format. Fix: In order to allow some cloud services detect the source and map the user-defined ‘x-amz-meta-’ attributes, below two additional new attributes are added to the objects being transitioned x-rgw-cloud : true/false (set to "true", by default, if the object is being transitioned from RGW) x-rgw-cloud-keep-attrs : true/false (if set to default value "true", the cloud service should map and store all the x-amz-meta-* attributes. If it cannot, then the operation should fail. if set to "false", the cloud service can ignore such attributes and just store the object data being sent.) Result: Using these new headers, MCG can detect transitioned objects from RGW and map accordingly
 Story Points: ---
Clone Of:
: 2141020 (view as bug list) Environment:
Last Closed: 2023-06-15 09:16:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2142941, 2173926    
Bug Blocks: 2141020    

Description daniel parkes 2022-11-03 09:31:42 UTC 
Description of problem:

Found Errors during cloud transition when using MCG Azure Namespacestore with a pre-created bucket.

If I use pre-created buckets with AWS S3, or with an MCG AWS S3 Namespacestore, the transitions work fine, but when using an MCG Azure Namespacestore I get the following errors, and no objects are transitioned:

022-11-03T08:42:29.675+0000 7fa1bffe7640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.675+0000 7fa1bffe7640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1111 , ret = -13
2022-11-03T08:42:29.675+0000 7fa1bffe7640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to create target bucket: rgwtransition, ret:-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to create target bucket on the cloud endpoint ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to transfer object(azns/aznss1115) to the cloud endpoint(https://s3-openshift-storage.apps.ocp410.077dazopenshift.com) ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to transition obj(key:azns/aznss1115) to cloud (r=-39)
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: remove_expired_obj :mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1115 (39) Directory not empty wp_thrd: 2, 2
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 20 lifecycle: ERROR: orule.process() returned ret=-39thread:wp_thrd: 2, 2
2022-11-03T08:42:29.707+0000 7fa1c07e8640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.707+0000 7fa1c07e8640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1113 , ret = -13
2022-11-03T08:42:29.707+0000 7fa1c07e8640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1211 , ret = -13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.756+0000 7fa1bf7e6640  0 lifecycle: ERROR: fail to complete request, ret = -22



Version-Release number of selected component (if applicable):


How reproducible:

Every time


Steps to Reproduce:
1.Create in ODF a Azure namespace store
2.Create an OBC, and use the credentials of the OBC and name of the bucket in the RGW SC configuration
3.Add a LC to the bucket that uses the SC
4. Check RGW logs for failure.

Actual results:


Expected results:

For MCG Azure namespace buckets to work with RGW cloud transitions


Additional info:
Comment 1 RHEL Program Management 2022-11-03 09:31:53 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 11 Scott Ostapovicz 2022-12-14 15:02:48 UTC 
Moved to 6.1 as per the Dec 7th program meeting.
Comment 33 errata-xmlrpc 2023-06-15 09:16:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3623