Bug 2139694 - RGW cloud Transition. Found Errors during transition when using MCG Azure Namespacestore with a pre-created bucket
Summary: RGW cloud Transition. Found Errors during transition when using MCG Azure Nam...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 6.0
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
: 6.1
Assignee: Soumya Koduri
QA Contact: Tejas
Akash Raj
URL:
Whiteboard:
Depends On: 2142941 2173926
Blocks: 2141020
TreeView+ depends on / blocked
 
Reported: 2022-11-03 09:31 UTC by daniel parkes
Modified: 2023-06-15 09:16 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Cause: RGW supports object transition to only other S3 cloud endpoint as per the set lifecycle rules. Consequence: If using MCG/Azure cloud store, transition can fail if MCG cannot map the S3 objects including xattrs to Azure format. Fix: In order to allow some cloud services detect the source and map the user-defined ‘x-amz-meta-’ attributes, below two additional new attributes are added to the objects being transitioned x-rgw-cloud : true/false (set to "true", by default, if the object is being transitioned from RGW) x-rgw-cloud-keep-attrs : true/false (if set to default value "true", the cloud service should map and store all the x-amz-meta-* attributes. If it cannot, then the operation should fail. if set to "false", the cloud service can ignore such attributes and just store the object data being sent.) Result: Using these new headers, MCG can detect transitioned objects from RGW and map accordingly
Clone Of:
: 2141020 (view as bug list)
Environment:
Last Closed: 2023-06-15 09:16:13 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph pull 50098 0 None Merged rgw/cloud: Add custom headers for objects transitioned to cloud 2023-04-10 21:14:35 UTC
Red Hat Issue Tracker RHCEPH-5550 0 None None None 2022-11-03 09:48:27 UTC
Red Hat Product Errata RHSA-2023:3623 0 None None None 2023-06-15 09:16:58 UTC

Description daniel parkes 2022-11-03 09:31:42 UTC
Description of problem:

Found Errors during cloud transition when using MCG Azure Namespacestore with a pre-created bucket.

If I use pre-created buckets with AWS S3, or with an MCG AWS S3 Namespacestore, the transitions work fine, but when using an MCG Azure Namespacestore I get the following errors, and no objects are transitioned:

022-11-03T08:42:29.675+0000 7fa1bffe7640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.675+0000 7fa1bffe7640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1111 , ret = -13
2022-11-03T08:42:29.675+0000 7fa1bffe7640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to create target bucket: rgwtransition, ret:-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to create target bucket on the cloud endpoint ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to transfer object(azns/aznss1115) to the cloud endpoint(https://s3-openshift-storage.apps.ocp410.077dazopenshift.com) ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to transition obj(key:azns/aznss1115) to cloud (r=-39)
2022-11-03T08:42:29.682+0000 7fa1bf7e6640  0 lifecycle: ERROR: remove_expired_obj :mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1115 (39) Directory not empty wp_thrd: 2, 2
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 20 lifecycle: ERROR: orule.process() returned ret=-39thread:wp_thrd: 2, 2
2022-11-03T08:42:29.707+0000 7fa1c07e8640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.707+0000 7fa1c07e8640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1113 , ret = -13
2022-11-03T08:42:29.707+0000 7fa1c07e8640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1211 , ret = -13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640  0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.756+0000 7fa1bf7e6640  0 lifecycle: ERROR: fail to complete request, ret = -22



Version-Release number of selected component (if applicable):


How reproducible:

Every time


Steps to Reproduce:
1.Create in ODF a Azure namespace store
2.Create an OBC, and use the credentials of the OBC and name of the bucket in the RGW SC configuration
3.Add a LC to the bucket that uses the SC
4. Check RGW logs for failure.

Actual results:


Expected results:

For MCG Azure namespace buckets to work with RGW cloud transitions


Additional info:

Comment 1 RHEL Program Management 2022-11-03 09:31:53 UTC
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 11 Scott Ostapovicz 2022-12-14 15:02:48 UTC
Moved to 6.1 as per the Dec 7th program meeting.

Comment 33 errata-xmlrpc 2023-06-15 09:16:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3623


Note You need to log in before you can comment on or make changes to this bug.