+++ This bug was initially created as a clone of Bug #2266579 +++

Etag element has empty string  for complete-multipart-upload output


Description of problem:
Etag element has empty string  for complete-multipart-upload output

Version-Release number of selected component (if applicable):
ceph version 18.2.1-40.el9cp (a842b5916501030e0c0be773e91c33daf38fd65f) reef (stable)

# aws --version
aws-cli/1.32.51 Python/3.9.18 Linux/5.14.0-362.18.1.el9_3.x86_64 botocore/1.34.51

How reproducible:
3/3

Steps to Reproduce:
1. Configure awscli for the ceph cluster
2. create a bucket and upload mutlipart object to the bucket via awscli
3. perform: complete-multipart-upload operation on a bucket

Actual results:
Etag returns emtpty string for complete-multipart-upload output

Expected results:
Should return valid Etag for complete-multipart-upload output

Additional info:

Workflow:

1.Create bucket
  # date;/usr/local/bin/aws s3api create-bucket --bucket aws-bkt5 --endpoint 
    http://10.0.208.176:80
    Wed Feb 28 06:11:18 EST 2024

2. Create multipart uploads
  # date;/usr/local/bin/aws s3api create-multipart-upload --bucket aws-bkt5 --key '   aws-ob' --endpoint http://10.0.208.176:80
    Wed Feb 28 06:11:51 EST 2024
	{
	    "Bucket": "aws-bkt5",
	    "Key": "aws-ob",
	    "UploadId": "2~AQ3Ar-O3PWqGPNg_f8HY4k1ZhKQzDUR"
	}

3. Upload part: part1
  # date;/usr/local/bin/aws s3api upload-part --bucket aws-bkt5 --key 'aws-ob' -- 
    part-number 1 --upload-id  2~AQ3Ar-O3PWqGPNg_f8HY4k1ZhKQzDUR --body key20m -- 
    endpoint http://10.0.208.176:80
    Wed Feb 28 06:12:25 EST 2024
	{
	    "ETag": "\"8f4e33f3dc3e414ff94e5fb6905cba8c\""
	}


4. Upload Part: part2
   # date;/usr/local/bin/aws s3api upload-part --bucket aws-bkt5 --key 'aws-ob' -- 
     part-number 2 --upload-id  2~AQ3Ar-O3PWqGPNg_f8HY4k1ZhKQzDUR --body key22m -- 
     endpoint http://10.0.208.176:80
     Wed Feb 28 06:12:55 EST 2024
	{
	    "ETag": "\"2b4006e7fa2246353d6646ae1668141d\""
	}

5. Perform complete-multipart-upload
   # date;/usr/local/bin/aws s3api complete-multipart-upload --multipart-upload 
     file://test --bucket aws-bkt5 --key 'aws-ob' --upload-id 2~AQ3Ar- 
     O3PWqGPNg_f8HY4k1ZhKQzDUR --endpoint http://10.0.208.176:80
     Wed Feb 28 06:14:28 EST 2024
	{
	    "Location": "http://10.0.208.176/aws-bkt5/aws-ob",
	    "Bucket": "aws-bkt5",
	    "Key": "aws-ob",
	    "ETag": "" <------------ Empty etag
	}

Debug log: http://magna002.ceph.redhat.com/ceph-qe-logs/Chaithra/etag_empty/ceph-client.rgw.rgw.1.ceph-test-ss-71-o6pszh-node5.rassnf.log

--- Additional comment from Matt Benjamin (redhat) on 2024-03-19 16:48:53 UTC ---

Hi Ali,

I know you worked on this upstream.  Can you try to help debug the downstream version of this issue?  (Your upstream changes are already present.)

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-03-22 17:16:11 UTC ---

Ok, since this was escalated, I took a stab at debugging.

I haven't checked the etag computation, but boto3 isn't complaining, when I make the following change (7.1):

"""
[mbenjamin@localhost build]$ git diff -p
diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc
index 3c4c59fc70c..441564b3f0e 100644
--- a/src/rgw/rgw_op.cc
+++ b/src/rgw/rgw_op.cc
@@ -6809,10 +6809,15 @@ void RGWCompleteMultipart::execute(optional_yield y)
     ldpp_dout(this, 0) << "WARNING: failed to remove object " << meta_obj << dendl;
   }
 
+  r = s->object->get_obj_attrs(s->yield, this);
+  if (r < 0) {
+    ldpp_dout(this, 0) << __func__ << "() ERROR: get_obj_attrs() returned ret=" << r << dendl;
+  }
+
   // send request to notification manager
-  int ret = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
-  if (ret < 0) {
-    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << ret << dendl;
+  r = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
+  if (r < 0) {
+    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << r << dendl;
     // too late to rollback operation, hence op_ret is not set here
   }
 } // RGWCompleteMultipart::execute
"""

it was possible to refresh object's attrs in MultipartUpload::complete, but then we would have no value for etag when we publish_commit notifications, if any.

could I get a more expert review here?

thanks!

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-03-22 17:31:48 UTC ---

sorry, should be:

"""
diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc
index 3c4c59fc70c..f3f4ce859cb 100644
--- a/src/rgw/rgw_op.cc
+++ b/src/rgw/rgw_op.cc
@@ -6809,10 +6809,17 @@ void RGWCompleteMultipart::execute(optional_yield y)
     ldpp_dout(this, 0) << "WARNING: failed to remove object " << meta_obj << dendl;
   }
 
+  r = s->object->get_obj_attrs(s->yield, this);
+  if (r < 0) {
+    ldpp_dout(this, 0) << __func__ << "() ERROR: get_obj_attrs() returned ret=" << r << dendl;
+  }
+
+  etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
+
   // send request to notification manager
-  int ret = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
-  if (ret < 0) {
-    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << ret << dendl;
+  r = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
+  if (r < 0) {
+    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << r << dendl;
     // too late to rollback operation, hence op_ret is not set here
   }
 } // RGWCompleteMultipart::execute
@@ -6866,8 +6873,6 @@ void RGWCompleteMultipart::complete()
     }
   }
 
-  etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
-
   send_response();
 }

"""