Description of problem:

RGW server crashed when setting replication rules using S3 put-bucket-replication API (https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-replication.html)


ceph version 18.0.0-3644-g0f690117439 (0f690117439384827cdfffaf662b71a99f73b34a) reef (dev)
1: /lib64/libc.so.6(+0x3ea70) [0x7f3b9003ea70]
2: /lib64/libc.so.6(+0x1578ee) [0x7f3b901578ee]
3: /root/workspace/ceph_dbstore/build/bin/radosgw(+0x864d83) [0x55bf8994bd83]
4: (rgw_bucket::operator<(rgw_bucket const&) const+0x33) [0x55bf89956253]
5: /root/workspace/ceph_dbstore/build/bin/radosgw(+0xc6cde2) [0x55bf89d53de2]
6: (RGWSI_Bucket_Sync_SObj::handle_bi_update(DoutPrefixProvider const*, RGWBucketInfo&, RGWBucketInfo*, optional_yield)+0x63c) [0x55bf89d55e9c]
7: (RGWSI_Bucket_SObj::store_bucket_instance_info(ptr_wrapper<RGWSI_MetaBackend::Context, 4>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWBucketInfo&, std::optional<RGWBucketInfo*>, bool, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >, optional_yield, DoutPrefixProvider const)+0x21c) [0x55bf89d4f05c]
8: (RGWBucketCtl::do_store_bucket_instance_info(ptr_wrapper<RGWSI_MetaBackend::Context, 4>&, rgw_bucket const&, RGWBucketInfo&, optional_yield, DoutPrefixProvider const*, RGWBucketCtl::BucketInstance::PutParams const&)+0xfd) [0x55bf89e3f66d]
9: /root/workspace/ceph_dbstore/build/bin/radosgw(+0xd586eb) [0x55bf89e3f6eb]
10: (std::_Function_handler<int (RGWSI_MetaBackend_Handler::Op*), RGWBucketInstanceMetadataHandler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (ptr_wrapper<RGWSI_MetaBackend::Context, 4>&)>)::{lambda(RGWSI_MetaBackend_Handler::Op*)#1}>::_M_invoke(std::_Any_data const&, RGWSI_MetaBackend_Handler::Op*&&)+0x30) [0x55bf89e55590]
11: /root/workspace/ceph_dbstore/build/bin/radosgw(+0xc8b5af) [0x55bf89d725af]
12: (RGWSI_MetaBackend_SObj::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend::Context*)>)+0x5c) [0x55bf89d7403c]
13: (RGWSI_MetaBackend_Handler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend_Handler::Op*)>)+0x7c) [0x55bf89d729cc]
14: (RGWBucketCtl::store_bucket_instance_info(rgw_bucket const&, RGWBucketInfo&, optional_yield, DoutPrefixProvider const*, RGWBucketCtl::BucketInstance::PutParams const&)+0x15e) [0x55bf89e3e6fe]
15: (RGWRados::put_bucket_instance_info(RGWBucketInfo&, bool, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >, DoutPrefixProvider const, optional_yield)+0x4f) [0x55bf89ba710f]
16: (rgw::sal::RadosBucket::put_info(DoutPrefixProvider const*, bool, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x36) [0x55bf89c2bde6]
17: /root/workspace/ceph_dbstore/build/bin/radosgw(+0x89c7bd) [0x55bf899837bd]
18: (RGWPutBucketReplication::execute(optional_yield)+0x18a) [0x55bf89984c3a]
19: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb94) [0x55bf897c1404]
20: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, int*)+0x236f) [0x55bf897c3fef]
21: /root/workspace/ceph_dbstore/build/bin/radosgw(+0x63ee5b) [0x55bf89725e5b]
22: /root/workspace/ceph_dbstore/build/bin/radosgw(+0x63fda7) [0x55bf89726da7]
23: make_fcontext()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Set replication rules on a multisite setup using S3 put-bucket-replication API

[Note: To set replication rules, multisite sync-policy should be configured]

# cat ../scripts/replication.json 
{
  "Role": "arn:aws:iam:::role/S3Access1",
  "Rules": [
    {
      "ID": "rule1",
      "Status": "Enabled",
	  "Priority": 1,
	  "Filter": {
	    "Prefix": "lc"
      },
	  "Destination": {
	    "Bucket": "bucket1"
	  }
    }
  ]
}

# 
# aws --no-verify-ssl --endpoint-url http://localhost:8101 s3api put-bucket-replication --bucket bucket1 --replication-configuration file://../../scripts/replication.json
# aws --no-verify-ssl --endpoint-url http://localhost:8101 s3api get-bucket-replication --bucket bucket1 {
"ReplicationConfiguration": {
"Role": "",
"Rules": [ {
"ID": "rule1",
"Priority": 1,
"Filter": {
"Prefix": "lc"
},
"Status": "Enabled",
"Destination": {
"Bucket": "bucket1"
}
}
]
}
}
#

2. Now change the Destination to "bucket2" in the rules and re-apply them on the same bucket using same put-bucket-replication API. 



Actual results:

The server now crashes


Expected results:
The server shouldn't crash.

Additional info: