Description of problem:

Randomly transitioned objects end up with 0 bytes on-premise and on the cloud provider losing the data in that object.


I have the on-premise ceph cluster:

aws s3 --ca-bundle  /etc/pki/ca-trust/source/anchors/myCA.pem --profile rgw  --endpoint https://ceph-bos-2.makestoragegreatagain.com:8043 --region default ls s3://mixtrans/aws/ | grep aws1116
2022-11-03 08:44:31          0 aws1116


aws s3api --ca-bundle  /etc/pki/ca-trust/source/anchors/myCA.pem --profile rgw --endpoint https://ceph-bos-2.makestoragegreatagain.com:8043 --region default  head-object --bucket mixtrans --key aws/aws1116
{
    "AcceptRanges": "bytes",
    "LastModified": "2022-11-03T07:44:31+00:00",
    "ContentLength": 0,
    "ETag": "\"46ecb42fd0def0e42f85922d62d06766\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "StorageClass": "CLOUDAWS"
}

And on the cloud provide, this AWS S3, we have the object with the metadata but with 0 bytes:

aws --profile aws s3 ls  s3://rgwx-default-cloudaws-cloud-bucket/mixtrans/aws/ | grep aws1116
2022-11-03 08:44:31          0 aws1116

aws --profile aws s3api  head-object --bucket rgwx-default-cloudaws-cloud-bucket --key mixtrans/aws/aws1116
{
    "AcceptRanges": "bytes",
    "LastModified": "2022-11-03T07:44:31+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {
        "rgwx-source-mtime": "1667461396.915537829",
        "rgwx-versioned-epoch": "0",
        "rgwx-source": "rgw",
        "rgwx-source-etag": "46ecb42fd0def0e42f85922d62d06766",
        "rgwx-source-key": "aws/aws1116"
    }
}


I have found no ERROR logs for the failed files in RGW with debug 20. I'm adding the output of logs with a grep of the affected file:

[root@ceph-bos-2 28a9131e-515d-11ed-8aa7-00505683f802]# cat ceph-client.rgw.objectgwhttps.ceph-bos-2.udyllp.log |  grep aws1116
2022-11-03T07:50:00.270+0000 7fab677ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T07:50:00.273+0000 7fab68ff1640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 0
2022-11-03T07:50:00.273+0000 7fab68ff1640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 0
2022-11-03T07:52:00.246+0000 7fab637e6640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T07:52:00.248+0000 7fab647e8640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 1
2022-11-03T07:52:00.248+0000 7fab647e8640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 1
2022-11-03T07:59:51.371+0000 7fab657ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T07:59:51.374+0000 7fab65feb640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 2
2022-11-03T07:59:51.374+0000 7fab65feb640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 2
2022-11-03T08:02:52.872+0000 7fab677ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:02:52.875+0000 7fab67fef640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 2
2022-11-03T08:02:52.875+0000 7fab67fef640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 2
2022-11-03T08:03:52.005+0000 7fab677ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:03:52.007+0000 7fab687f0640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 1
2022-11-03T08:03:52.007+0000 7fab687f0640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 1
2022-11-03T08:08:15.068+0000 7f032ffe7640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:08:15.070+0000 7f0330fe9640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 1
2022-11-03T08:08:15.070+0000 7f0330fe9640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 1
2022-11-03T08:09:15.960+0000 7f032ffe7640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:09:15.963+0000 7f03307e8640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 2
2022-11-03T08:09:15.963+0000 7f03307e8640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 2
2022-11-03T08:10:15.420+0000 7f0333fef640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:10:15.422+0000 7f03347f0640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 2
2022-11-03T08:10:15.422+0000 7f03347f0640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 2
2022-11-03T08:22:50.203+0000 7fc1597ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:22:50.205+0000 7fc15a7ec640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 1
2022-11-03T08:22:50.205+0000 7fc15a7ec640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 1
2022-11-03T08:24:48.383+0000 7fc15d7f2640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:24:48.385+0000 7fc15dff3640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 2
2022-11-03T08:24:48.385+0000 7fc15dff3640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 2
2022-11-03T08:28:44.669+0000 7fc15b7ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:28:44.671+0000 7fc15c7f0640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 1
2022-11-03T08:28:44.671+0000 7fc15c7f0640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 1
2022-11-03T08:29:44.517+0000 7fc15b7ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:29:44.522+0000 7fc15bfef640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 2
2022-11-03T08:29:44.522+0000 7fc15bfef640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 2
2022-11-03T08:30:44.040+0000 7fc15b7ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:30:44.042+0000 7fc15bfef640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 2
2022-11-03T08:30:44.042+0000 7fc15bfef640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 2
2022-11-03T08:34:33.964+0000 7fa1befe5640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:34:33.966+0000 7fa1bf7e6640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 2
2022-11-03T08:34:33.966+0000 7fa1bf7e6640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 2
2022-11-03T08:35:33.447+0000 7fa1c2fed640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:35:33.449+0000 7fa1c3fef640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 1
2022-11-03T08:35:33.449+0000 7fa1c3fef640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 1
2022-11-03T08:36:33.360+0000 7fa1befe5640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:36:33.363+0000 7fa1bf7e6640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 2
2022-11-03T08:36:33.363+0000 7fa1bf7e6640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 2
2022-11-03T08:42:28.005+0000 7fa1befe5640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:42:28.007+0000 7fa1bf7e6640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 2
2022-11-03T08:42:28.007+0000 7fa1bf7e6640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 2
2022-11-03T08:55:08.130+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:55:08.132+0000 7fefdefed640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 1
2022-11-03T08:55:08.132+0000 7fefdefed640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 1
2022-11-03T08:56:08.068+0000 7fefdffef640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:56:08.070+0000 7fefe07f0640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 2
2022-11-03T08:56:08.070+0000 7fefe07f0640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 2
2022-11-03T08:58:13.850+0000 7fefdbfe7640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T08:58:13.852+0000 7fefdc7e8640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 2
2022-11-03T08:58:13.852+0000 7fefdc7e8640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 2
2022-11-03T09:04:00.984+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:04:00.986+0000 7fefdf7ee640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T09:04:00.986+0000 7fefdf7ee640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T09:05:00.706+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:05:00.707+0000 7fefdf7ee640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T09:05:00.707+0000 7fefdf7ee640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T09:06:00.008+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:06:00.010+0000 7fefdf7ee640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T09:06:00.010+0000 7fefdf7ee640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T09:12:54.716+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:12:54.718+0000 7fefde7ec640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 2
2022-11-03T09:12:54.718+0000 7fefde7ec640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 2
2022-11-03T09:13:54.130+0000 7fefdbfe7640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:13:54.132+0000 7fefdcfe9640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 1
2022-11-03T09:13:54.132+0000 7fefdcfe9640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 1
2022-11-03T09:14:54.298+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:14:54.299+0000 7fefdefed640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 1
2022-11-03T09:14:54.299+0000 7fefdefed640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 1
2022-11-03T09:15:54.010+0000 7fefddfeb640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:15:54.012+0000 7fefdf7ee640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T09:15:54.012+0000 7fefdf7ee640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T09:26:51.370+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:26:51.372+0000 7f72d6fed640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T09:26:51.372+0000 7f72d6fed640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T09:32:40.798+0000 7f72d37e6640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:32:40.800+0000 7f72d4fe9640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 0
2022-11-03T09:32:40.800+0000 7f72d4fe9640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 0
2022-11-03T09:33:41.680+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:33:41.683+0000 7f72d67ec640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 1
2022-11-03T09:33:41.683+0000 7f72d67ec640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 1
2022-11-03T09:38:12.970+0000 7f7258e39640 20 req 13627548752279526512 0.001000017s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): currently processing aws/aws1116 from shard 4
2022-11-03T09:38:12.970+0000 7f7258e39640 10 req 13627548752279526512 0.001000017s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): got aws/aws1116
2022-11-03T09:38:12.970+0000 7f7258e39640 20 req 13627548752279526512 0.001000017s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): considering entry aws/aws1116
2022-11-03T09:38:12.970+0000 7f7258e39640 20 req 13627548752279526512 0.001000017s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): adding entry aws/aws1116 to result
2022-11-03T09:42:33.158+0000 7f72d77ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:42:33.160+0000 7f72d87f0640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 1
2022-11-03T09:42:33.160+0000 7f72d87f0640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 1
2022-11-03T09:43:36.059+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:43:36.061+0000 7f72d5feb640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 2
2022-11-03T09:43:36.061+0000 7f72d5feb640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 2
2022-11-03T09:44:39.826+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:44:39.828+0000 7f72d6fed640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T09:44:39.828+0000 7f72d6fed640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T09:48:16.722+0000 7f7290ea9640 20 req 15149317889304163185 0.001000017s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): currently processing aws/aws1116 from shard 4
2022-11-03T09:48:16.722+0000 7f7290ea9640 10 req 15149317889304163185 0.001000017s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): got aws/aws1116
2022-11-03T09:48:16.722+0000 7f7290ea9640 20 req 15149317889304163185 0.001000017s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): considering entry aws/aws1116
2022-11-03T09:48:16.722+0000 7f7290ea9640 20 req 15149317889304163185 0.001000017s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): adding entry aws/aws1116 to result
2022-11-03T09:49:26.508+0000 7f72d77ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:49:26.510+0000 7f72d8ff1640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 0
2022-11-03T09:49:26.510+0000 7f72d8ff1640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 0
2022-11-03T09:50:29.113+0000 7f72d37e6640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:50:29.115+0000 7f72d47e8640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 1
2022-11-03T09:50:29.115+0000 7f72d47e8640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 1
2022-11-03T09:51:29.038+0000 7f72d37e6640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:51:29.040+0000 7f72d4fe9640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 0
2022-11-03T09:51:29.040+0000 7f72d4fe9640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 0
2022-11-03T09:52:29.190+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:52:29.191+0000 7f72d5feb640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 2
2022-11-03T09:52:29.191+0000 7f72d5feb640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 2
2022-11-03T09:53:29.306+0000 7f72d77ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T09:53:29.308+0000 7f72d7fef640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 2
2022-11-03T09:53:29.308+0000 7f72d7fef640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 2
2022-11-03T09:58:52.730+0000 7f7288e99640 20 REQUEST_URI=/mixtrans/aws/aws1116
2022-11-03T09:58:52.730+0000 7f7288e99640 20 SCRIPT_URI=/mixtrans/aws/aws1116
2022-11-03T09:58:52.730+0000 7f7288e99640 20 req 11353814582499664161 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/mixtrans/aws/aws1116
/mixtrans/aws/aws1116
2022-11-03T09:58:52.732+0000 7f727f686640 10 req 11353814582499664161 0.002000034s s->object=aws/aws1116 s->bucket=mixtrans
2022-11-03T09:58:52.732+0000 7f727fe87640 20 req 11353814582499664161 0.002000034s s3:get_obj get_obj_state: rctx=0x7f7379e98950 obj=mixtrans:aws/aws1116 state=0x7f735c0e4e98 s->prefetch_data=0
2022-11-03T09:58:52.733+0000 7f727fe87640 20 req 11353814582499664161 0.003000050s s3:get_obj get_obj_state: rctx=0x7f7379e98950 obj=mixtrans:aws/aws1116 state=0x7f735c0e4e98 s->prefetch_data=0
2022-11-03T09:58:52.733+0000 7f727ee85640  1 beast: 0x7f7379e995f0: 10.33.36.156 - test [03/Nov/2022:09:58:52.730 +0000] "HEAD /mixtrans/aws/aws1116 HTTP/1.1" 200 0 - "aws-cli/2.4.12 Python/3.8.8 Linux/5.19.16-200.fc36.x86_64 exe/x86_64.fedora.36 prompt/off command/s3api.head-object" - latency=0.003000050s
2022-11-03T10:00:19.939+0000 7f72d37e6640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T10:00:19.941+0000 7f72d4fe9640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 2, 0
2022-11-03T10:00:19.941+0000 7f72d4fe9640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 2, 0
2022-11-03T10:01:19.927+0000 7f72d77ee640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T10:01:19.929+0000 7f72d7fef640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 0, 2
2022-11-03T10:01:19.929+0000 7f72d7fef640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 0, 2
2022-11-03T10:02:21.740+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T10:02:21.743+0000 7f72d6fed640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T10:02:21.743+0000 7f72d6fed640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T10:03:25.483+0000 7f72d57ea640 10 lifecycle: cls_bucket_list_unordered: got aws/aws1116
2022-11-03T10:03:25.485+0000 7f72d6fed640 20 lifecycle: operator()(): key=aws/aws1116wp_thrd: 1, 0
2022-11-03T10:03:25.485+0000 7f72d6fed640 20 lifecycle: check(): key=aws/aws1116: is_expired=1 wp_thrd: 1, 0
2022-11-03T10:03:40.708+0000 7f7276674640 20 req 16412538649144004115 0.001000016s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): currently processing aws/aws1116 from shard 4
2022-11-03T10:03:40.708+0000 7f7276674640 10 req 16412538649144004115 0.001000016s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): got aws/aws1116
2022-11-03T10:03:40.708+0000 7f7276674640 20 req 16412538649144004115 0.001000016s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): considering entry aws/aws1116
2022-11-03T10:03:40.708+0000 7f7276674640 20 req 16412538649144004115 0.001000016s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): adding entry aws/aws1116 to result
2022-11-03T10:03:50.281+0000 7f726de63640 20 req 844329589889571599 0.001000017s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): currently processing aws/aws1116 from shard 4
2022-11-03T10:03:50.281+0000 7f726de63640 10 req 844329589889571599 0.001000017s s3:list_bucket int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): got aws/aws1116
2022-11-03T10:03:50.281+0000 7f726de63640 20 req 844329589889571599 0.001000017s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): considering entry aws/aws1116
2022-11-03T10:03:50.281+0000 7f726de63640 20 req 844329589889571599 0.001000017s s3:list_bucket int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield): adding entry aws/aws1116 to result


Version-Release number of selected component (if applicable):


How reproducible:

Random

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: