Description of problem: Segmentation fault encountered during manipulation of an object's content using ceph-objectstore-tool Reference from upstream documentation: https://docs.ceph.com/en/latest/man/8/ceph-objectstore-tool/#manipulating-an-object-s-content Segmentation fault was specifically observed only during get-bytes and set-bytes operations. Get bytes: ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID $OBJECT get-bytes > $OBJECT_FILE_NAME Set bytes: ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID $OBJECT set-bytes < $OBJECT_FILE_NAME Version-Release number of selected component (if applicable): - 18.2.1-10.el9cp How reproducible: 2/5 Steps to Reproduce: 1. Create a pool and write data to it, may use rados bench or any other tool 2. Stop an OSD service on any of the OSD nodes 3. From inside the OSD container use ceph-objectstore-tool to list list of objects # cephadm shell --name osd.1 -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --op list 4. Choose an object from the list obtained in the previous step 5. Obtain the content of the object using get-bytes operation # cephadm shell --name osd.1 --mount /tmp/ -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --pgid 29.1d '{"oid": "benchmark_last_metadata", "key": "", "snapid": -2, "hash": 2390394397, "max": 0, "pool": 29, "namespace": ""}' get-bytes > /mnt/obj_work 6. Modify or leave alone the file where object data was redirected. 7. Push content to the object using set-bytes operation # cephadm shell --name osd.1 --mount /tmp/ -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --pgid 29.1d '{"oid": "benchmark_last_metadata", "key": "", "snapid": -2, "hash": 2390394397, "max": 0, "pool": 29, "namespace": ""}' set-bytes < /mnt/obj_work Actual results: Observed segmentation fault during few executions of the above mentioned operations Expected results: No segmentation faults expected even in case of failures Additional info: Logs: get-bytes -------------------------------------------------------------------------------------------------------------------------------------------- http://magna002.ceph.redhat.com/ceph-qe-logs/harsh_magna/cot-seg-fault/cephci-run-EGRKQ9/ceph-objectstore-tool_utility_0.log Exception hit while command execution. cephadm shell --name osd.6 --mount /tmp/ -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-6 --pgid 24.17 '{"oid": "benchmark_data_ceph-hakumar-h01h6v-node1-ins_3_object88", "key": "", "snapid": -2, "hash": 4154654743, "max": 0, "pool": 24, "namespace": ""}' get-bytes > /mnt/obj_backup Error: Inferring fsid 628d2de6-c0d7-11ee-af8c-fa163e4240c7 Inferring config /var/lib/ceph/628d2de6-c0d7-11ee-af8c-fa163e4240c7/osd.6/config Using ceph image with id '18a49f4e73b3' and tag '<none>' created on 2024-01-31 00:23:56 +0000 UTC registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:5e19702546ffe42b24b5c05936fae05045083a2103a54fb9400a37fabdcd2e50 *** Caught signal (Segmentation fault) ** in thread 7ff6642ba580 thread_name:ceph-objectstor ceph version 18.2.1-10.el9cp (ccf42acecc9e7ec19c8994e4d2ca0180b612ad1e) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7ff6648c2db0] 2: __pthread_rwlock_rdlock() 3: (BlueStore::collection_bits(boost::intrusive_ptr<ObjectStore::CollectionImpl>&)+0x48) [0x558623e3ff38] 4: main() 5: /lib64/libc.so.6(+0x3feb0) [0x7ff6648adeb0] 6: __libc_start_main() 7: _start() http://magna002.ceph.redhat.com/ceph-qe-logs/harsh_magna/cot-seg-fault/cephci-run-F819AS/ceph-objectstore-tool_utility_0.log Exception hit while command execution. cephadm shell --name osd.11 --mount /tmp/ -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-11 --pgid 21.12 "{'oid': 'benchmark_data_ceph-hakumar-h01h6v-node1-ins_3_object131', 'key': '', 'snapid': -2, 'hash': 685328530, 'max': 0, 'pool': 21, 'namespace': ''}" get-bytes > /mnt/obj_backup Error: Inferring fsid 628d2de6-c0d7-11ee-af8c-fa163e4240c7 Inferring config /var/lib/ceph/628d2de6-c0d7-11ee-af8c-fa163e4240c7/osd.11/config Using ceph image with id '18a49f4e73b3' and tag '<none>' created on 2024-01-31 00:23:56 +0000 UTC registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:5e19702546ffe42b24b5c05936fae05045083a2103a54fb9400a37fabdcd2e50 *** Caught signal (Segmentation fault) ** in thread 7f93cf8de580 thread_name:ceph-objectstor ceph version 18.2.1-10.el9cp (ccf42acecc9e7ec19c8994e4d2ca0180b612ad1e) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7f93cfee6db0] 2: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x4b) [0x55c4cc5d6e5b] 3: (_action_on_all_objects_in_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x4cc) [0x55c4cc1217dc] 4: (action_on_all_objects_in_exact_pg(ObjectStore*, coll_t, action_on_object_t&, bool)+0x64) [0x55c4cc1226c4] 5: main() 6: /lib64/libc.so.6(+0x3feb0) [0x7f93cfed1eb0] 7: __libc_start_main() 8: _start() set-bytes -------------------------------------------------------------------------------------------------------------------------------------------- http://magna002.ceph.redhat.com/ceph-qe-logs/harsh_magna/cot-seg-fault/cephci-run-O7JK37/ceph-objectstore-tool_utility_0.log Exception hit while command execution. cephadm shell --name osd.3 --mount /tmp/ -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --pgid 30.16 '{"oid": "benchmark_data_ceph-hakumar-h01h6v-node1-ins_3_object96", "key": "", "snapid": -2, "hash": 2879658006, "max": 0, "pool": 30, "namespace": ""}' set-bytes < /mnt/obj_work Error: Inferring fsid 628d2de6-c0d7-11ee-af8c-fa163e4240c7 Inferring config /var/lib/ceph/628d2de6-c0d7-11ee-af8c-fa163e4240c7/osd.3/config Using ceph image with id '18a49f4e73b3' and tag '<none>' created on 2024-01-31 00:23:56 +0000 UTC registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:5e19702546ffe42b24b5c05936fae05045083a2103a54fb9400a37fabdcd2e50 *** Caught signal (Segmentation fault) ** in thread 7f9452740580 thread_name:ceph-objectstor ceph version 18.2.1-10.el9cp (ccf42acecc9e7ec19c8994e4d2ca0180b612ad1e) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7f9452d48db0] 2: __pthread_rwlock_rdlock() 3: (BlueStore::collection_bits(boost::intrusive_ptr<ObjectStore::CollectionImpl>&)+0x48) [0x55a6e489ff38] 4: main() 5: /lib64/libc.so.6(+0x3feb0) [0x7f9452d33eb0] 6: __libc_start_main() 7: _start() http://magna002.ceph.redhat.com/ceph-qe-logs/harsh_magna/cot-seg-fault/cephci-run-WSITZT/ceph-objectstore-tool_utility_0.log Exception hit while command execution. cephadm shell --name osd.1 --mount /tmp/ -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --pgid 29.1d '{"oid": "benchmark_last_metadata", "key": "", "snapid": -2, "hash": 2390394397, "max": 0, "pool": 29, "namespace": ""}' set-bytes < /mnt/obj_work Error: Inferring fsid 628d2de6-c0d7-11ee-af8c-fa163e4240c7 Inferring config /var/lib/ceph/628d2de6-c0d7-11ee-af8c-fa163e4240c7/osd.1/config Using ceph image with id '18a49f4e73b3' and tag '<none>' created on 2024-01-31 00:23:56 +0000 UTC registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:5e19702546ffe42b24b5c05936fae05045083a2103a54fb9400a37fabdcd2e50 *** Caught signal (Segmentation fault) ** in thread 7fed4fc40580 thread_name:ceph-objectstor ceph version 18.2.1-10.el9cp (ccf42acecc9e7ec19c8994e4d2ca0180b612ad1e) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7fed50248db0] 2: __pthread_rwlock_rdlock() 3: (BlueStore::collection_bits(boost::intrusive_ptr<ObjectStore::CollectionImpl>&)+0x48) [0x5644a49bbf38] 4: main() 5: /lib64/libc.so.6(+0x3feb0) [0x7fed50233eb0] 6: __libc_start_main() 7: _start()