Description of problem: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable) We are observing a crash from a teuthology rgw_singleton suite at the line : https://github.com/ceph/ceph/blob/master/qa/tasks/radosgw_admin.py#L796 Teuthology run log: 6051eae0 msgr2=0x562360518170 unknown :-1 s=STATE_NONE l=0).mark_down 2019-11-27 23:57:16,508.508 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=217 failed=True 2019-11-27 23:57:16,668.668 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2019-11-27 23:57:16,998.998 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2019-11-27 23:57:17,249.249 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2019-11-27 23:57:17,436.436 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=5 bytes_in=0 failed=False 2019-11-27 23:57:17,439.439 INFO:tasks.rgw.client.0.extensa022.stdout:*** Caught signal (Segmentation fault) ** 2019-11-27 23:57:17,440.440 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw 2019-11-27 23:57:17,443.443 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable) 2019-11-27 23:57:17,443.443 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0] 2019-11-27 23:57:17,443.443 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9] 2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f] 2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5] 2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e] 2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707] 2019-11-27 23:57:17,445.445 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853] 2019-11-27 23:57:17,445.445 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e] 2019-11-27 23:57:17,445.445 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509] 2019-11-27 23:57:17,446.446 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0] 2019-11-27 23:57:17,446.446 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8] 2019-11-27 23:57:17,446.446 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df] 2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562] 2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f] 2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout:2019-11-28 04:57:17.444 7fbc23e6e700 -1 *** Caught signal (Segmentation fault) ** 2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw 2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: 2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable) 2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0] 2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9] 2019-11-27 23:57:17,449.449 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f] 2019-11-27 23:57:17,449.449 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5] 2019-11-27 23:57:17,449.449 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e] 2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707] 2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853] 2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e] 2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509] 2019-11-27 23:57:17,451.451 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0] 2019-11-27 23:57:17,451.451 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8] 2019-11-27 23:57:17,451.451 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df] 2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562] 2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f] 2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: 2019-11-27 23:57:17,453.453 INFO:tasks.rgw.client.0.extensa022.stdout: 0> 2019-11-28 04:57:17.444 7fbc23e6e700 -1 *** Caught signal (Segmentation fault) ** 2019-11-27 23:57:17,453.453 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw 2019-11-27 23:57:17,454.454 INFO:tasks.rgw.client.0.extensa022.stdout: 2019-11-27 23:57:17,454.454 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable) 2019-11-27 23:57:17,454.454 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0] 2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9] 2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f] 2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5] 2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e] 2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707] 2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853] 2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e] 2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509] 2019-11-27 23:57:17,457.457 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0] 2019-11-27 23:57:17,457.457 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8] 2019-11-27 23:57:17,457.457 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df] 2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562] 2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f] 2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: 2019-11-27 23:57:17,459.459 INFO:tasks.rgw.client.0.extensa022.stdout: 0> 2019-11-28 04:57:17.444 7fbc23e6e700 -1 *** Caught signal (Segmentation fault) ** 2019-11-27 23:57:17,459.459 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw 2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: 2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable) 2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0] 2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9] 2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f] 2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5] 2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e] 2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707] 2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853] 2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e] 2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509] 2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0] 2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8] 2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df] 2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562] 2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f] 2019-11-27 23:57:17,464.464 INFO:tasks.rgw.client.0.extensa022.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2019-11-27 23:57:17,464.464 INFO:tasks.rgw.client.0.extensa022.stdout: 2019-11-27 23:57:20,214.214 INFO:tasks.rgw.client.0.extensa022.stderr:daemon-helper: command crashed with signal 11 2019-11-27 23:59:14,454.454 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/tchandra/teuthology/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/tchandra/teuthology/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/tchandra/ceph_src/ceph/qa/tasks/radosgw_admin.py", line 748, in task key.set_acl('private') File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 569, in set_acl self.bucket.set_acl(acl_str, self.name, headers=headers) File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 944, in set_acl headers, version_id) File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 904, in set_canned_acl headers=headers, query_args=query_args) File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request retry_handler=retry_handler File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request retry_handler=retry_handler) File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe raise ex error: [Errno 111] Connection refused I have attached the debug rgw logs to this BZ.
Created attachment 1640320 [details] debug rgw logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0312