Bug 1777743

Summary:

[RGW]: Crash observed on set_acl call

Product:

[Red Hat Storage] Red Hat Ceph Storage

Reporter:

Tejas <tchandra>

Component:

RGW

Assignee:

Matt Benjamin (redhat) <mbenjamin>

Status:

CLOSED ERRATA

QA Contact:

Tejas <tchandra>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.0

CC:

assingh, cbodley, ceph-eng-bugs, kbader, mbenjamin, sweil, tserlin

Target Milestone:

Keywords:

Automation, AutomationBlocker, TestBlocker

Target Release:

4.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

ceph-14.2.4-89.el8cp, ceph-14.2.4-27.el7cp

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-01-31 12:48:19 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
debug rgw logs	none

Description Tejas 2019-11-28 09:01:54 UTC

Description of problem:
ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable)
We are observing a crash from a teuthology rgw_singleton suite at the line :
https://github.com/ceph/ceph/blob/master/qa/tasks/radosgw_admin.py#L796

Teuthology run log:

6051eae0 msgr2=0x562360518170 unknown :-1 s=STATE_NONE l=0).mark_down
2019-11-27 23:57:16,508.508 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=217 failed=True
2019-11-27 23:57:16,668.668 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False
2019-11-27 23:57:16,998.998 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False
2019-11-27 23:57:17,249.249 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False
2019-11-27 23:57:17,436.436 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=5 bytes_in=0 failed=False
2019-11-27 23:57:17,439.439 INFO:tasks.rgw.client.0.extensa022.stdout:*** Caught signal (Segmentation fault) **
2019-11-27 23:57:17,440.440 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw
2019-11-27 23:57:17,443.443 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable)
2019-11-27 23:57:17,443.443 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0]
2019-11-27 23:57:17,443.443 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9]
2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f]
2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5]
2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e]
2019-11-27 23:57:17,444.444 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707]
2019-11-27 23:57:17,445.445 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853]
2019-11-27 23:57:17,445.445 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e]
2019-11-27 23:57:17,445.445 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509]
2019-11-27 23:57:17,446.446 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0]
2019-11-27 23:57:17,446.446 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8]
2019-11-27 23:57:17,446.446 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df]
2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562]
2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f]
2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout:2019-11-28 04:57:17.444 7fbc23e6e700 -1 *** Caught signal (Segmentation fault) **
2019-11-27 23:57:17,447.447 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw
2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout:
2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable)
2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0]
2019-11-27 23:57:17,448.448 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9]
2019-11-27 23:57:17,449.449 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f]
2019-11-27 23:57:17,449.449 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5]
2019-11-27 23:57:17,449.449 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e]
2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707]
2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853]
2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e]
2019-11-27 23:57:17,450.450 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509]
2019-11-27 23:57:17,451.451 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0]
2019-11-27 23:57:17,451.451 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8]
2019-11-27 23:57:17,451.451 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df]
2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562]
2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f]
2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-11-27 23:57:17,452.452 INFO:tasks.rgw.client.0.extensa022.stdout:
2019-11-27 23:57:17,453.453 INFO:tasks.rgw.client.0.extensa022.stdout:     0> 2019-11-28 04:57:17.444 7fbc23e6e700 -1 *** Caught signal (Segmentation fault) **
2019-11-27 23:57:17,453.453 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw
2019-11-27 23:57:17,454.454 INFO:tasks.rgw.client.0.extensa022.stdout:
2019-11-27 23:57:17,454.454 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable)
2019-11-27 23:57:17,454.454 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0]
2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9]
2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f]
2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5]
2019-11-27 23:57:17,455.455 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e]
2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707]
2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853]
2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e]
2019-11-27 23:57:17,456.456 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509]
2019-11-27 23:57:17,457.457 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0]
2019-11-27 23:57:17,457.457 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8]
2019-11-27 23:57:17,457.457 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df]
2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562]
2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f]
2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-11-27 23:57:17,458.458 INFO:tasks.rgw.client.0.extensa022.stdout:
2019-11-27 23:57:17,459.459 INFO:tasks.rgw.client.0.extensa022.stdout:     0> 2019-11-28 04:57:17.444 7fbc23e6e700 -1 *** Caught signal (Segmentation fault) **
2019-11-27 23:57:17,459.459 INFO:tasks.rgw.client.0.extensa022.stdout: in thread 7fbc23e6e700 thread_name:radosgw
2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout:
2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: ceph version 14.2.4-67.el8cp (ed755eb2a9049380ab251633c9387a8b51497777) nautilus (stable)
2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: 1: (()+0x12dc0) [0x7fbc5ff8ddc0]
2019-11-27 23:57:17,460.460 INFO:tasks.rgw.client.0.extensa022.stdout: 2: (std::_Rb_tree<rgw_obj, std::pair<rgw_obj const, RGWObjState>, std::_Select1st<std::pair<rgw_obj const, RGWObjState> >, std::less<rgw_obj>, std::allocator<std::pair<rgw_obj const, RGWObjState> > >::find(rgw_obj const&)+0x59) [0x55fba8f753f9]
2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 3: (RGWObjectCtx::get_state(rgw_obj const&)+0x7f) [0x55fba8f7e46f]
2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 4: (RGWRados::get_obj_state_impl(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x85) [0x55fba8f544f5]
2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 5: (RGWRados::get_obj_state(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, RGWObjState**, bool, bool)+0x4e) [0x55fba8f55e0e]
2019-11-27 23:57:17,461.461 INFO:tasks.rgw.client.0.extensa022.stdout: 6: (RGWRados::append_atomic_test(RGWObjectCtx*, RGWBucketInfo const&, rgw_obj const&, librados::v14_2_0::ObjectOperation&, RGWObjState**)+0x27) [0x55fba8f57707]
2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 7: (RGWRados::set_attrs(void*, RGWBucketInfo const&, rgw_obj&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v14_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v14_2_0::list> > >*)+0x123) [0x55fba8f57853]
2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 8: (()+0x5f102e) [0x55fba8ed002e]
2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 9: (RGWPutACLs::execute()+0xee9) [0x55fba8ee8509]
2019-11-27 23:57:17,462.462 INFO:tasks.rgw.client.0.extensa022.stdout: 10: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x9a0) [0x55fba8c8a2c0]
2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 11: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x2178) [0x55fba8c8ccd8]
2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 12: (()+0x3145df) [0x55fba8bf35df]
2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 13: (()+0x315562) [0x55fba8bf4562]
2019-11-27 23:57:17,463.463 INFO:tasks.rgw.client.0.extensa022.stdout: 14: (make_fcontext()+0x2f) [0x55fba90f4c8f]
2019-11-27 23:57:17,464.464 INFO:tasks.rgw.client.0.extensa022.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-11-27 23:57:17,464.464 INFO:tasks.rgw.client.0.extensa022.stdout:
2019-11-27 23:57:20,214.214 INFO:tasks.rgw.client.0.extensa022.stderr:daemon-helper: command crashed with signal 11
2019-11-27 23:59:14,454.454 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/tchandra/teuthology/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/tchandra/teuthology/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/tchandra/ceph_src/ceph/qa/tasks/radosgw_admin.py", line 748, in task
    key.set_acl('private')
  File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 569, in set_acl
    self.bucket.set_acl(acl_str, self.name, headers=headers)
  File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 944, in set_acl
    headers, version_id)
  File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 904, in set_canned_acl
    headers=headers, query_args=query_args)
  File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
    retry_handler=retry_handler
  File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/home/tchandra/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
    raise ex
error: [Errno 111] Connection refused

I have attached the debug rgw logs to this BZ.

Comment 1 Tejas 2019-11-28 09:06:41 UTC

Created attachment 1640320 [details]
debug rgw logs

Comment 10 errata-xmlrpc 2020-01-31 12:48:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0312