Bug 2271637

Summary: Getting an RGW service Segfault when assigning an attribute to an IAM role
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Matt Benjamin (redhat) <mbenjamin>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED ERRATA QA Contact: Vidushi Mishra <vimishra>
Severity: medium Docs Contact: Disha Walvekar <dwalveka>
Priority: unspecified    
Version: 5.3CC: cbodley, ceph-eng-bugs, cephqe-warriors, dparkes, dwalveka, gsitlani, mbenjamin, mkasturi, prsrivas, ssanchet, tserlin, vereddy, vimishra, vumrao
Target Milestone: ---   
Target Release: 7.0z2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-18.2.0-160.el9cp Doc Type: Bug Fix
Doc Text:
Previously, a variable representing an RGW role was being accessed before it was initialized. As a result, the illegal access induced a segfault (crash). With this fix, operations have been re-ordered. Now, there is no illegal access and roles are enforced as required.
 Story Points: ---
Clone Of: 2252048 Environment:
Last Closed: 2024-05-07 12:09:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2252048    
Bug Blocks: 2260859, 2270485    

Description Matt Benjamin (redhat) 2024-03-26 16:14:16 UTC 
+++ This bug was initially created as a clone of Bug #2252048 +++

Hi,

I'm Getting an RGW service Segfault when assigning an attribute to an IAM role.

# ceph --version
ceph version 16.2.10-187.el8cp (5d6355e2bccd18b5c6457a34cb666d773f21823d) pacific (stable)
# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 (Ootpa)


# aws --no-verify-ssl  --profile admin --endpoint https://cephproxy1.stg.local:8443 iam tag-role --role-name rgwabac-department --tags Key=Department,Value=Engineering
Could not connect to the endpoint URL: "https://cephproxy1.stg.local:8443/"


Snippet:

Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -121> 2023-11-29T08:07:00.935+0000 7f2e7610c700  2 req 12755686132095393521 0.000000000s initializing for trans_id = tx00000b1054cf461ec1ef1-006566f124-1e7f7-zone1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -120> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s rgw api priority: s3=8 s3website=7
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -119> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s host=cephproxy1.stg.local
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -118> 2023-11-29T08:07:00.935+0000 7f2e7610c700 20 req 12755686132095393521 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -117> 2023-11-29T08:07:00.935+0000 7f2e7610c700 20 req 12755686132095393521 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -116> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s meta>> HTTP_X_AMZ_DATE
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -115> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s x>> x-amz-date:20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -114> 2023-11-29T08:07:00.935+0000 7f2e7610c700 20 req 12755686132095393521 0.000000000s get_handler handler=26RGWHandler_REST_Service_S3
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -113> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s handler=26RGWHandler_REST_Service_S3
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -112> 2023-11-29T08:07:00.935+0000 7f2e7610c700  2 req 12755686132095393521 0.000000000s getting op 4
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -111> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s Content of POST: Action=TagRole&Version=2010-05-08&RoleName=rgwabac-department&Tags.member.1.Key=Department&Tags.member.1.Value=Engineering
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -110> 2023-11-29T08:07:00.935+0000 7f2e7610c700 10 req 12755686132095393521 0.000000000s Content of POST: Action=TagRole&Version=2010-05-08&RoleName=rgwabac-department&Tags.member.1.Key=Department&Tags.member.1.Value=Engineering
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -109> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s iam:tag_role scheduling with throttler client=0 cost=1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -108> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s iam:tag_role op=10RGWTagRole
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -107> 2023-11-29T08:07:00.937+0000 7f2e73907700  2 req 12755686132095393521 0.002000052s iam:tag_role verifying requester
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -106> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -105> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -104> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::S3AnonymousEngine denied with reason=-1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -103> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::STSAuthStrategy
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -102> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::STSAuthStrategy: trying rgw::auth::s3::STSEngine
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -101> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s v4 signature format = 10aba06de39cff79fd7a8973bbe1d87ffa2f6dd9d84448e39308227e9eaf723d
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug   -100> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s v4 credential format = admin/20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -99> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s access key id = admin
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -98> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s credential scope = 20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -97> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s canonical headers format = content-type:application/x-www-form-urlencoded; charset=utf-8
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: host:cephproxy1.stg.local:8443
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: x-amz-date:20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -96> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s payload request hash = 5ec91c71abcab6e51968cee900f6f5089bf02262d287ae76e9c0a73cc5815694
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -95> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s canonical request = POST
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: /
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: content-type:application/x-www-form-urlencoded; charset=utf-8
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: host:cephproxy1.stg.local:8443
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: x-amz-date:20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: content-type;host;x-amz-date
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 5ec91c71abcab6e51968cee900f6f5089bf02262d287ae76e9c0a73cc5815694
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -94> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s canonical request hash = 7070eea678f642bc178d645225bba0dbd026eddad658e427080fded2de153092
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -93> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s string to sign = AWS4-HMAC-SHA256
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 7070eea678f642bc178d645225bba0dbd026eddad658e427080fded2de153092
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -92> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::STSEngine denied with reason=-13
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -91> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::STSAuthStrategy denied with reason=-13
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: deb
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: ug    -90> 2023-11-29T08:07:00.937+0000 7f2e73907700 20 req 12755686132095393521 0.002000052s iam:tag_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -89> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s v4 signature format = 10aba06de39cff79fd7a8973bbe1d87ffa2f6dd9d84448e39308227e9eaf723d
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -88> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s v4 credential format = admin/20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -87> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s access key id = admin
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -86> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s credential scope = 20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -85> 2023-11-29T08:07:00.937+0000 7f2e73907700 10 req 12755686132095393521 0.002000052s canonical headers format = content-type:application/x-www-form-urlencoded; charset=utf-8
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: host:cephproxy1.stg.local:8443
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: x-amz-date:20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -84> 2023-11-29T08:07:00.938+0000 7f2e73907700 10 req 12755686132095393521 0.003000078s payload request hash = 5ec91c71abcab6e51968cee900f6f5089bf02262d287ae76e9c0a73cc5815694
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -83> 2023-11-29T08:07:00.938+0000 7f2e73907700 10 req 12755686132095393521 0.003000078s canonical request = POST
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: /
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: content-type:application/x-www-form-urlencoded; charset=utf-8
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: host:cephproxy1.stg.local:8443
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: x-amz-date:20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: content-type;host;x-amz-date
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 5ec91c71abcab6e51968cee900f6f5089bf02262d287ae76e9c0a73cc5815694
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -82> 2023-11-29T08:07:00.938+0000 7f2e73907700 10 req 12755686132095393521 0.003000078s canonical request hash = 7070eea678f642bc178d645225bba0dbd026eddad658e427080fded2de153092
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -81> 2023-11-29T08:07:00.938+0000 7f2e73907700 10 req 12755686132095393521 0.003000078s string to sign = AWS4-HMAC-SHA256
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 7070eea678f642bc178d645225bba0dbd026eddad658e427080fded2de153092
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -80> 2023-11-29T08:07:00.938+0000 7f2e73907700 20 req 12755686132095393521 0.003000078s iam:tag_role get_system_obj_state: rctx=0x7f2ebad34838 obj=zone1.rgw.meta:users.keys:admin state=0x5577b6858e20 s->prefetch_data=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -79> 2023-11-29T08:07:00.938+0000 7f2e73907700 10 req 12755686132095393521 0.003000078s iam:tag_role cache get: name=zone1.rgw.meta+users.keys+admin : miss
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -78> 2023-11-29T08:07:00.939+0000 7f2e73106700 10 req 12755686132095393521 0.004000104s iam:tag_role cache put: name=zone1.rgw.meta+users.keys+admin info.flags=0x6
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -77> 2023-11-29T08:07:00.939+0000 7f2e73106700 10 req 12755686132095393521 0.004000104s iam:tag_role adding zone1.rgw.meta+users.keys+admin to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -76> 2023-11-29T08:07:00.939+0000 7f2e73106700 20 req 12755686132095393521 0.004000104s iam:tag_role get_system_obj_state: s->obj_tag was set empty
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -75> 2023-11-29T08:07:00.939+0000 7f2e73106700 10 req 12755686132095393521 0.004000104s iam:tag_role cache get: name=zone1.rgw.meta+users.keys+admin : type miss (requested=0x1, cached=0x6)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -74> 2023-11-29T08:07:00.939+0000 7f2e73106700 20 req 12755686132095393521 0.004000104s iam:tag_role rados->read ofs=0 len=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -73> 2023-11-29T08:07:00.940+0000 7f2e72104700 20 req 12755686132095393521 0.005000131s iam:tag_role rados_obj.operate() r=0 bl.length=9
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -72> 2023-11-29T08:07:00.940+0000 7f2e72104700 10 req 12755686132095393521 0.005000131s iam:tag_role cache put: name=zone1.rgw.meta+users.keys+admin info.flags=0x1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -71> 2023-11-29T08:07:00.940+0000 7f2e72104700 10 req 12755686132095393521 0.005000131s iam:tag_role moving zone1.rgw.meta+users.keys+admin to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -70> 2023-11-29T08:07:00.940+0000 7f2e72104700 20 req 12755686132095393521 0.005000131s iam:tag_role get_system_obj_state: rctx=0x7f2ebad34838 obj=zone1.rgw.meta:users.uid:admin state=0x5577b6859060 s->prefetch_data=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -69> 2023-11-29T08:07:00.940+0000 7f2e72104700 10 req 12755686132095393521 0.005000131s iam:tag_role cache get: name=zone1.rgw.meta+users.uid+admin : miss
Nov 29 11:07:01 cephproxy1.stg.local systemd[1]: Started Process Core Dump (PID 2295645/UID 0).
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -68> 2023-11-29T08:07:00.942+0000 7f2e72905700 10 req 12755686132095393521 0.007000183s iam:tag_role cache put: name=zone1.rgw.meta+users.uid+admin info.flags=0x16
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -67> 2023-11-29T08:07:00.942+0000 7f2e72905700 10 req 12755686132095393521 0.007000183s iam:tag_role adding zone1.rgw.meta+users.uid+admin to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -66> 2023-11-29T08:07:00.942+0000 7f2e72905700 10 req 12755686132095393521 0.007000183s iam:tag_role updating xattr: name=ceph.objclass.version bl.length()=42
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -65> 2023-11-29T08:07:00.942+0000 7f2e72905700 20 req 12755686132095393521 0.007000183s iam:tag_role get_system_obj_state: s->obj_tag was set empty
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -64> 2023-11-29T08:07:00.942+0000 7f2e72905700 10 req 12755686132095393521 0.007000183s iam:tag_role cache get: name=zone1.rgw.meta+users.uid+admin : type miss (requested=0x11, cached=0x16)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -63> 2023-11-29T08:07:00.942+0000 7f2e72905700 20 req 12755686132095393521 0.007000183s iam:tag_role rados->read ofs=0 len=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -62> 2023-11-29T08:07:00.942+0000 7f2e71903700 20 req 12755686132095393521 0.007000183s iam:tag_role rados_obj.operate() r=0 bl.length=237
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -61> 2023-11-29T08:07:00.942+0000 7f2e71903700 10 req 12755686132095393521 0.007000183s iam:tag_role cache put: name=zone1.rgw.meta+users.uid+admin info.flags=0x11
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -60> 2023-11-29T08:07:00.942+0000 7f2e71903700 10 req 12755686132095393521 0.007000183s iam:tag_role moving zone1.rgw.meta+users.uid+admin to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -59> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s iam:tag_role chain_cache_entry: cache_locator=zone1.rgw.meta+users.uid+admin
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -58> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s date_k    = d6d25b0f94133cff17bdae5c3e4cf9d502d3519169d5f99f0cad340d2f40048c
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -57> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s region_k  = a257cbd9c56d51fc7b645922030b8ace8689c7964994168a069a446c414bc8aa
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -56> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s service_k = 69e5457b2dd506415e3369697613d74af5ab1753c44c5aeb8bd4bd8767500ad7
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -55> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s signing_k = 56df3a6e89fc28196609fd5735c2be3f70bb02f2b57f98b75c0b71487ba5169c
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -54> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s generated signature = 10aba06de39cff79fd7a8973bbe1d87ffa2f6dd9d84448e39308227e9eaf723d
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -53> 2023-11-29T08:07:00.943+0000 7f2e71903700 15 req 12755686132095393521 0.008000209s iam:tag_role string_to_sign=AWS4-HMAC-SHA256
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 20231129T080700Z
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 20231129/multizg/iam/aws4_request
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 7070eea678f642bc178d645225bba0dbd026eddad658e427080fded2de153092
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -52> 2023-11-29T08:07:00.943+0000 7f2e71903700 15 req 12755686132095393521 0.008000209s iam:tag_role server signature=10aba06de39cff79fd7a8973bbe1d87ffa2f6dd9d84448e39308227e9eaf723d
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -51> 2023-11-29T08:07:00.943+0000 7f2e71903700 15 req 12755686132095393521 0.008000209s iam:tag_role client signature=10aba06de39cff79fd7a8973bbe1d87ffa2f6dd9d84448e39308227e9eaf723d
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -50> 2023-11-29T08:07:00.943+0000 7f2e71903700 15 req 12755686132095393521 0.008000209s iam:tag_role compare=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -49> 2023-11-29T08:07:00.943+0000 7f2e71903700 20 req 12755686132095393521 0.008000209s iam:tag_role rgw::auth::s3::LocalEngine granted access
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -48> 2023-11-29T08:07:00.943+0000 7f2e71903700 20 req 12755686132095393521 0.008000209s iam:tag_role rgw::auth::s3::AWSAuthStrategy granted access
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -47> 2023-11-29T08:07:00.943+0000 7f2e71903700  2 req 12755686132095393521 0.008000209s iam:tag_role normalizing buckets and tenants
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -46> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s s->object=<NULL> s->bucket=
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -45> 2023-11-29T08:07:00.943+0000 7f2e71903700  2 req 12755686132095393521 0.008000209s iam:tag_role init permissions
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -44> 2023-11-29T08:07:00.943+0000 7f2e71903700 20 req 12755686132095393521 0.008000209s iam:ta
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: g_role get_system_obj_state: rctx=0x7f2ebad347b8 obj=zone1.rgw.meta:users.uid:admin state=0x5577b6858e20 s->prefetch_data=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -43> 2023-11-29T08:07:00.943+0000 7f2e71903700 10 req 12755686132095393521 0.008000209s iam:tag_role cache get: name=zone1.rgw.meta+users.uid+admin : type miss (requested=0x6, cached=0x13)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -42> 2023-11-29T08:07:00.944+0000 7f2e71102700 10 req 12755686132095393521 0.009000235s iam:tag_role cache put: name=zone1.rgw.meta+users.uid+admin info.flags=0x6
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -41> 2023-11-29T08:07:00.944+0000 7f2e71102700 10 req 12755686132095393521 0.009000235s iam:tag_role moving zone1.rgw.meta+users.uid+admin to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -40> 2023-11-29T08:07:00.944+0000 7f2e71102700 10 req 12755686132095393521 0.009000235s iam:tag_role updating xattr: name=ceph.objclass.version bl.length()=42
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -39> 2023-11-29T08:07:00.944+0000 7f2e71102700 20 req 12755686132095393521 0.009000235s iam:tag_role get_system_obj_state: s->obj_tag was set empty
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -38> 2023-11-29T08:07:00.944+0000 7f2e71102700 20 req 12755686132095393521 0.009000235s iam:tag_role Read xattr: user.rgw.idtag
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -37> 2023-11-29T08:07:00.944+0000 7f2e71102700 10 req 12755686132095393521 0.009000235s iam:tag_role cache get: name=zone1.rgw.meta+users.uid+admin : hit (requested=0x3, cached=0x7)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -36> 2023-11-29T08:07:00.944+0000 7f2e71102700  2 req 12755686132095393521 0.009000235s iam:tag_role recalculating target
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -35> 2023-11-29T08:07:00.944+0000 7f2e71102700  2 req 12755686132095393521 0.009000235s iam:tag_role reading permissions
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -34> 2023-11-29T08:07:00.944+0000 7f2e71102700  2 req 12755686132095393521 0.009000235s iam:tag_role init op
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -33> 2023-11-29T08:07:00.944+0000 7f2e71102700  2 req 12755686132095393521 0.009000235s iam:tag_role verifying op mask
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -32> 2023-11-29T08:07:00.944+0000 7f2e71102700 20 req 12755686132095393521 0.009000235s iam:tag_role required_mask= 0 user.op_mask=7
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -31> 2023-11-29T08:07:00.944+0000 7f2e71102700  2 req 12755686132095393521 0.009000235s iam:tag_role verifying op permissions
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -30> 2023-11-29T08:07:00.944+0000 7f2e71102700 20 req 12755686132095393521 0.009000235s get_system_obj_state: rctx=0x7f2ebad35170 obj=zone1.rgw.meta:roles:role_names.rgwabac-department state=0x5577b6858e20 s->prefetch_data=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -29> 2023-11-29T08:07:00.944+0000 7f2e71102700 10 req 12755686132095393521 0.009000235s cache get: name=zone1.rgw.meta+roles+role_names.rgwabac-department : miss
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -28> 2023-11-29T08:07:00.945+0000 7f2e70901700 10 req 12755686132095393521 0.010000262s cache put: name=zone1.rgw.meta+roles+role_names.rgwabac-department info.flags=0x6
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -27> 2023-11-29T08:07:00.945+0000 7f2e70901700 10 req 12755686132095393521 0.010000262s adding zone1.rgw.meta+roles+role_names.rgwabac-department to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -26> 2023-11-29T08:07:00.945+0000 7f2e70901700 20 req 12755686132095393521 0.010000262s get_system_obj_state: s->obj_tag was set empty
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -25> 2023-11-29T08:07:00.945+0000 7f2e70901700 10 req 12755686132095393521 0.010000262s cache get: name=zone1.rgw.meta+roles+role_names.rgwabac-department : type miss (requested=0x1, cached=0x6)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -24> 2023-11-29T08:07:00.945+0000 7f2e70901700 20 req 12755686132095393521 0.010000262s rados->read ofs=0 len=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -23> 2023-11-29T08:07:00.946+0000 7f2e70100700 20 req 12755686132095393521 0.011000288s rados_obj.operate() r=0 bl.length=46
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -22> 2023-11-29T08:07:00.946+0000 7f2e70100700 10 req 12755686132095393521 0.011000288s cache put: name=zone1.rgw.meta+roles+role_names.rgwabac-department info.flags=0x1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -21> 2023-11-29T08:07:00.946+0000 7f2e70100700 10 req 12755686132095393521 0.011000288s moving zone1.rgw.meta+roles+role_names.rgwabac-department to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -20> 2023-11-29T08:07:00.946+0000 7f2e70100700 20 req 12755686132095393521 0.011000288s get_system_obj_state: rctx=0x7f2ebad35170 obj=zone1.rgw.meta:roles:roles.e71a04d2-9233-4082-856c-412ccc668397 state=0x5577b68594e0 s->prefetch_data=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -19> 2023-11-29T08:07:00.946+0000 7f2e70100700 10 req 12755686132095393521 0.011000288s cache get: name=zone1.rgw.meta+roles+roles.e71a04d2-9233-4082-856c-412ccc668397 : miss
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -18> 2023-11-29T08:07:00.946+0000 7f2e70100700 20 req 12755686132095393521 0.011000288s WARNING: blocking librados call
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -17> 2023-11-29T08:07:00.947+0000 7f2e70100700 10 req 12755686132095393521 0.012000314s cache put: name=zone1.rgw.meta+roles+roles.e71a04d2-9233-4082-856c-412ccc668397 info.flags=0x6
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -16> 2023-11-29T08:07:00.947+0000 7f2e70100700 10 req 12755686132095393521 0.012000314s adding zone1.rgw.meta+roles+roles.e71a04d2-9233-4082-856c-412ccc668397 to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -15> 2023-11-29T08:07:00.947+0000 7f2e70100700 20 req 12755686132095393521 0.012000314s get_system_obj_state: s->obj_tag was set empty
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -14> 2023-11-29T08:07:00.947+0000 7f2e70100700 20 req 12755686132095393521 0.012000314s Read xattr: user.rgw.idtag
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -13> 2023-11-29T08:07:00.947+0000 7f2e70100700 10 req 12755686132095393521 0.012000314s cache get: name=zone1.rgw.meta+roles+roles.e71a04d2-9233-4082-856c-412ccc668397 : type miss (requested=0x3, cached=0x6)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -12> 2023-11-29T08:07:00.947+0000 7f2e70100700 20 req 12755686132095393521 0.012000314s rados->read ofs=0 len=0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -11> 2023-11-29T08:07:00.947+0000 7f2e70100700 20 req 12755686132095393521 0.012000314s WARNING: blocking librados call
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug    -10> 2023-11-29T08:07:00.947+0000 7f2e70100700 20 req 12755686132095393521 0.012000314s rados_obj.operate() r=0 bl.length=465
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -9> 2023-11-29T08:07:00.948+0000 7f2e70100700 10 req 12755686132095393521 0.013000340s cache put: name=zone1.rgw.meta+roles+roles.e71a04d2-9233-4082-856c-412ccc668397 info.flags=0x3
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -8> 2023-11-29T08:07:00.948+0000 7f2e70100700 10 req 12755686132095393521 0.013000340s moving zone1.rgw.meta+roles+roles.e71a04d2-9233-4082-856c-412ccc668397 to cache LRU end
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -7> 2023-11-29T08:07:00.948+0000 7f2e70100700  2 overriding permissions due to admin operation
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -6> 2023-11-29T08:07:00.948+0000 7f2e70100700  2 req 12755686132095393521 0.013000340s iam:tag_role verifying op params
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -5> 2023-11-29T08:07:00.948+0000 7f2e70100700  2 req 12755686132095393521 0.013000340s iam:tag_role pre-executing
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -4> 2023-11-29T08:07:00.948+0000 7f2e70100700  2 req 12755686132095393521 0.013000340s iam:tag_role executing
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -3> 2023-11-29T08:07:00.948+0000 7f2e70100700 20 Key index: 1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -2> 2023-11-29T08:07:00.948+0000 7f2e70100700 20 Value index: 1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug     -1> 2023-11-29T08:07:00.948+0000 7f2e70100700  0 Tag Key: Department Tag Value is: Engineering
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: debug      0> 2023-11-29T08:07:00.955+0000 7f2e70100700 -1 *** Caught signal (Segmentation fault) **
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  in thread 7f2e70100700 thread_name:radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  ceph version 16.2.10-187.el8cp (5d6355e2bccd18b5c6457a34cb666d773f21823d) pacific (stable)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f2eaf37ecf0]
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  2: (RGWRole::update(DoutPrefixProvider const*, optional_yield)+0x34) [0x7f2eba431414]
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  3: (RGWTagRole::execute(optional_yield)+0x61) [0x7f2eba607dc1]
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  4: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0xb36) [0x7f2eb9f1f5e6]
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  5: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x28b1) [0x7f2eb9f234a1]
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  6: /lib64/libradosgw.so.2(+0x45cbb0) [0x7f2eb9ea4bb0]
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  7: make_fcontext()
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: 
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: --- logging levels ---
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 none
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 lockdep
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 context
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 1 crush
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mds
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mds_balancer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mds_locker
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mds_log
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mds_log_expire
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mds_migrator
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 buffer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 timer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 filer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 striper
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 1 objecter
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 rados
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:  
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   0/ 5 rbd
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 rbd_mirror
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 rbd_replay
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 rbd_pwl
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 journaler
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 objectcacher
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 immutable_obj_cache
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 client
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 osd
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 optracker
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 objclass
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 3 filestore
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 3 journal
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 0 ms
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mon
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/10 monc
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 paxos
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 tp
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 auth
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 crypto
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 1 finisher
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 1 reserver
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 heartbeatmap
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 perfcounter
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   20/20 rgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 rgw_sync
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/10 civetweb
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 rgw_access
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 javaclient
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 asok
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 1 throttle
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 0 refs
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 compressor
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 bluestore
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 bluefs
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 3 bdev
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 kstore
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    4/ 5 rocksdb
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    4/ 5 leveldb
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    4/ 5 memdb
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 fuse
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    2/ 5 mgr
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 mgrc
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 dpdk
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 eventtrace
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    1/ 5 prioritycache
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 test
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 cephfs_mirror
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:    0/ 5 cephsqlite
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   -2/-2 (syslog threshold)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   99/99 (stderr threshold)
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: --- pthread ID / name mapping for recent threads ---
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139833237362432 / safe_timer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139833254147840 / ms_dispatch
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139833262540544 / ceph_timer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139833279325952 / io_context_pool
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837425321728 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837433714432 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837442107136 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837450499840 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837458892544 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837467285248 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837475677952 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837484070656 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837526034176 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837609961216 / notif-worker0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837626746624 / rgw_reshard
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837635139328 / rgw_user_st_syn
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837643532032 / rgw_buck_st_syn
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837651924736 / lifecycle_thr_2
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837685495552 / lifecycle_thr_1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837719066368 / lifecycle_thr_0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837752637184 / sync-log-trim
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837761029888 / http_manager
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837811386112 / http_manager
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837828171520 / rgw_obj_expirer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837836564224 / rgw_gc
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837853349632 / safe_timer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837870135040 / ms_dispatch
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837895313152 / io_context_pool
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139837903705856 / rgw_dt_lg_renew
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838189057792 / safe_timer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838205843200 / ms_dispatch
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838214235904 / ceph_timer
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838222628608 / io_context_pool
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838231021312 / io_context_pool
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838256199424 / kmip worker
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838264592128 / http_manager
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838289770240 / admin_socket
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838298162944 / service
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838306555648 / msgr-worker-2
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838314948352 / msgr-worker-1
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838323341056 / msgr-worker-0
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   139838679946688 / radosgw
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   max_recent     10000
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   max_new        10000
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]:   log_file /var/lib/ceph/crash/2023-11-29T08:07:00.955029Z_6fb7bef6-db03-4ecf-bbef-34ec28f367fb/log
Nov 29 11:07:01 cephproxy1.stg.local conmon[2293721]: --- end dump of recent events ---
Nov 29 11:07:01 cephproxy1.stg.local systemd-coredump[2295646]: Resource limits disable core dumping for process 2293746 (radosgw).
Nov 29 11:07:01 cephproxy1.stg.local systemd-coredump[2295646]: Process 2293746 (radosgw) of user 167 dumped core.
Nov 29 11:07:01 cephproxy1.stg.local systemd[1]: systemd-coredump: Succeeded.

--- Additional comment from daniel parkes on 2023-12-07 12:31:20 UTC ---

Hi, 

I did the same test with IBM Storage Ceph 7.0, and also saw the same crash.

Regards.

--- Additional comment from Matt Benjamin (redhat) on 2024-01-18 16:01:22 UTC ---

Hi Shreyansh,

I'm not sure there's time to take any change for 7.0z1, frankly, but we can try.  A fix to prevent crashing would be valuable..  Please provide feedback on what you have at our next engineering meeting.

thanks!

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-01-18 16:02:00 UTC ---

(looks like I should have known about this one--please bring to the downstream meeting any issues reaching your team)

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-01-24 12:26:36 UTC ---

Hi Pritha,

Can you offer any suggestions for debugging?

thanks,

Matt

--- Additional comment from Shreyansh Sancheti on 2024-01-24 13:22:55 UTC ---

I have tested the reproducer steps after adding the capabilities to the user.

./bin/radosgw-admin caps add --uid=testid --caps="roles=*"

The upstream and downstream both works fine, its only after we have the roles added but removed the caps for the same user we see "access denied" in the debug messages which is expected.

In @daniel 's cluster under these same conditions rgws crashed (and restarted on their own) saying "Could not connect to the endpoint URL". 

After attaching the pid to gdb I have come to the following analysis:

This is the function where the crash occurred. The function RGWRole::update is part of the Ceph RADOS Gateway code, and the crash happened at line 250 in the rgw_role.cc file.
  
   auto& pool = ctl->svc->zone->get_zone_params().roles_pool;

#3 0x00007fd253e87511 in RGWTagRole::execute (this=0x561f98ac6e00, y=...) at /usr/src/debug/ceph-16.2.10-208.el8cp.x86_64/src/rgw/rgw_rest_role.cc:588 
The crash seems to be related to the execution of the RGWTagRole::execute function
6:04
(gdb) p ctl
$2 = (RGWCtl *) 0x0

In this case, ctl is a null pointer, and attempting to dereference it leads to undefined behavior. The same is fixed in between pacific to quincy builds.
@daniel can you try in a quincy stable version, maybe?

--- Additional comment from Casey Bodley on 2024-01-24 15:59:11 UTC ---

hey Shreyansh,

the problem with users created with the --admin flag is how they interact with the RGWOp::verify_permission() interface:

https://github.com/ceph/ceph/blob/f4758e5/src/rgw/rgw_process.cc#L221-L236

you can see there that --system and --admin users just ignore any errors returned by verify_permission() and go on to call execute() anyway

looking at RGWTagRole, that inherits RGWRestRole::verify_permission() from its base class. that function is responsible for loading the RGWRole from storage and initializing the RGWRestRole::_role member variable

but any error it encounters while doing that (like -ENOENT for a RoleName that doesn't exist, or -EPERM when the user doesn't have "roles" caps) would leave RGWRestRole::_role uninitialized. that's why RGWTagRole::execute() is crashing on a null pointer

to fix this, we need to fail in a way that system/admin users can't ignore. the approach that i've been using elsewhere is to separate the initialization logic from the permission-checking logic. looking at the rgw_process.cc code linked above, we call op->init_processing() just before op->verify_permission(), and errors from init_processing() don't get overridden for system/admin users

so you could add a RGWRestRole::init_processing() override that does just the `_role` initialization part. does that make sense?

just note that some ops (RGWCreateRole, RGWGetRole, RGWListRoles) provide their own verify_permission() to avoid calling it on the base class - so those ops should also avoid calling the base RGWRestRole::init_processing(). for example, with an override like:

int RGWCreateRole::init_processing(optional_yield y)
{
  return 0; // don't call RGWRestRole::init_processing()
}

--- Additional comment from Vidushi Mishra on 2024-01-29 09:27:43 UTC ---

Hi Shreyansh, 

I am observing the segfault on ceph version 17.2.6-189.el9cp.

steps.
======

1. Create a rgw user 
radosgw-admin user create --uid user2 --access-key 1234 --secret 1234 --display-name user2

2. Add role capabilities
radosgw-admin caps add --uid user2 --caps="roles=*"

3. Add admin flag to the user
radosgw-admin user modify  --uid user2 --access-key 1234 --secret 1234 --display-name user2 --admin

4. Assign an attribute to IAM role.
[root@depressa013 ~]# aws --no-verify-ssl --endpoint  http://extensa027:8080 iam tag-role --role-name rgwabac-department --tags Key=Department,Value=Engineering

Could not connect to the endpoint URL: "http://extensa027:8080/"

Thanks,
Vidushi

--- Additional comment from Vidushi Mishra on 2024-01-29 09:29:12 UTC ---



--- Additional comment from Shreyansh Sancheti on 2024-02-14 06:42:32 UTC ---

(In reply to Vidushi Mishra from comment #7)
> Hi Shreyansh, 
> 
> I am observing the segfault on ceph version 17.2.6-189.el9cp.
> 
> steps.
> ======
> 
> 1. Create a rgw user 
> radosgw-admin user create --uid user2 --access-key 1234 --secret 1234
> --display-name user2
> 
> 2. Add role capabilities
> radosgw-admin caps add --uid user2 --caps="roles=*"
> 
> 3. Add admin flag to the user
> radosgw-admin user modify  --uid user2 --access-key 1234 --secret 1234
> --display-name user2 --admin
> 
> 4. Assign an attribute to IAM role.
> [root@depressa013 ~]# aws --no-verify-ssl --endpoint  http://extensa027:8080
> iam tag-role --role-name rgwabac-department --tags
> Key=Department,Value=Engineering
> 
> Could not connect to the endpoint URL: "http://extensa027:8080/"
> 
> Thanks,
> Vidushi

The fix has been merged in upstream and downstream target branches.

--- Additional comment from  on 2024-02-15 06:42:21 UTC ---

This is downstream in ceph-7.1-rhel-patches (git tag: ceph-18.2.1-12), but not ceph-7.0-rhel-patches.

Confirm that we're waiting for 7.0z2 before cherry-picking to ceph-7.0-rhel-patches? (7.0z1 isn't GA yet but close.)

Thanks,

Thomas
Comment 8 errata-xmlrpc 2024-05-07 12:09:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:2743
Comment 9 Red Hat Bugzilla 2024-09-07 04:25:17 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days