Bug 2297966

Summary: [Backport bug:2249003] - Observing client.admin crash in thread_name 'rados' on executing 'rados clearomap.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: skanta
Component: RADOSAssignee: Nitzan mordechai <nmordech>
Status: CLOSED ERRATA QA Contact: skanta
Severity: high Docs Contact: Rivka Pollack <rpollack>
Priority: high    
Version: 6.1CC: bhubbard, bkunal, ceph-eng-bugs, cephqe-warriors, dwalveka, ngangadh, nmordech, nojha, rpollack, rpollak, tserlin, vumrao
Target Milestone: ---Keywords: Automation
Target Release: 6.1z9   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-17.2.6-264.el9cp Doc Type: Bug Fix
Doc Text:
.Error messages now show the correct object name Previously, the error handling failed to correctly retrieve the object name, leading to a segmentation fault when the object was missing. With this fix, the code ensures that the object name is correctly displayed in the error messages.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2025-04-28 05:29:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description skanta 2024-07-15 15:53:39 UTC 
Description of problem:
+++ This bug was initially created as a clone of Bug #2249003  +++

Backport is required for Quincy.

While executing the test case we observed the crashes.

[ceph: root@ceph-rados-hotfix-4ht6oa-node1-installer crash]# rados clearomap -p re_pool_crash omap_obj_221489_4 
error clearing omap keys re_pool_crash//usr/include/c++/11/optional:439: constexpr _Tp& std::_Optional_base_impl<_Tp, _Dp>::_M_get() [with _Tp = std::__cxx11::basic_string<char>; _Dp = std::_Optional_base<std::__cxx11::basic_string<char>, false, false>]: Assertion 'this->_M_is_engaged()' failed.
*** Caught signal (Aborted) **
 in thread 7fca05342e00 thread_name:rados
 ceph version 17.2.6-216.0.hotfix.bz2266538.el9cp (e3968f91dc6b6b52eea5a64d169887c551d0d99c) quincy (stable)
 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca0594c6f0]
 2: /lib64/libc.so.6(+0x8b94c) [0x7fca0599994c]
 3: raise()
 4: abort()
 5: rados(+0x384c8) [0x55747187e4c8]
 6: rados(+0x433f9) [0x5574718893f9]
 7: main()
 8: /lib64/libc.so.6(+0x29590) [0x7fca05937590]
 9: __libc_start_main()
 10: _start()
2024-07-15T13:40:00.274+0000 7fca05342e00 -1 *** Caught signal (Aborted) **
 in thread 7fca05342e00 thread_name:rados

 ceph version 17.2.6-216.0.hotfix.bz2266538.el9cp (e3968f91dc6b6b52eea5a64d169887c551d0d99c) quincy (stable)
 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca0594c6f0]
 2: /lib64/libc.so.6(+0x8b94c) [0x7fca0599994c]
 3: raise()
 4: abort()
 5: rados(+0x384c8) [0x55747187e4c8]
 6: rados(+0x433f9) [0x5574718893f9]
 7: main()
 8: /lib64/libc.so.6(+0x29590) [0x7fca05937590]
 9: __libc_start_main()
 10: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


--- begin dump of recent events ---
  -273> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command assert hook 0x55747280b110

  -272> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command abort hook 0x55747280b110

  -271> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command leak_some_memory hook 0x55747280b110

  -270> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perfcounters_dump hook 0x55747280b110

  -269> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command 1 hook 0x55747280b110

  -268> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perf dump hook 0x55747280b110

  -267> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perfcounters_schema hook 0x55747280b110

  -266> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perf histogram dump hook 0x55747280b110

  -265> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command 2 hook 0x55747280b110

  -264> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perf schema hook 0x55747280b110

  -263> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command counter dump hook 0x55747280b110

  -262> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command counter schema hook 0x55747280b110

  -261> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perf histogram schema hook 0x55747280b110

  -260> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command perf reset hook 0x55747280b110

  -259> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config show hook 0x55747280b110

  -258> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config help hook 0x55747280b110

  -257> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config set hook 0x55747280b110

  -256> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config unset hook 0x55747280b110

  -255> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config get hook 0x55747280b110

  -254> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config diff hook 0x55747280b110

  -253> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command config diff get hook 0x55747280b110

  -252> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command injectargs hook 0x55747280b110

  -251> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command log flush hook 0x55747280b110

  -250> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command log dump hook 0x55747280b110

  -249> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command log reopen hook 0x55747280b110

  -248> 2024-07-15T13:40:00.232+0000 7fca05342e00  5 asok(0x55747284a380) register_command dump_mempools hook 0x5574728b3b88

  -247> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: get_monmap_and_config

  -246> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: build_initial_monmap

  -245> 2024-07-15T13:40:00.238+0000 7fca05342e00  1 build_initial for_mkfs: 0

  -244> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: monmap:
epoch 0
fsid 6028e958-4036-11ef-922d-fa163ef53815
last_changed 2024-07-15T13:40:00.239117+0000
created 2024-07-15T13:40:00.239117+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.noname-b
1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.noname-c
2: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.noname-a


  -243> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding auth protocol: cephx

  -242> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding auth protocol: cephx

  -241> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding auth protocol: cephx

  -240> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding auth protocol: none

  -239> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: secure

  -238> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: crc

  -237> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: secure

  -236> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: crc

  -235> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: secure

  -234> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: crc

  -233> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: crc

  -232> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: secure

  -231> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: crc

  -230> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: secure

  -229> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: crc

  -228> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x5574728b6cf0) adding con mode: secure

  -227> 2024-07-15T13:40:00.238+0000 7fca05342e00  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring

  -226> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: init

  -225> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: cephx

  -224> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: cephx

  -223> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: cephx

  -222> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: none

  -221> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure

  -220> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc

  -219> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure

  -218> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc

  -217> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure

  -216> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc

  -215> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc

  -214> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure

  -213> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc

  -212> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure

  -211> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc

  -210> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure

  -209> 2024-07-15T13:40:00.238+0000 7fca05342e00  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring

  -208> 2024-07-15T13:40:00.238+0000 7fca05342e00  2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring

  -207> 2024-07-15T13:40:00.238+0000 7fca05342e00  5 asok(0x55747284a380) register_command rotate-key hook 0x7ffd8a8ec9e8

  -206> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: _reopen_session rank -1

  -205> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: _add_conns ranks=[1,2,0]

  -204> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): picked mon.noname-c con 0x557472951470 addr [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0]

  -203> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): picked mon.noname-a con 0x557472951d90 addr [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0]

  -202> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): picked mon.noname-b con 0x55747294cc90 addr [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0]

  -201> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): start opening mon connection

  -200> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): start opening mon connection

  -199> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): start opening mon connection

  -198> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): _renew_subs

  -197> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): authenticate will time out at 272554.292452s

  -196> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): get_auth_request con 0x557472951d90 auth_method 0

  -195> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]

  -194> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): _init_auth method 2

  -193> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): _init_auth creating new auth

  -192> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9

  -191> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9

  -190> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes

  -189> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_done global_id 136158 payload 274

  -188> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient: _finish_hunting 0

  -187> 2024-07-15T13:40:00.240+0000 7fc9fbfff640  1 monclient: found mon.noname-a

  -186> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.0.210.10:3300/0

  -185> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: handle_monmap mon_map magic: 0 v1

  -184> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient:  got monmap 3 from mon.noname-a (according to old e3)

  -183> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: dump:
epoch 3
fsid 6028e958-4036-11ef-922d-fa163ef53815
last_changed 2024-07-12T10:11:44.956959+0000
created 2024-07-12T10:06:34.011282+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.ceph-rados-hotfix-4ht6oa-node1-installer
1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.ceph-rados-hotfix-4ht6oa-node6
2: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.ceph-rados-hotfix-4ht6oa-node2


  -182> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: _finish_auth 0

  -181> 2024-07-15T13:40:00.242+0000 7fca05342e00  5 monclient: authenticate success, global_id 136158

  -180> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: handle_config config(3 keys) v1

  -179> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 monclient: get_monmap_and_config success

  -178> 2024-07-15T13:40:00.242+0000 7fca05342e00  4 set_mon_vals no callback set

  -177> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 set_mon_vals container_image = registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:292a103ce9f694f7773d3dc68d9ad05bdd2509767e294ac997d3f75f7f06dfd1

  -176> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: handle_monmap mon_map magic: 0 v1

  -175> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 set_mon_vals log_to_file = true

  -174> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient:  got monmap 3 from mon.ceph-rados-hotfix-4ht6oa-node1-installer (according to old e3)

  -173> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 set_mon_vals mon_cluster_log_to_file = true

  -172> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: dump:
epoch 3
fsid 6028e958-4036-11ef-922d-fa163ef53815
last_changed 2024-07-12T10:11:44.956959+0000
created 2024-07-12T10:06:34.011282+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.ceph-rados-hotfix-4ht6oa-node1-installer
1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.ceph-rados-hotfix-4ht6oa-node6
2: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.ceph-rados-hotfix-4ht6oa-node2


  -171> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 monclient: shutdown

  -170> 2024-07-15T13:40:00.242+0000 7fca05342e00  5 asok(0x55747284a380) unregister_commands rotate-key

  -169> 2024-07-15T13:40:00.243+0000 7fca05342e00 10 monclient: get_monmap_and_config

  -168> 2024-07-15T13:40:00.243+0000 7fca05342e00 10 monclient: build_initial_monmap

  -167> 2024-07-15T13:40:00.243+0000 7fca05342e00  1 build_initial for_mkfs: 0

  -166> 2024-07-15T13:40:00.243+0000 7fca05342e00 10 monclient: monmap:
epoch 0
fsid 00000000-0000-0000-0000-000000000000
last_changed 0.000000
created 0.000000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.noname-b
1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.noname-c
2: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.noname-a


  -165> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding auth protocol: cephx

  -164> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding auth protocol: cephx

  -163> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding auth protocol: cephx

  -162> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding auth protocol: none

  -161> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: secure

  -160> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: crc

  -159> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: secure

  -158> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: crc

  -157> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: secure

  -156> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: crc

  -155> 2024-07-15T13:40:00.243+0000 7fca05342e00  5 AuthRegistry(0x55747294e550) adding con mode: crc


Automation test suite:

 - test:
      name: Omap creations on objects
      module: test_omap_entries.py
      polarion-id: CEPH-83571702
      config:
        # Pool created to  verify the Bug#2249003
        crash_config:
          pool_name: re_pool_crash
        # EC pool config is commented out because omaps can be written only on RE pools
        omap_config:
          small_omap:
            pool_name: re_pool_small_omap
            pg_num: 1
            pg_num_max: 1
            large_warn: false
            obj_start: 0
            obj_end: 5
            normal_objs: 400
            num_keys_obj: 200000
          large_omap:
            pool_name: re_pool_large_omap
            pg_num: 1
            pg_num_max: 1
            large_warn: true
            obj_start: 0
            obj_end: 5
            normal_objs: 400
            num_keys_obj: 200001
      desc: Large number of omap creation on objects and OSD resiliency
Comment 13 errata-xmlrpc 2025-04-28 05:29:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 6.1 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:4238