Description of problem: +++ This bug was initially created as a clone of Bug #2249003 +++ Backport is required for Quincy. While executing the test case we observed the crashes. [ceph: root@ceph-rados-hotfix-4ht6oa-node1-installer crash]# rados clearomap -p re_pool_crash omap_obj_221489_4 error clearing omap keys re_pool_crash//usr/include/c++/11/optional:439: constexpr _Tp& std::_Optional_base_impl<_Tp, _Dp>::_M_get() [with _Tp = std::__cxx11::basic_string<char>; _Dp = std::_Optional_base<std::__cxx11::basic_string<char>, false, false>]: Assertion 'this->_M_is_engaged()' failed. *** Caught signal (Aborted) ** in thread 7fca05342e00 thread_name:rados ceph version 17.2.6-216.0.hotfix.bz2266538.el9cp (e3968f91dc6b6b52eea5a64d169887c551d0d99c) quincy (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca0594c6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fca0599994c] 3: raise() 4: abort() 5: rados(+0x384c8) [0x55747187e4c8] 6: rados(+0x433f9) [0x5574718893f9] 7: main() 8: /lib64/libc.so.6(+0x29590) [0x7fca05937590] 9: __libc_start_main() 10: _start() 2024-07-15T13:40:00.274+0000 7fca05342e00 -1 *** Caught signal (Aborted) ** in thread 7fca05342e00 thread_name:rados ceph version 17.2.6-216.0.hotfix.bz2266538.el9cp (e3968f91dc6b6b52eea5a64d169887c551d0d99c) quincy (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca0594c6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fca0599994c] 3: raise() 4: abort() 5: rados(+0x384c8) [0x55747187e4c8] 6: rados(+0x433f9) [0x5574718893f9] 7: main() 8: /lib64/libc.so.6(+0x29590) [0x7fca05937590] 9: __libc_start_main() 10: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -273> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command assert hook 0x55747280b110 -272> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command abort hook 0x55747280b110 -271> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command leak_some_memory hook 0x55747280b110 -270> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perfcounters_dump hook 0x55747280b110 -269> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command 1 hook 0x55747280b110 -268> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perf dump hook 0x55747280b110 -267> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perfcounters_schema hook 0x55747280b110 -266> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perf histogram dump hook 0x55747280b110 -265> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command 2 hook 0x55747280b110 -264> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perf schema hook 0x55747280b110 -263> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command counter dump hook 0x55747280b110 -262> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command counter schema hook 0x55747280b110 -261> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perf histogram schema hook 0x55747280b110 -260> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command perf reset hook 0x55747280b110 -259> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config show hook 0x55747280b110 -258> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config help hook 0x55747280b110 -257> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config set hook 0x55747280b110 -256> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config unset hook 0x55747280b110 -255> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config get hook 0x55747280b110 -254> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config diff hook 0x55747280b110 -253> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command config diff get hook 0x55747280b110 -252> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command injectargs hook 0x55747280b110 -251> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command log flush hook 0x55747280b110 -250> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command log dump hook 0x55747280b110 -249> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command log reopen hook 0x55747280b110 -248> 2024-07-15T13:40:00.232+0000 7fca05342e00 5 asok(0x55747284a380) register_command dump_mempools hook 0x5574728b3b88 -247> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: get_monmap_and_config -246> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: build_initial_monmap -245> 2024-07-15T13:40:00.238+0000 7fca05342e00 1 build_initial for_mkfs: 0 -244> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: monmap: epoch 0 fsid 6028e958-4036-11ef-922d-fa163ef53815 last_changed 2024-07-15T13:40:00.239117+0000 created 2024-07-15T13:40:00.239117+0000 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.noname-b 1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.noname-c 2: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.noname-a -243> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding auth protocol: cephx -242> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding auth protocol: cephx -241> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding auth protocol: cephx -240> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding auth protocol: none -239> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: secure -238> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: crc -237> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: secure -236> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: crc -235> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: secure -234> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: crc -233> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: crc -232> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: secure -231> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: crc -230> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: secure -229> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: crc -228> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x5574728b6cf0) adding con mode: secure -227> 2024-07-15T13:40:00.238+0000 7fca05342e00 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring -226> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: init -225> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: cephx -224> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: cephx -223> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: cephx -222> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding auth protocol: none -221> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure -220> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc -219> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure -218> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc -217> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure -216> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc -215> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc -214> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure -213> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc -212> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure -211> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: crc -210> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 AuthRegistry(0x7ffd8a8ec8a0) adding con mode: secure -209> 2024-07-15T13:40:00.238+0000 7fca05342e00 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring -208> 2024-07-15T13:40:00.238+0000 7fca05342e00 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring -207> 2024-07-15T13:40:00.238+0000 7fca05342e00 5 asok(0x55747284a380) register_command rotate-key hook 0x7ffd8a8ec9e8 -206> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: _reopen_session rank -1 -205> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient: _add_conns ranks=[1,2,0] -204> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): picked mon.noname-c con 0x557472951470 addr [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] -203> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): picked mon.noname-a con 0x557472951d90 addr [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] -202> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): picked mon.noname-b con 0x55747294cc90 addr [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] -201> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): start opening mon connection -200> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): start opening mon connection -199> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): start opening mon connection -198> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): _renew_subs -197> 2024-07-15T13:40:00.238+0000 7fca05342e00 10 monclient(hunting): authenticate will time out at 272554.292452s -196> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): get_auth_request con 0x557472951d90 auth_method 0 -195> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] -194> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): _init_auth method 2 -193> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): _init_auth creating new auth -192> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 -191> 2024-07-15T13:40:00.239+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 -190> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -189> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_done global_id 136158 payload 274 -188> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient: _finish_hunting 0 -187> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 1 monclient: found mon.noname-a -186> 2024-07-15T13:40:00.240+0000 7fc9fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.0.210.10:3300/0 -185> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: handle_monmap mon_map magic: 0 v1 -184> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: got monmap 3 from mon.noname-a (according to old e3) -183> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: dump: epoch 3 fsid 6028e958-4036-11ef-922d-fa163ef53815 last_changed 2024-07-12T10:11:44.956959+0000 created 2024-07-12T10:06:34.011282+0000 min_mon_release 17 (quincy) election_strategy: 1 0: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.ceph-rados-hotfix-4ht6oa-node1-installer 1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.ceph-rados-hotfix-4ht6oa-node6 2: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.ceph-rados-hotfix-4ht6oa-node2 -182> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: _finish_auth 0 -181> 2024-07-15T13:40:00.242+0000 7fca05342e00 5 monclient: authenticate success, global_id 136158 -180> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: handle_config config(3 keys) v1 -179> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 monclient: get_monmap_and_config success -178> 2024-07-15T13:40:00.242+0000 7fca05342e00 4 set_mon_vals no callback set -177> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 set_mon_vals container_image = registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:292a103ce9f694f7773d3dc68d9ad05bdd2509767e294ac997d3f75f7f06dfd1 -176> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: handle_monmap mon_map magic: 0 v1 -175> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 set_mon_vals log_to_file = true -174> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: got monmap 3 from mon.ceph-rados-hotfix-4ht6oa-node1-installer (according to old e3) -173> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 set_mon_vals mon_cluster_log_to_file = true -172> 2024-07-15T13:40:00.242+0000 7fca02643640 10 monclient: dump: epoch 3 fsid 6028e958-4036-11ef-922d-fa163ef53815 last_changed 2024-07-12T10:11:44.956959+0000 created 2024-07-12T10:06:34.011282+0000 min_mon_release 17 (quincy) election_strategy: 1 0: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.ceph-rados-hotfix-4ht6oa-node1-installer 1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.ceph-rados-hotfix-4ht6oa-node6 2: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.ceph-rados-hotfix-4ht6oa-node2 -171> 2024-07-15T13:40:00.242+0000 7fca05342e00 10 monclient: shutdown -170> 2024-07-15T13:40:00.242+0000 7fca05342e00 5 asok(0x55747284a380) unregister_commands rotate-key -169> 2024-07-15T13:40:00.243+0000 7fca05342e00 10 monclient: get_monmap_and_config -168> 2024-07-15T13:40:00.243+0000 7fca05342e00 10 monclient: build_initial_monmap -167> 2024-07-15T13:40:00.243+0000 7fca05342e00 1 build_initial for_mkfs: 0 -166> 2024-07-15T13:40:00.243+0000 7fca05342e00 10 monclient: monmap: epoch 0 fsid 00000000-0000-0000-0000-000000000000 last_changed 0.000000 created 0.000000 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.0.208.12:3300/0,v1:10.0.208.12:6789/0] mon.noname-b 1: [v2:10.0.208.85:3300/0,v1:10.0.208.85:6789/0] mon.noname-c 2: [v2:10.0.210.10:3300/0,v1:10.0.210.10:6789/0] mon.noname-a -165> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding auth protocol: cephx -164> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding auth protocol: cephx -163> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding auth protocol: cephx -162> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding auth protocol: none -161> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: secure -160> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: crc -159> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: secure -158> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: crc -157> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: secure -156> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: crc -155> 2024-07-15T13:40:00.243+0000 7fca05342e00 5 AuthRegistry(0x55747294e550) adding con mode: crc Automation test suite: - test: name: Omap creations on objects module: test_omap_entries.py polarion-id: CEPH-83571702 config: # Pool created to verify the Bug#2249003 crash_config: pool_name: re_pool_crash # EC pool config is commented out because omaps can be written only on RE pools omap_config: small_omap: pool_name: re_pool_small_omap pg_num: 1 pg_num_max: 1 large_warn: false obj_start: 0 obj_end: 5 normal_objs: 400 num_keys_obj: 200000 large_omap: pool_name: re_pool_large_omap pg_num: 1 pg_num_max: 1 large_warn: true obj_start: 0 obj_end: 5 normal_objs: 400 num_keys_obj: 200001 desc: Large number of omap creation on objects and OSD resiliency
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 6.1 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:4238