Bug 2297267

Description of problem (please be detailed as possible and provide log
snippests):
==================================================================================
Ceph mon (mon.b) aborted in thread_name:msgr-worker-1 while performing repeated OCP worker machine config pool reboots.

sh-5.1$ ceph crash ls
h cID                                                                ENTITY                NEW  
2024-07-11T05:01:39.337305Z_bc04869e-9361-4a42-886b-0206f18e1d23  mon.b                  *   
Log snip:
==========

    -8> 2024-07-11T05:01:39.288+0000 7f77c3992900  5 AuthRegistry(0x55b46d9cee20) adding con mode: secure
    -7> 2024-07-11T05:01:39.288+0000 7f77c3992900  5 AuthRegistry(0x55b46d9cee20) adding con mode: secure
    -6> 2024-07-11T05:01:39.288+0000 7f77c3992900  2 auth: KeyRing::load: loaded key file /etc/ceph/keyring-store/keyring
    -5> 2024-07-11T05:01:39.288+0000 7f77c3992900  2 mon.b@-1(???) e3 init
    -4> 2024-07-11T05:01:39.290+0000 7f77c3992900  4 mgrc handle_mgr_map Got map version 928
    -3> 2024-07-11T05:01:39.291+0000 7f77c3992900  4 mgrc handle_mgr_map Active mgr is now [v2:10.131.0.21:6800/3571147502,v1:10.131.0.21:6801/3571147502]
    -2> 2024-07-11T05:01:39.291+0000 7f77c3992900  4 mgrc reconnect Starting new session with [v2:10.131.0.21:6800/3571147502,v1:10.131.0.21:6801/3571147502]
    -1> 2024-07-11T05:01:39.315+0000 7f77c3992900  0 mon.b@-1(probing) e3  my rank is now 1 (was -1)
     0> 2024-07-11T05:01:39.339+0000 7f77baf63640 -1 *** Caught signal (Aborted) **
 in thread 7f77baf63640 thread_name:msgr-worker-1

 ceph version 17.2.6-216.0.hotfix.bz2266538.el9cp (e3968f91dc6b6b52eea5a64d169887c551d0d99c) quincy (stable)
 1: /lib64/libc.so.6(+0x3e6f0) [0x7f77c3f336f0]
 2: /lib64/libc.so.6(+0x8b94c) [0x7f77c3f8094c]
 3: raise()
 4: abort()
 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f77c4295b21]
 6: /lib64/libstdc++.so.6(+0xad52c) [0x7f77c42a152c]
 7: /lib64/libstdc++.so.6(+0xad597) [0x7f77c42a1597]
 8: /lib64/libstdc++.so.6(+0xad7f9) [0x7f77c42a17f9]
 9: /usr/lib64/ceph/libceph-common.so.2(+0x137e4b) [0x7f77c4813e4b]
 10: (ProtocolV2::handle_auth_done(ceph::buffer::v15_2_0::list&)+0x613) [0x7f77c4acdb83]
 11: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x39) [0x7f77c4ab97a9]
 12: (AsyncConnection::process()+0x42b) [0x7f77c4a99e7b]
 13: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1c1) [0x7f77c4ae1401]
 14: /usr/lib64/ceph/libceph-common.so.2(+0x405eb6) [0x7f77c4ae1eb6]
 15: /lib64/libstdc++.so.6(+0xdbad4) [0x7f77c42cfad4]
 16: /lib64/libc.so.6(+0x89c02) [0x7f77c3f7ec02]
 17: /lib64/libc.so.6(+0x10ec40) [0x7f77c4003c40]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_pwl
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 immutable_obj_cache
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/ 5 rgw_datacache
   1/10 civetweb
   1/ 5 rgw_access
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 fuse
   2/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   1/ 5 prioritycache
   0/ 5 test
   0/ 5 cephfs_mirror
   0/ 5 cephsqlite
   0/ 5 seastore
   0/ 5 seastore_onode
   0/ 5 seastore_odata
   0/ 5 seastore_omap
   0/ 5 seastore_tm
   0/ 5 seastore_cleaner
   0/ 5 seastore_lba
   0/ 5 seastore_cache
   0/ 5 seastore_journal
   0/ 5 seastore_device
   0/ 5 alienstore
   1/ 5 mclock
   1/ 5 ceph_exporter
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  7f77b9760640 / ceph-mon
  7f77baf63640 / msgr-worker-1
  7f77c2930640 / admin_socket
  7f77c3992900 / ceph-mon
  max_recent     10000
  max_new        10000

Version of all relevant components (if applicable):
OCP: 4.14.31
ODF: 4.14.9
Ceph: 17.2.6_216.0.hotfix.bz2266538


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
The ceph health went to warning state due to this mon crash which can be made healthy by archiving the crash.


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
3


Can this issue reproducible?
Reporting upon the first occurrence. 


Can this issue reproduce from the UI?
N/A


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
===================
1) Deploy a OCP + ODF cluster
2) Reboot OCP worker machine config pool
3) Wait for the machine config pool worker to start updating
4) wait for the machine config pool worker to stop updating

Repeat the above steps many times.

Actual results:
===============
mon.b aborted during the 71st iteration.

Expected results:
=================
No crashes should be observed.