Bug 1881985 - Mon crash observed at std::_Rb_tree_rebalance_for_erase
Summary: Mon crash observed at std::_Rb_tree_rebalance_for_erase
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RADOS
Version: 4.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.*
Assignee: Neha Ojha
QA Contact: Manohar Murthy
Depends On:
TreeView+ depends on / blocked
Reported: 2020-09-23 14:24 UTC by Tejas
Modified: 2020-12-07 05:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-12-07 05:34:26 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Tejas 2020-09-23 14:24:52 UTC
Description of problem:
Mon crashed on changing Cluster Ip subnet on the OSDs. The Cluster Ip was wrongly configured on the OSDs, so the ceph.conf on the MoN abd the OSDs were having different Cluster IP subnets.
On changing the Cluster IP subnet on the OSDs to match that on the Mon , and restarting all OSDs, the primary MON crashed.
No other MONs crashed.

Version-Release number of selected component (if applicable):
ceph version 14.2.8-50.el7cp

How reproducible:
observed only once

 ~]# ceph crash info 2020-09-23_06:19:50.288218Z_967aefe7-4aec-440e-9176-e0d6cb588206
    "os_version_id": "7.8", 
    "utsname_release": "3.10.0-1127.19.1.el7.x86_64", 
    "os_name": "Red Hat Enterprise Linux Server", 
    "entity_name": "mon.f03-h02-000-r620", 
    "timestamp": "2020-09-23 06:19:50.288218Z", 
    "process_name": "ceph-mon", 
    "utsname_machine": "x86_64", 
    "utsname_sysname": "Linux", 
    "os_version": "7.8 (Maipo)", 
    "os_id": "rhel", 
    "utsname_version": "#1 SMP Tue Aug 11 19:12:04 EDT 2020", 
    "backtrace": [
        "(()+0xf630) [0x7f05da68c630]", 
        "(std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x184) [0x7f05d9da55a4]", 
        "(std::_Rb_tree<int, std::pair<int const, failure_reporter_t>, std::_Select1st<std::pair<int const, failure_reporter_t> >, std::less<int>, std::allocator<std::pair<int const, failure_reporter_t> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<int const, failure_reporter_t> >)+0x15) [0x555967fc3a25]", 
        "(OSDMonitor::check_failure(utime_t, int, failure_info_t&)+0x951) [0x555967f51101]", 
        "(OSDMonitor::check_failures(utime_t)+0x61) [0x555967f516c1]", 
        "(OSDMonitor::tick()+0x1aa) [0x555967f7a3ea]", 
        "(Monitor::tick()+0x120) [0x555967e437e0]", 
        "(C_MonContext::finish(int)+0x39) [0x555967e1f3b9]", 
        "(Context::complete(int)+0x9) [0x555967e59e99]", 
        "(SafeTimer::timer_thread()+0x180) [0x7f05dd969290]", 
        "(SafeTimerThread::entry()+0xd) [0x7f05dd96aafd]", 
        "(()+0x7ea5) [0x7f05da684ea5]", 
        "(clone()+0x6d) [0x7f05d95488dd]"
    "utsname_hostname": "f03-h02-000-r620.rdu2.scalelab.redhat.com", 
    "crash_id": "2020-09-23_06:19:50.288218Z_967aefe7-4aec-440e-9176-e0d6cb588206", 
    "ceph_version": "14.2.8-50.el7cp"

Note You need to log in before you can comment on or make changes to this bug.