Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1661713

Summary: [support] OSD asserts in DBObjectMap.cc: 662: FAILED assert(state.legacy)
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Mike Hackett <mhackett>
Component: RADOSAssignee: Brad Hubbard <bhubbard>
Status: CLOSED WONTFIX QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1CC: bhubbard, ceph-eng-bugs, dzafman, jdurgin, kchai, mhackett, nojha, torkil, tpetr, vumrao
Target Milestone: rc   
Target Release: 3.*   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-08 15:52:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
OSD log none

Description Mike Hackett 2018-12-22 15:45:14 UTC 
Description of problem:

During leveldb compaction OSD asserts with the following:

2018-12-22 11:02:18.538671 7fe8c7e2b700  1 leveldb: Generated table #509857: 44245 keys, 2142602 bytes
2018-12-22 11:02:18.663360 7fe8c7e2b700  1 leveldb: Generated table #509858: 58140 keys, 2135933 bytes
2018-12-22 11:02:18.751228 7fe8c7e2b700  1 leveldb: Generated table #509859: 59005 keys, 2136061 bytes
2018-12-22 11:02:18.850778 7fe8c7e2b700  1 leveldb: Generated table #509860: 58279 keys, 2135710 bytes
2018-12-22 11:02:18.964761 7fe8c7e2b700  1 leveldb: Generated table #509861: 55658 keys, 2084112 bytes
2018-12-22 11:02:18.964771 7fe8c7e2b700  1 leveldb: Compacted 1@2 + 9@3 files => 19192019 bytes
2018-12-22 11:02:18.981652 7fe8c7e2b700  1 leveldb: compacted to: files[ 0 3 51 138 0 0 0 ]
2018-12-22 11:02:18.982213 7fe8c7e2b700  1 leveldb: Delete type=2 #509815

2018-12-22 11:02:19.006828 7fe8c7e2b700  1 leveldb: Delete type=2 #509816

2018-12-22 11:02:19.045234 7fe8c7e2b700  1 leveldb: Delete type=2 #509817

2018-12-22 11:02:19.045766 7fe8c7e2b700  1 leveldb: Delete type=2 #509838

2018-12-22 11:02:19.046293 7fe8c7e2b700  1 leveldb: Delete type=2 #507765

2018-12-22 11:02:19.048877 7fe8c7e2b700  1 leveldb: Delete type=2 #509818

2018-12-22 11:02:19.049309 7fe8c7e2b700  1 leveldb: Delete type=2 #507803

2018-12-22 11:02:19.059027 7fe8c7e2b700  1 leveldb: Delete type=2 #507804

2018-12-22 11:02:19.059440 7fe8c7e2b700  1 leveldb: Delete type=2 #507805

2018-12-22 11:02:19.060090 7fe8c7e2b700  1 leveldb: Delete type=2 #509814

2018-12-22 11:03:35.732313 7fe8e6e9f700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/filestore/DBObjectMap.cc: In function 'virtual int DBObjectMap::rm_keys(const ghobject_t&, const std::set<std::basic_string<char> >&, const SequencerPosition*)' thread 7fe8e6e9f700 time 2018-12-22 11:03:35.706007
/builddir/build/BUILD/ceph-12.2.5/src/os/filestore/DBObjectMap.cc: 662: FAILED assert(state.legacy)

 ceph version 12.2.5-42.el7cp (82d52d7efa6edec70f6a0fc306f40b89265535fb) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x5610b5b27880]
 2: (DBObjectMap::rm_keys(ghobject_t const&, std::set<std::string, std::less<std::string>, std::allocator<std::string> > const&, SequencerPosition const*)+0xcc1) [0x5610b5a6ca71]
 3: (FileStore::_omap_rmkeys(coll_t const&, ghobject_t const&, std::set<std::string, std::less<std::string>, std::allocator<std::string> > const&, SequencerPosition const&)+0xac) [0x5610b58e121c]
 4: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2185) [0x5610b590b585]
 5: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, unsigned long, ThreadPool::TPHandle*)+0x3b) [0x5610b590f0cb]
 6: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x3fa) [0x5610b590f4fa]
 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x5610b5b2e46e]
 8: (ThreadPool::WorkThread::entry()+0x10) [0x5610b5b2f350]
 9: (()+0x7dd5) [0x7fe8f8429dd5]
 10: (clone()+0x6d) [0x7fe8f751ab3d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


Need to confirm if OSD media is bad and we possibly have leveldb corruption due to this which leads to assertion when compacting (deleting).


Version-Release number of selected component (if applicable):
RHCS 3.1z1 
ceph version 12.2.5-59.el7cp

How reproducible:
In customer environment restart of the OSD and it crashes.

Upstream tracker with similar failure but no actions:

http://tracker.ceph.com/issues/34321
Comment 3 Mike Hackett 2018-12-22 18:20:04 UTC 
Created attachment 1516272 [details]
OSD log