| Summary: | ceph-objectstore-tool crashing in basic operation | ||
|---|---|---|---|
| Product: | Red Hat Ceph Storage | Reporter: | shylesh <shmohan> |
| Component: | RADOS | Assignee: | Kefu Chai <kchai> |
| Status: | CLOSED ERRATA | QA Contact: | Ramakrishnan Periyasamy <rperiyas> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.3.3 | CC: | ceph-eng-bugs, dzafman, flucifre, hnallurv, jdurgin, kchai, kdreyer, uboppana |
| Target Milestone: | rc | Keywords: | Reopened |
| Target Release: | 3.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | RHEL: ceph-12.1.2-1.el7cp Ubuntu: ceph_12.1.2-2redhat1xenial | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-05 23:31:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
> /bin/ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1
we open(2) the journal directory successfully, but failed to read from it. because it's a directory.
int FileJournal::open(uint64_t fs_op_seq)
{
dout(2) << "open " << fn << " fsid " << fsid << " fs_op_seq " << fs_op_seq << dendl;
uint64_t next_seq = fs_op_seq + 1;
int err = _open(false);
if (err)
return err;
// assume writeable, unless...
read_pos = 0;
write_pos = get_top();
// read header?
err = read_header(&header);
if (err < 0)
return err;
and it returns EISDIR when reading in FileJournal::read_header().
int err = journal->open(op_seq);
if (err < 0) {
dout(3) << "journal_replay open failed with "
<< cpp_strerror(err) << dendl;
delete journal;
journal = 0;
return err;
}
so in the dtor of FileJournal the fd is not -1 (set by _open()),
~FileJournal() {
assert(fd == -1);
delete[] zero_buf;
}
that's why it panicked.
Kefu seems to think this is a fixable crash - https://github.com/ceph/ceph/pull/11146 basic ceph-objectstore-tool is not crashing but seeing some other problem. without passing <dir>/journal path command is failing for mount error. [root@host031 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-2/ --journal-path /var/lib/ceph/osd/qetest-2/ --op list 1000010dd74.00000000__head_018F0000__1 --cluster qetest dump Mount failed with '(22) Invalid argument' with <dir>/journal path command is hung [root@host031 ~]# sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-2/ --journal-path /var/lib/ceph/osd/qetest-2/journal --op list 1000010dd74.00000000__head_018F0000__1 --cluster qetest dump ^C [root@host031 ~]# ^C Not sure either i missed anything in above command. what is "dump" for? and <object> should be an object name. are you sure it is "1000010dd74.00000000__head_018F0000__1"? if yes, how/where did you get ""1000010dd74.00000000__head_018F0000__1""? Yes it is a object name, I picked that from "/var/lib/ceph/osd/qetest-2/current/**_head/DIR_1/*" path. To clarify it, Created a object using "rados put" object name is 'obj1'. 'osd map' output is below. [ubuntu@host006 ~]$ sudo ceph osd map pool1 obj1 --cluster qetest osdmap e805 pool 'pool1' (3) object 'obj1' -> pg 3.6cf8deff (3.7) -> up ([1,7,6], p1) acting ([1,7,6], p1) Seeing same issue with and without 'dump' on 'obj1'. [root@host046 ubuntu]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-1/ --journal-path /var/lib/ceph/osd/qetest-1/ --op list obj1 dump --cluster qetest Mount failed with '(22) Invalid argument' [root@host046 ubuntu]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-1/ --journal-path /var/lib/ceph/osd/qetest-1/journal --op list obj1 dump --cluster qetest [root@host046 ubuntu]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-1/ --journal-path /var/lib/ceph/osd/qetest-1/ --op list obj1 --cluster qetest Mount failed with '(22) Invalid argument' [root@host046 ubuntu]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-1/ --journal-path /var/lib/ceph/osd/qetest-1/journal --op list obj1 --cluster qetest (In reply to Ramakrishnan Periyasamy from comment #9) > Yes it is a object name, I picked that from > "/var/lib/ceph/osd/qetest-2/current/**_head/DIR_1/*" path. > > To clarify it, Created a object using "rados put" object name is 'obj1'. > 'osd map' output is below. > > [ubuntu@host006 ~]$ sudo ceph osd map pool1 obj1 --cluster qetest > osdmap e805 pool 'pool1' (3) object 'obj1' -> pg 3.6cf8deff (3.7) -> up > ([1,7,6], p1) acting ([1,7,6], p1) > > Seeing same issue with and without 'dump' on 'obj1'. > > [root@host046 ubuntu]# ceph-objectstore-tool --data-path > /var/lib/ceph/osd/qetest-1/ --journal-path /var/lib/ceph/osd/qetest-1/ --op > list obj1 dump --cluster qetest > Mount failed with '(22) Invalid argument' > [root@host046 ubuntu]# ceph-objectstore-tool --data-path > /var/lib/ceph/osd/qetest-1/ --journal-path > /var/lib/ceph/osd/qetest-1/journal --op list obj1 dump --cluster qetest > > [root@host046 ubuntu]# ceph-objectstore-tool --data-path > /var/lib/ceph/osd/qetest-1/ --journal-path /var/lib/ceph/osd/qetest-1/ --op > list obj1 --cluster qetest > Mount failed with '(22) Invalid argument' > [root@host046 ubuntu]# ceph-objectstore-tool --data-path > /var/lib/ceph/osd/qetest-1/ --journal-path > /var/lib/ceph/osd/qetest-1/journal --op list obj1 --cluster qetest The --journal-path is invalid so "Mount failed." Usually you don't have to even specify --journal-path. The "ceph-objectstore-tool ... --op list obj1 dump" is not a valid command. Use --op list to get JSON representation of the object. If there are multiple objects with the same name in different pools or namespaces, you'll have to use the JSON. Otherwise, you can use the name only. $ ceph-objectstore-tool --data-path dev/osd0 --pgid 1.6 --op list foo ["1.6",{"oid":"foo","key":"","snapid":-2,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}] $ ceph-objectstore-tool --data-path dev/osd0 '["1.6",{"oid":"foo","key":"","snapid":-2,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' dump { "id": { "oid": "foo", ... You can specify the object name only if only one object with that name exists. On a cluster with large number of PGs, this command can take a long time if the --pgid is not specified. $ ceph-objectstore-tool --data-path dev/osd0 foo dump { "id": { "oid": "foo", ... $ ceph-objectstore-tool --data-path dev/osd0 another_obj dump Found 2 objects with id 'another_obj', please use a JSON spec from --op list instead $ ceph-objectstore-tool --data-path dev/osd0 --op list another_obj ["1.7",{"oid":"another_obj","key":"","snapid":-2,"hash":2087953111,"max":0,"pool":1,"namespace":"n1","max":0}] ["1.6",{"oid":"another_obj","key":"","snapid":-2,"hash":1911284566,"max":0,"pool":1,"namespace":"","max":0}] $ ceph-objectstore-tool --data-path dev/osd0 '["1.6",{"oid":"another_obj","key":"","snapid":-2,"hash":1911284566,"max":0,"pool":1,"namespace":"","max":0}]' dump { "id": { "oid": "another_obj", "key": "", "snapid": -2, "hash": 1911284566, "max": 0, "pool": 1, "namespace": "", "max": 0 }, ... $ ceph-objectstore-tool --data-path dev/osd0 '["1.7",{"oid":"another_obj","key":"","snapid":-2,"hash":2087953111,"max":0,"pool":1,"namespace":"n1","max":0}]' dump { "id": { "oid": "another_obj", "key": "", "snapid": -2, "hash": 2087953111, "max": 0, "pool": 1, "namespace": "n1", "max": 0 }, ... also, the object name is not the full path of it in underlying file store, it should be "obj1". Thanks kefu for the information.
Moving this bug verified state.
command output
[root@host046 ubuntu]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-8/ --pgid 3.5 --op list test --cluster qetest
["3.5",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":3,"namespace":"","max":0}]
Hi David, With Correct/Incorrect --journal-path option the command is failing with "Mount failed with '(22) Invalid argument'" message. [root@host031 ubuntu]# cd /var/lib/ceph/osd/qetest-4/ [root@host031 qetest-4]# ls activate.monmap current journal_uuid ready systemd active fsid keyring store_version type ceph_fsid journal magic superblock whoami [root@magna031 qetest-4]# cd =========With Correct Journal Path============================== [ubuntu@host031 ~]$ sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-4/ --journal-path /var/lib/ceph/osd/qetest-4/ --pgid 3.6 --op list store --cluster qetest Mount failed with '(22) Invalid argument' [ubuntu@hoat031 ~]$ ========With Incorrect Journal Path============================= [ubuntu@magna031 ~]$ sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-4/ --journal-path /var/lib/ceph/osd/ --pgid 3.6 --op list store --cluster qetestMount failed with '(22) Invalid argument' [ubuntu@magna031 ~]$ [ubuntu@magna031 ~]$ sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-4/ --journal-path /var/lib/ceph/osd/qetest-4/current/ --pgid 3.6 --op list store --cluster qetest Mount failed with '(22) Invalid argument' =========================================================== Did i miss anything here ? Thanks for the clarification.
With incorrect journal-path command is not crashing. Marking this bug as verified.
Expected Command outputs of correct and incorrect journal path.
==========with incorrect journal path================
[ubuntu@magna031 ~]$ sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-4/ --journal-path /var/lib/ceph/osd/qetest-4/ --pgid 3.6 --op list store --cluster qetest
Mount failed with '(22) Invalid argument'
==========With Correct journal Path==================
[ubuntu@host031 ~]$ sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-4/ --journal-path /var/lib/ceph/osd/qetest-4/journal --pgid 3.6 --op list store --cluster qetest
["3.6",{"oid":"store","key":"","snapid":-2,"hash":3312803054,"max":0,"pool":3,"namespace":"","max":0}]
===========Without Journal path=====================
[ubuntu@magna031 ~]$ sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/qetest-4/ --pgid 3.6 --op list store --cluster qetest
["3.6",{"oid":"store","key":"","snapid":-2,"hash":3312803054,"max":0,"pool":3,"namespace":"","max":0}]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3387 |
Description of problem: Trying make use of ceph-objectstore-tool, basic list operation crashed the tool Version-Release number of selected component (if applicable): ceph-0.94.9-2.el7cp.x86_64 How reproducible: always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: here is the BT =====here is the BT ============ (gdb) r --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1 Starting program: /bin/ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff1865700 (LWP 8573)] [New Thread 0x7ffff08f1700 (LWP 8574)] [New Thread 0x7ffff00f0700 (LWP 8575)] [New Thread 0x7fffef1cf700 (LWP 8576)] [New Thread 0x7fffee9ce700 (LWP 8577)] os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ffff7fdf7c0 time 2016-09-17 08:36:42.716479 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 0.94.9-2.el7cp (b94f6b36c22fd3a111cd8c0deda43b6c7cfba331) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb2a0d5] 2: (FileJournal::~FileJournal()+0x1d5) [0x807595] 3: (FileJournal::~FileJournal()+0x9) [0x807769] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xba3) [0x9a5c63] 5: (FileStore::mount()+0x3c6f) [0x976c6f] 6: (main()+0x1d0e) [0x6231ce] 7: (__libc_start_main()+0xf5) [0x7ffff3187b15] 8: /bin/ceph-objectstore-tool() [0x62e3b9] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' Program received signal SIGABRT, Aborted. 0x00007ffff319b5f7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install boost-program-options-1.53.0-25.el7.x86_64 boost-system-1.53.0-25.el7.x86_64 boost-thread-1.53.0-25.el7.x86_64 glibc-2.17-106.el7_2.8.x86_64 gperftools-libs-2.4-7.el7.x86_64 leveldb-1.12.0-11.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libstdc++-4.8.5-4.el7.x86_64 libunwind-1.1-5.el7_2.2.x86_64 libuuid-2.23.2-26.el7_2.3.x86_64 lttng-ust-2.4.1-1.el7cp.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-9.el7_2.x86_64 nss-softokn-3.16.2.3-14.2.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 userspace-rcu-0.7.9-2.el7rhgs.x86_64 zlib-1.2.7-15.el7.x86_64 BT FULL ======= #0 0x00007ffff319b5f7 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007ffff319cce8 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007ffff3aa09d5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6 No symbol table info available. #3 0x00007ffff3a9e946 in ?? () from /lib64/libstdc++.so.6 No symbol table info available. #4 0x00007ffff3a9e973 in std::terminate() () from /lib64/libstdc++.so.6 No symbol table info available. #5 0x00007ffff3a9eb93 in __cxa_throw () from /lib64/libstdc++.so.6 No symbol table info available. #6 0x0000000000b2a2ca in ceph::__ceph_assert_fail (assertion=assertion@entry=0xca2ea2 "fd == -1", file=file@entry=0xca304a "os/FileJournal.h", line=line@entry=406, func=func@entry=0xca4f80 <FileJournal::~FileJournal()::__PRETTY_FUNCTION__> "virtual FileJournal::~FileJournal()") at common/assert.cc:77 tss = <incomplete type> buf = "os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ffff7fdf7c0 time 2016-09-17 08:36:42.716479\nos/FileJournal.h: 406: FAILED assert(fd == -1)\n", '\000' <repeats 393 times>... bt = 0x37d29c0 oss = <incomplete type> #7 0x0000000000807595 in FileJournal::~FileJournal (this=0x38a6000, __in_chrg=<optimized out>) at os/FileJournal.h:406 No locals. #8 0x0000000000807769 in FileJournal::~FileJournal (this=0x38a6000, __in_chrg=<optimized out>) at os/FileJournal.h:408 No locals. #9 0x00000000009a5c63 in JournalingObjectStore::journal_replay (this=this@entry=0x37d6000, fs_op_seq=<optimized out>) at os/JournalingObjectStore.cc:60 op_seq = <optimized out> err = -21 count = <optimized out> __PRETTY_FUNCTION__ = "int JournalingObjectStore::journal_replay(uint64_t)" #10 0x0000000000976c6f in FileStore::mount (this=0x37d6000) at os/FileStore.cc:1568 ret = <optimized out> buf = "/var/lib/ceph/osd/ceph-7//fsid\000\000\000\000\000\000\000\000\000\000\237\000\000\000\001\000\000\000\002\000\000\000\000\000\000\000(6\003\000\000\000\000\000(6\003\000\000\000\000\000\334\177", '\000' <repeats 14 times>, "\b", '\000' <r epeats 15 times>, "\251\000\000\000\016\000\000\000\003\000\000\000\000\000\000\000\300\305#\000\000\000\000\000\300\305\003\000\000\000\000\000\020", '\000' <repeats 15 times>, "\b", '\000' <repeats 15 times>, "\265\000\000\000\017\000\000\000\003\000\000\000\000\000\000\000\320\305#\000\000\000\000\000\320\305\003\000\000\000\000\000"... initial_op_seq = 322361 supported_compat_set = {compat = {mask = 1, names = std::map with 0 elements}, ro_compat = {mask = 1, names = std::map with 0 elements}, incompat = {mask = 3, names = std::map with 1 elements = { [1] = "sharded objects"}}} __PRETTY_FUNCTION__ = "virtual int FileStore::mount()" cluster_snaps = std::set with 0 elements version_stamp = 4 nosnapfn = "/var/lib/ceph/osd/ceph-7//current/nosnap\000\000\000\000\000\000\000\000\030\000\000\000\000\000\000\000m\000\000\000\004\000\000\000\002\000\000\000\000\000\000\000 .\000\000\000\000\000\000 .\000\000\000\000\000\000H\f\000\000\000\000\000\000\003\000\000\000\v\000\000\000\b\000\000\000\000\000\000\000\030\000\000\000\000\000\000\000w\000\000\000\001\000\000\000\006\000\000\000\000\000\000\000h:\000\000\000\000\000\000h:\000\000\000\000\000\000\032", '\000' <repeats 15 times>, "\004", '\000' <repeats 15 times>, "r\000\000\000\001\000\000\000\006\000\000\000\000\000\000" #11 0x00000000006231ce in main (argc=<optimized out>, argv=0x7fffffffe4c8) at tools/ceph_objectstore_tool.cc:3375 arg2 = "" ghobj = {hobj = {oid = {name = ""}, snap = {val = 0}, hash = 0, max = false, filestore_key_cache = 0, static POOL_IS_TEMP = -1, pool = -1, nspace = "", key = ""}, generation = 18446744073709551615, shard_id = { id = 255 '\377', static NO_SHARD = {id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}, static NO_GEN = 18446744073709551615} human_readable = true desc = {static m_default_line_length = 80, m_caption = "Allowed options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 16, capacity 16 = {{px = 0x371a640, pn = { pi_ = 0x36f6260}}, {px = 0x371a6a0, pn = {pi_ = 0x36f62c0}}, {px = 0x371a6d0, pn = {pi_ = 0x36f6320}}, { px = 0x371a730, pn = {pi_ = 0x36f6340}}, {px = 0x371a790, pn = {pi_ = 0x36f63a0}}, {px = 0x371a7c0, pn = { pi_ = 0x36f6400}}, {px = 0x371a7f0, pn = {pi_ = 0x36f6460}}, {px = 0x371a820, pn = {pi_ = 0x36f64c0}}, { px = 0x371a850, pn = {pi_ = 0x36f6520}}, {px = 0x371a880, pn = {pi_ = 0x36f6580}}, {px = 0x371a8b0, pn = { pi_ = 0x36f65e0}}, {px = 0x371a8e0, pn = {pi_ = 0x36f6620}}, {px = 0x371a970, pn = {pi_ = 0x36f6660}}, { px = 0x371a9d0, pn = {pi_ = 0x36f66c0}}, {px = 0x371aa00, pn = {pi_ = 0x36f6720}}, {px = 0x371aa30, pn = { pi_ = 0x36f6760}}}, belong_to_group = std::vector<bool> of length 16, capacity 64 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} __PRETTY_FUNCTION__ = "int main(int, char**)" jpath = "/var/lib/ceph/osd/ceph-7/" no_overwrite = <optimized out> ceph_option_strings = std::vector of length 1, capacity 1 = {"obj1"} supported = {compat = {mask = 0, names = std::map with 4294967296 elements<error reading variable: Cannot access memory at address 0x54495f00656c6271>}, ro_compat = {mask = 4294967594, names = std::map with 140737354111408 elements<error reading variable: Cannot access memory at address 0x19>}, incompat = {mask = 140737354110552, names = std::map with 17179869185 elements<error reading variable: Cannot access memory at address 0x19>}} map_epoch = 0 r = <optimized out> pgidstr = "" object = "obj1" epoch = 0 all = {static m_default_line_length = 80, m_caption = "All options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 21, capacity 32 = {{px = 0x371a640, pn = { pi_ = 0x36f6260}}, {px = 0x371a6a0, pn = {pi_ = 0x36f62c0}}, {px = 0x371a6d0, pn = {pi_ = 0x36f6320}}, { px = 0x371a730, pn = {pi_ = 0x36f6340}}, {px = 0x371a790, pn = {pi_ = 0x36f63a0}}, {px = 0x371a7c0, pn = { pi_ = 0x36f6400}}, {px = 0x371a7f0, pn = {pi_ = 0x36f6460}}, {px = 0x371a820, pn = {pi_ = 0x36f64c0}}, { px = 0x371a850, pn = {pi_ = 0x36f6520}}, {px = 0x371a880, pn = {pi_ = 0x36f6580}}, {px = 0x371a8b0, pn = { pi_ = 0x36f65e0}}, {px = 0x371a8e0, pn = {pi_ = 0x36f6620}}, {px = 0x371a970, pn = {pi_ = 0x36f6660}}, { px = 0x371a9d0, pn = {pi_ = 0x36f66c0}}, {px = 0x371aa00, pn = {pi_ = 0x36f6720}}, {px = 0x371aa30, pn = { pi_ = 0x36f6760}}, {px = 0x371aac0, pn = {pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, { px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, {px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = { pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 21, capacity 64 = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, groups = std::vector of length 2, capacity 2 = {{px = 0x37307e0, pn = { pi_ = 0x36f6960}}, {px = 0x3730850, pn = {pi_ = 0x36f6980}}}} ceph_options = std::vector of length 1, capacity 1 = {0x36f6ab8 "obj1"} st = {st_dev = 2081, st_ino = 28, st_nlink = 209, st_mode = 16877, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 12288, st_blksize = 4096, st_blocks = 32, st_atim = {tv_sec = 1473137810, tv_nsec = 151598372}, st_mtim = {tv_sec = 1474009291, tv_nsec = 984209427}, st_ctim = {tv_sec = 1474009291, tv_nsec = 984209427}, __unused = {0, 0, 0}} it = <optimized out> bl = {_buffers = std::list<error reading variable: Cannot access memory at address 0x2f>, _len = 4294967288, _memcopy_count = 4294967295, append_buffer = {_raw = 0x36fcf00, _off = 47, _len = 0}, last_p = {bl = 0x1b, ls = 0x0, off = 26, p = <error reading variable: Cannot access memory at address 0x2>, p_off = 4158546424}, static CLAIM_DEFAULT = 0, static CLAIM_ALLOW_NONSHAREABLE = 1} superblock = {cluster_fsid = {uuid = "\330\035 \000\000\000\000\000\330\035 \000\000\000\000"}, osd_fsid = { uuid = "(\002\000\000\000\000\000\000(\002\000\000\000\000\000"}, whoami = 0, current_epoch = 0, oldest_map = 4160723376, newest_map = 32767, weight = 6.9533558073559583e-310, compat_features = {compat = { mask = 140737488345616, names = std::map with 140737354048712 elements<error reading variable: Cannot access memory at address 0x70000001a>}, ro_compat = {mask = 85899345932, names = std::map with 2287792071848225636 elements<error reading variable: Cannot access memory at address 0xc7f8da00fb65c718>}, incompat = {mask = 7693550737134481880, names = std::map with 140737488347124 elements<error reading variable: Cannot access memory at address 0xfd70e90000002778>}}, mounted = 1, clean_thru = 0, last_map_marked_full = 4158321824} objcmd = "" positional = {static m_default_line_length = 80, m_caption = "Positional options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 5, capacity 8 = {{px = 0x371aac0, pn = { pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, {px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, { px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = {pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 5, capacity 64 = {0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} flags = 0 p = {bl = 0x1, ls = 0x7ffff7daf480, off = 4294960328, p = {_raw = , _off = 2231369728, _len = 1682666688}, p_off = 1} dpath = "/var/lib/ceph/osd/ceph-7/" op = "list" file = "" ls = std::vector of length 0, capacity 0 = {<error reading variable ls (Cannot access memory at address 0x1)> format = "json" pgid = {pgid = {m_pool = 0, m_seed = 0, m_preferred = -1}, shard = {id = 255 '\377', static NO_SHARD = { id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}} force = false head = false pd = {m_names = std::vector of length 4, capacity 4 = {"object", "objcmd", "arg1", "arg2"}, m_trailing = ""} formatter = 0x37d2000 fs_sharded_objects = <optimized out> type = "filestore" vm = <incomplete type> fs = 0x37d6000 ret = <optimized out> oldest_map = 4160723376, newest_map = 32767, weight = 6.9533558073559583e-310, compat_features = {compat = { mask = 140737488345616, names = std::map with 140737354048712 elements<error reading variable: Cannot access memory at address 0x70000001a>}, ro_compat = {mask = 85899345932, names = std::map with 2287792071848225636 elements<error reading variable: Cannot access memory at address 0xc7f8da00fb65c718>}, incompat = {mask = 7693550737134481880, names = std::map with 140737488347124 elements<error reading variable: Cannot access memory at address 0xfd70e90000002778>}}, mounted = 1, clean_thru = 0, last_map_marked_full = 4158321824} objcmd = "" positional = {static m_default_line_length = 80, m_caption = "Positional options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 5, capacity 8 = {{px = 0x371aac0, pn = { pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, {px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, { px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = {pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 5, capacity 64 = {0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} flags = 0 p = {bl = 0x1, ls = 0x7ffff7daf480, off = 4294960328, p = {_raw = , _off = 2231369728, _len = 1682666688}, p_off = 1} dpath = "/var/lib/ceph/osd/ceph-7/" op = "list" file = "" ls = std::vector of length 0, capacity 0 = {<error reading variable ls (Cannot access memory at address 0x1)> format = "json" pgid = {pgid = {m_pool = 0, m_seed = 0, m_preferred = -1}, shard = {id = 255 '\377', static NO_SHARD = { id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}} force = false head = false pd = {m_names = std::vector of length 4, capacity 4 = {"object", "objcmd", "arg1", "arg2"}, m_trailing = ""} formatter = 0x37d2000 fs_sharded_objects = <optimized out> type = "filestore" vm = <incomplete type> fs = 0x37d6000 ret = <optimized out> ---Type <return> to continue, or q <return> to quit--- arg1 = "" Object is present in the bakcend and it also has few snapsets ===================================== [root@magna053 1.ff_head]# ll *obj1* -rw-r--r--. 1 root root 359 Sep 6 07:11 obj1__1_6CF8DEFF__1 -rw-r--r--. 1 root root 359 Sep 6 07:11 obj1__2_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__3_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__4_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__5_6CF8DEFF__1 -rw-rhere is the BT ============ (gdb) r --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1 Starting program: /bin/ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff1865700 (LWP 8573)] [New Thread 0x7ffff08f1700 (LWP 8574)] [New Thread 0x7ffff00f0700 (LWP 8575)] [New Thread 0x7fffef1cf700 (LWP 8576)] [New Thread 0x7fffee9ce700 (LWP 8577)] os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ffff7fdf7c0 time 2016-09-17 08:36:42.716479 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 0.94.9-2.el7cp (b94f6b36c22fd3a111cd8c0deda43b6c7cfba331) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb2a0d5] 2: (FileJournal::~FileJournal()+0x1d5) [0x807595] 3: (FileJournal::~FileJournal()+0x9) [0x807769] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xba3) [0x9a5c63] 5: (FileStore::mount()+0x3c6f) [0x976c6f] 6: (main()+0x1d0e) [0x6231ce] 7: (__libc_start_main()+0xf5) [0x7ffff3187b15] 8: /bin/ceph-objectstore-tool() [0x62e3b9] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' Program received signal SIGABRT, Aborted. 0x00007ffff319b5f7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install boost-program-options-1.53.0-25.el7.x86_64 boost-system-1.53.0-25.el7.x86_64 boost-thread-1.53.0-25.el7.x86_64 glibc-2.17-106.el7_2.8.x86_64 gperftools-libs-2.4-7.el7.x86_64 leveldb-1.12.0-11.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libstdc++-4.8.5-4.el7.x86_64 libunwind-1.1-5.el7_2.2.x86_64 libuuid-2.23.2-26.el7_2.3.x86_64 lttng-ust-2.4.1-1.el7cp.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-9.el7_2.x86_64 nss-softokn-3.16.2.3-14.2.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 userspace-rcu-0.7.9-2.el7rhgs.x86_64 zlib-1.2.7-15.el7.x86_64 BT FULL ======= #0 0x00007ffff319b5f7 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007ffff319cce8 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007ffff3aa09d5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6 No symbol table info available. #3 0x00007ffff3a9e946 in ?? () from /lib64/libstdc++.so.6 No symbol table info available. #4 0x00007ffff3a9e973 in std::terminate() () from /lib64/libstdc++.so.6 No symbol table info available. #5 0x00007ffff3a9eb93 in __cxa_throw () from /lib64/libstdc++.so.6 No symbol table info available. #6 0x0000000000b2a2ca in ceph::__ceph_assert_fail (assertion=assertion@entry=0xca2ea2 "fd == -1", file=file@entry=0xca304a "os/FileJournal.h", line=line@entry=406, func=func@entry=0xca4f80 <FileJournal::~FileJournal()::__PRETTY_FUNCTION__> "virtual FileJournal::~FileJournal()") at common/assert.cc:77 tss = <incomplete type> buf = "os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ffff7fdf7c0 time 2016-09-17 08:36:42.716479\nos/FileJournal.h: 406: FAILED assert(fd == -1)\n", '\000' <repeats 393 times>... bt = 0x37d29c0 oss = <incomplete type> #7 0x0000000000807595 in FileJournal::~FileJournal (this=0x38a6000, __in_chrg=<optimized out>) at os/FileJournal.h:406 No locals. #8 0x0000000000807769 in FileJournal::~FileJournal (this=0x38a6000, __in_chrg=<optimized out>) at os/FileJournal.h:408 No locals. #9 0x00000000009a5c63 in JournalingObjectStore::journal_replay (this=this@entry=0x37d6000, fs_op_seq=<optimized out>) at os/JournalingObjectStore.cc:60 op_seq = <optimized out> err = -21 count = <optimized out> __PRETTY_FUNCTION__ = "int JournalingObjectStore::journal_replay(uint64_t)" #10 0x0000000000976c6f in FileStore::mount (this=0x37d6000) at os/FileStore.cc:1568 ret = <optimized out> buf = "/var/lib/ceph/osd/ceph-7//fsid\000\000\000\000\000\000\000\000\000\000\237\000\000\000\001\000\000\000\002\000\000\000\000\000\000\000(6\003\000\000\000\000\000(6\003\000\000\000\000\000\334\177", '\000' <repeats 14 times>, "\b", '\000' <r epeats 15 times>, "\251\000\000\000\016\000\000\000\003\000\000\000\000\000\000\000\300\305#\000\000\000\000\000\300\305\003\000\000\000\000\000\020", '\000' <repeats 15 times>, "\b", '\000' <repeats 15 times>, "\265\000\000\000\017\000\000\000\003\000\000\000\000\000\000\000\320\305#\000\000\000\000\000\320\305\003\000\000\000\000\000"... initial_op_seq = 322361 supported_compat_set = {compat = {mask = 1, names = std::map with 0 elements}, ro_compat = {mask = 1, names = std::map with 0 elements}, incompat = {mask = 3, names = std::map with 1 elements = { [1] = "sharded objects"}}} __PRETTY_FUNCTION__ = "virtual int FileStore::mount()" cluster_snaps = std::set with 0 elements version_stamp = 4 nosnapfn = "/var/lib/ceph/osd/ceph-7//current/nosnap\000\000\000\000\000\000\000\000\030\000\000\000\000\000\000\000m\000\000\000\004\000\000\000\002\000\000\000\000\000\000\000 .\000\000\000\000\000\000 .\000\000\000\000\000\000H\f\000\000\000\000\000\000\003\000\000\000\v\000\000\000\b\000\000\000\000\000\000\000\030\000\000\000\000\000\000\000w\000\000\000\001\000\000\000\006\000\000\000\000\000\000\000h:\000\000\000\000\000\000h:\000\000\000\000\000\000\032", '\000' <repeats 15 times>, "\004", '\000' <repeats 15 times>, "r\000\000\000\001\000\000\000\006\000\000\000\000\000\000" #11 0x00000000006231ce in main (argc=<optimized out>, argv=0x7fffffffe4c8) at tools/ceph_objectstore_tool.cc:3375 arg2 = "" ghobj = {hobj = {oid = {name = ""}, snap = {val = 0}, hash = 0, max = false, filestore_key_cache = 0, static POOL_IS_TEMP = -1, pool = -1, nspace = "", key = ""}, generation = 18446744073709551615, shard_id = { id = 255 '\377', static NO_SHARD = {id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}, static NO_GEN = 18446744073709551615} human_readable = true desc = {static m_default_line_length = 80, m_caption = "Allowed options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 16, capacity 16 = {{px = 0x371a640, pn = { pi_ = 0x36f6260}}, {px = 0x371a6a0, pn = {pi_ = 0x36f62c0}}, {px = 0x371a6d0, pn = {pi_ = 0x36f6320}}, { px = 0x371a730, pn = {pi_ = 0x36f6340}}, {px = 0x371a790, pn = {pi_ = 0x36f63a0}}, {px = 0x371a7c0, pn = { pi_ = 0x36f6400}}, {px = 0x371a7f0, pn = {pi_ = 0x36f6460}}, {px = 0x371a820, pn = {pi_ = 0x36f64c0}}, { px = 0x371a850, pn = {pi_ = 0x36f6520}}, {px = 0x371a880, pn = {pi_ = 0x36f6580}}, {px = 0x371a8b0, pn = { pi_ = 0x36f65e0}}, {px = 0x371a8e0, pn = {pi_ = 0x36f6620}}, {px = 0x371a970, pn = {pi_ = 0x36f6660}}, { px = 0x371a9d0, pn = {pi_ = 0x36f66c0}}, {px = 0x371aa00, pn = {pi_ = 0x36f6720}}, {px = 0x371aa30, pn = { pi_ = 0x36f6760}}}, belong_to_group = std::vector<bool> of length 16, capacity 64 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} __PRETTY_FUNCTION__ = "int main(int, char**)" jpath = "/var/lib/ceph/osd/ceph-7/" no_overwrite = <optimized out> ceph_option_strings = std::vector of length 1, capacity 1 = {"obj1"} supported = {compat = {mask = 0, names = std::map with 4294967296 elements<error reading variable: Cannot access memory at address 0x54495f00656c6271>}, ro_compat = {mask = 4294967594, names = std::map with 140737354111408 elements<error reading variable: Cannot access memory at address 0x19>}, incompat = {mask = 140737354110552, names = std::map with 17179869185 elements<error reading variable: Cannot access memory at address 0x19>}} map_epoch = 0 r = <optimized out> pgidstr = "" object = "obj1" epoch = 0 all = {static m_default_line_length = 80, m_caption = "All options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 21, capacity 32 = {{px = 0x371a640, pn = { pi_ = 0x36f6260}}, {px = 0x371a6a0, pn = {pi_ = 0x36f62c0}}, {px = 0x371a6d0, pn = {pi_ = 0x36f6320}}, { px = 0x371a730, pn = {pi_ = 0x36f6340}}, {px = 0x371a790, pn = {pi_ = 0x36f63a0}}, {px = 0x371a7c0, pn = { pi_ = 0x36f6400}}, {px = 0x371a7f0, pn = {pi_ = 0x36f6460}}, {px = 0x371a820, pn = {pi_ = 0x36f64c0}}, { px = 0x371a850, pn = {pi_ = 0x36f6520}}, {px = 0x371a880, pn = {pi_ = 0x36f6580}}, {px = 0x371a8b0, pn = { pi_ = 0x36f65e0}}, {px = 0x371a8e0, pn = {pi_ = 0x36f6620}}, {px = 0x371a970, pn = {pi_ = 0x36f6660}}, { px = 0x371a9d0, pn = {pi_ = 0x36f66c0}}, {px = 0x371aa00, pn = {pi_ = 0x36f6720}}, {px = 0x371aa30, pn = { pi_ = 0x36f6760}}, {px = 0x371aac0, pn = {pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, { px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, {px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = { pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 21, capacity 64 = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, groups = std::vector of length 2, capacity 2 = {{px = 0x37307e0, pn = { pi_ = 0x36f6960}}, {px = 0x3730850, pn = {pi_ = 0x36f6980}}}} ceph_options = std::vector of length 1, capacity 1 = {0x36f6ab8 "obj1"} st = {st_dev = 2081, st_ino = 28, st_nlink = 209, st_mode = 16877, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 12288, st_blksize = 4096, st_blocks = 32, st_atim = {tv_sec = 1473137810, tv_nsec = 151598372}, st_mtim = {tv_sec = 1474009291, tv_nsec = 984209427}, st_ctim = {tv_sec = 1474009291, tv_nsec = 984209427}, __unused = {0, 0, 0}} it = <optimized out> bl = {_buffers = std::list<error reading variable: Cannot access memory at address 0x2f>, _len = 4294967288, _memcopy_count = 4294967295, append_buffer = {_raw = 0x36fcf00, _off = 47, _len = 0}, last_p = {bl = 0x1b, ls = 0x0, off = 26, p = <error reading variable: Cannot access memory at address 0x2>, p_off = 4158546424}, static CLAIM_DEFAULT = 0, static CLAIM_ALLOW_NONSHAREABLE = 1} superblock = {cluster_fsid = {uuid = "\330\035 \000\000\000\000\000\330\035 \000\000\000\000"}, osd_fsid = { uuid = "(\002\000\000\000\000\000\000(\002\000\000\000\000\000"}, whoami = 0, current_epoch = 0, oldest_map = 4160723376, newest_map = 32767, weight = 6.9533558073559583e-310, compat_features = {compat = { mask = 140737488345616, names = std::map with 140737354048712 elements<error reading variable: Cannot access memory at address 0x70000001a>}, ro_compat = {mask = 85899345932, names = std::map with 2287792071848225636 elements<error reading variable: Cannot access memory at address 0xc7f8da00fb65c718>}, incompat = {mask = 7693550737134481880, names = std::map with 140737488347124 elements<error reading variable: Cannot access memory at address 0xfd70e90000002778>}}, mounted = 1, clean_thru = 0, last_map_marked_full = 4158321824} objcmd = "" positional = {static m_default_line_length = 80, m_caption = "Positional options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 5, capacity 8 = {{px = 0x371aac0, pn = { pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, {px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, { px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = {pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 5, capacity 64 = {0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} flags = 0 p = {bl = 0x1, ls = 0x7ffff7daf480, off = 4294960328, p = {_raw = , _off = 2231369728, _len = 1682666688}, p_off = 1} dpath = "/var/lib/ceph/osd/ceph-7/" op = "list" file = "" ls = std::vector of length 0, capacity 0 = {<error reading variable ls (Cannot access memory at address 0x1)> format = "json" pgid = {pgid = {m_pool = 0, m_seed = 0, m_preferred = -1}, shard = {id = 255 '\377', static NO_SHARD = { id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}} force = false head = false pd = {m_names = std::vector of length 4, capacity 4 = {"object", "objcmd", "arg1", "arg2"}, m_trailing = ""} formatter = 0x37d2000 fs_sharded_objects = <optimized out> type = "filestore" vm = <incomplete type> fs = 0x37d6000 ret = <optimized out> oldest_map = 4160723376, newest_map = 32767, weight = 6.9533558073559583e-310, compat_features = {compat = { mask = 140737488345616, names = std::map with 140737354048712 elements<error reading variable: Cannot access memory at address 0x70000001a>}, ro_compat = {mask = 85899345932, names = std::map with 2287792071848225636 elements<error reading variable: Cannot access memory at address 0xc7f8da00fb65c718>}, incompat = {mask = 7693550737134481880, names = std::map with 140737488347124 elements<error reading variable: Cannot access memory at address 0xfd70e90000002778>}}, mounted = 1, clean_thru = 0, last_map_marked_full = 4158321824} objcmd = "" positional = {static m_default_line_length = 80, m_caption = "Positional options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 5, capacity 8 = {{px = 0x371aac0, pn = { pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, {px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, { px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = {pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 5, capacity 64 = {0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} flags = 0 p = {bl = 0x1, ls = 0x7ffff7daf480, off = 4294960328, p = {_raw = , _off = 2231369728, _len = 1682666688}, p_off = 1} dpath = "/var/lib/ceph/osd/ceph-7/" op = "list" file = "" ls = std::vector of length 0, capacity 0 = {<error reading variable ls (Cannot access memory at address 0x1)> format = "json" pgid = {pgid = {m_pool = 0, m_seed = 0, m_preferred = -1}, shard = {id = 255 '\377', static NO_SHARD = { id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}} force = false head = false pd = {m_names = std::vector of length 4, capacity 4 = {"object", "objcmd", "arg1", "arg2"}, m_trailing = ""} formatter = 0x37d2000 fs_sharded_objects = <optimized out> type = "filestore" vm = <incomplete type> fs = 0x37d6000 ret = <optimized out> ---Type <return> to continue, or q <return> to quit--- arg1 = "" Object is present in the bakcend and it also has few snapsets ===================================== [root@magna053 1.ff_head]# ll *obj1* -rw-r--r--. 1 root root 359 Sep 6 07:11 obj1__1_6CF8DEFF__1 -rw-r--r--. 1 root root 359 Sep 6 07:11 obj1__2_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__3_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__4_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__5_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__6_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__7_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__head_6CF8DEFF__1--r--. 1 root root 9889630 Sep 17 08:14 obj1__6_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__7_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__head_6CF8DEFF__1======= (gdb) r --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1 Starting program: /bin/ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7/ --journal-path /var/lib/ceph/osd/ceph-7/ --op list obj1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff1865700 (LWP 8573)] [New Thread 0x7ffff08f1700 (LWP 8574)] [New Thread 0x7ffff00f0700 (LWP 8575)] [New Thread 0x7fffef1cf700 (LWP 8576)] [New Thread 0x7fffee9ce700 (LWP 8577)] os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ffff7fdf7c0 time 2016-09-17 08:36:42.716479 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 0.94.9-2.el7cp (b94f6b36c22fd3a111cd8c0deda43b6c7cfba331) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb2a0d5] 2: (FileJournal::~FileJournal()+0x1d5) [0x807595] 3: (FileJournal::~FileJournal()+0x9) [0x807769] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xba3) [0x9a5c63] 5: (FileStore::mount()+0x3c6f) [0x976c6f] 6: (main()+0x1d0e) [0x6231ce] 7: (__libc_start_main()+0xf5) [0x7ffff3187b15] 8: /bin/ceph-objectstore-tool() [0x62e3b9] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' Program received signal SIGABRT, Aborted. 0x00007ffff319b5f7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install boost-program-options-1.53.0-25.el7.x86_64 boost-system-1.53.0-25.el7.x86_64 boost-thread-1.53.0-25.el7.x86_64 glibc-2.17-106.el7_2.8.x86_64 gperftools-libs-2.4-7.el7.x86_64 leveldb-1.12.0-11.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libstdc++-4.8.5-4.el7.x86_64 libunwind-1.1-5.el7_2.2.x86_64 libuuid-2.23.2-26.el7_2.3.x86_64 lttng-ust-2.4.1-1.el7cp.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-9.el7_2.x86_64 nss-softokn-3.16.2.3-14.2.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 userspace-rcu-0.7.9-2.el7rhgs.x86_64 zlib-1.2.7-15.el7.x86_64 BT FULL ======= #0 0x00007ffff319b5f7 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007ffff319cce8 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007ffff3aa09d5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6 No symbol table info available. #3 0x00007ffff3a9e946 in ?? () from /lib64/libstdc++.so.6 No symbol table info available. #4 0x00007ffff3a9e973 in std::terminate() () from /lib64/libstdc++.so.6 No symbol table info available. #5 0x00007ffff3a9eb93 in __cxa_throw () from /lib64/libstdc++.so.6 No symbol table info available. #6 0x0000000000b2a2ca in ceph::__ceph_assert_fail (assertion=assertion@entry=0xca2ea2 "fd == -1", file=file@entry=0xca304a "os/FileJournal.h", line=line@entry=406, func=func@entry=0xca4f80 <FileJournal::~FileJournal()::__PRETTY_FUNCTION__> "virtual FileJournal::~FileJournal()") at common/assert.cc:77 tss = <incomplete type> buf = "os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ffff7fdf7c0 time 2016-09-17 08:36:42.716479\nos/FileJournal.h: 406: FAILED assert(fd == -1)\n", '\000' <repeats 393 times>... bt = 0x37d29c0 oss = <incomplete type> #7 0x0000000000807595 in FileJournal::~FileJournal (this=0x38a6000, __in_chrg=<optimized out>) at os/FileJournal.h:406 No locals. #8 0x0000000000807769 in FileJournal::~FileJournal (this=0x38a6000, __in_chrg=<optimized out>) at os/FileJournal.h:408 No locals. #9 0x00000000009a5c63 in JournalingObjectStore::journal_replay (this=this@entry=0x37d6000, fs_op_seq=<optimized out>) at os/JournalingObjectStore.cc:60 op_seq = <optimized out> err = -21 count = <optimized out> __PRETTY_FUNCTION__ = "int JournalingObjectStore::journal_replay(uint64_t)" #10 0x0000000000976c6f in FileStore::mount (this=0x37d6000) at os/FileStore.cc:1568 ret = <optimized out> buf = "/var/lib/ceph/osd/ceph-7//fsid\000\000\000\000\000\000\000\000\000\000\237\000\000\000\001\000\000\000\002\000\000\000\000\000\000\000(6\003\000\000\000\000\000(6\003\000\000\000\000\000\334\177", '\000' <repeats 14 times>, "\b", '\000' <r epeats 15 times>, "\251\000\000\000\016\000\000\000\003\000\000\000\000\000\000\000\300\305#\000\000\000\000\000\300\305\003\000\000\000\000\000\020", '\000' <repeats 15 times>, "\b", '\000' <repeats 15 times>, "\265\000\000\000\017\000\000\000\003\000\000\000\000\000\000\000\320\305#\000\000\000\000\000\320\305\003\000\000\000\000\000"... initial_op_seq = 322361 supported_compat_set = {compat = {mask = 1, names = std::map with 0 elements}, ro_compat = {mask = 1, names = std::map with 0 elements}, incompat = {mask = 3, names = std::map with 1 elements = { [1] = "sharded objects"}}} __PRETTY_FUNCTION__ = "virtual int FileStore::mount()" cluster_snaps = std::set with 0 elements version_stamp = 4 nosnapfn = "/var/lib/ceph/osd/ceph-7//current/nosnap\000\000\000\000\000\000\000\000\030\000\000\000\000\000\000\000m\000\000\000\004\000\000\000\002\000\000\000\000\000\000\000 .\000\000\000\000\000\000 .\000\000\000\000\000\000H\f\000\000\000\000\000\000\003\000\000\000\v\000\000\000\b\000\000\000\000\000\000\000\030\000\000\000\000\000\000\000w\000\000\000\001\000\000\000\006\000\000\000\000\000\000\000h:\000\000\000\000\000\000h:\000\000\000\000\000\000\032", '\000' <repeats 15 times>, "\004", '\000' <repeats 15 times>, "r\000\000\000\001\000\000\000\006\000\000\000\000\000\000" #11 0x00000000006231ce in main (argc=<optimized out>, argv=0x7fffffffe4c8) at tools/ceph_objectstore_tool.cc:3375 arg2 = "" ghobj = {hobj = {oid = {name = ""}, snap = {val = 0}, hash = 0, max = false, filestore_key_cache = 0, static POOL_IS_TEMP = -1, pool = -1, nspace = "", key = ""}, generation = 18446744073709551615, shard_id = { id = 255 '\377', static NO_SHARD = {id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}, static NO_GEN = 18446744073709551615} human_readable = true desc = {static m_default_line_length = 80, m_caption = "Allowed options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 16, capacity 16 = {{px = 0x371a640, pn = { pi_ = 0x36f6260}}, {px = 0x371a6a0, pn = {pi_ = 0x36f62c0}}, {px = 0x371a6d0, pn = {pi_ = 0x36f6320}}, { px = 0x371a730, pn = {pi_ = 0x36f6340}}, {px = 0x371a790, pn = {pi_ = 0x36f63a0}}, {px = 0x371a7c0, pn = { pi_ = 0x36f6400}}, {px = 0x371a7f0, pn = {pi_ = 0x36f6460}}, {px = 0x371a820, pn = {pi_ = 0x36f64c0}}, { px = 0x371a850, pn = {pi_ = 0x36f6520}}, {px = 0x371a880, pn = {pi_ = 0x36f6580}}, {px = 0x371a8b0, pn = { pi_ = 0x36f65e0}}, {px = 0x371a8e0, pn = {pi_ = 0x36f6620}}, {px = 0x371a970, pn = {pi_ = 0x36f6660}}, { px = 0x371a9d0, pn = {pi_ = 0x36f66c0}}, {px = 0x371aa00, pn = {pi_ = 0x36f6720}}, {px = 0x371aa30, pn = { pi_ = 0x36f6760}}}, belong_to_group = std::vector<bool> of length 16, capacity 64 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} __PRETTY_FUNCTION__ = "int main(int, char**)" jpath = "/var/lib/ceph/osd/ceph-7/" no_overwrite = <optimized out> ceph_option_strings = std::vector of length 1, capacity 1 = {"obj1"} supported = {compat = {mask = 0, names = std::map with 4294967296 elements<error reading variable: Cannot access memory at address 0x54495f00656c6271>}, ro_compat = {mask = 4294967594, names = std::map with 140737354111408 elements<error reading variable: Cannot access memory at address 0x19>}, incompat = {mask = 140737354110552, names = std::map with 17179869185 elements<error reading variable: Cannot access memory at address 0x19>}} map_epoch = 0 r = <optimized out> pgidstr = "" object = "obj1" epoch = 0 all = {static m_default_line_length = 80, m_caption = "All options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 21, capacity 32 = {{px = 0x371a640, pn = { pi_ = 0x36f6260}}, {px = 0x371a6a0, pn = {pi_ = 0x36f62c0}}, {px = 0x371a6d0, pn = {pi_ = 0x36f6320}}, { px = 0x371a730, pn = {pi_ = 0x36f6340}}, {px = 0x371a790, pn = {pi_ = 0x36f63a0}}, {px = 0x371a7c0, pn = { pi_ = 0x36f6400}}, {px = 0x371a7f0, pn = {pi_ = 0x36f6460}}, {px = 0x371a820, pn = {pi_ = 0x36f64c0}}, { px = 0x371a850, pn = {pi_ = 0x36f6520}}, {px = 0x371a880, pn = {pi_ = 0x36f6580}}, {px = 0x371a8b0, pn = { pi_ = 0x36f65e0}}, {px = 0x371a8e0, pn = {pi_ = 0x36f6620}}, {px = 0x371a970, pn = {pi_ = 0x36f6660}}, { px = 0x371a9d0, pn = {pi_ = 0x36f66c0}}, {px = 0x371aa00, pn = {pi_ = 0x36f6720}}, {px = 0x371aa30, pn = { pi_ = 0x36f6760}}, {px = 0x371aac0, pn = {pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, { px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, {px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = { pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 21, capacity 64 = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, groups = std::vector of length 2, capacity 2 = {{px = 0x37307e0, pn = { pi_ = 0x36f6960}}, {px = 0x3730850, pn = {pi_ = 0x36f6980}}}} ceph_options = std::vector of length 1, capacity 1 = {0x36f6ab8 "obj1"} st = {st_dev = 2081, st_ino = 28, st_nlink = 209, st_mode = 16877, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 12288, st_blksize = 4096, st_blocks = 32, st_atim = {tv_sec = 1473137810, tv_nsec = 151598372}, st_mtim = {tv_sec = 1474009291, tv_nsec = 984209427}, st_ctim = {tv_sec = 1474009291, tv_nsec = 984209427}, __unused = {0, 0, 0}} it = <optimized out> bl = {_buffers = std::list<error reading variable: Cannot access memory at address 0x2f>, _len = 4294967288, _memcopy_count = 4294967295, append_buffer = {_raw = 0x36fcf00, _off = 47, _len = 0}, last_p = {bl = 0x1b, ls = 0x0, off = 26, p = <error reading variable: Cannot access memory at address 0x2>, p_off = 4158546424}, static CLAIM_DEFAULT = 0, static CLAIM_ALLOW_NONSHAREABLE = 1} superblock = {cluster_fsid = {uuid = "\330\035 \000\000\000\000\000\330\035 \000\000\000\000"}, osd_fsid = { uuid = "(\002\000\000\000\000\000\000(\002\000\000\000\000\000"}, whoami = 0, current_epoch = 0, oldest_map = 4160723376, newest_map = 32767, weight = 6.9533558073559583e-310, compat_features = {compat = { mask = 140737488345616, names = std::map with 140737354048712 elements<error reading variable: Cannot access memory at address 0x70000001a>}, ro_compat = {mask = 85899345932, names = std::map with 2287792071848225636 elements<error reading variable: Cannot access memory at address 0xc7f8da00fb65c718>}, incompat = {mask = 7693550737134481880, names = std::map with 140737488347124 elements<error reading variable: Cannot access memory at address 0xfd70e90000002778>}}, mounted = 1, clean_thru = 0, last_map_marked_full = 4158321824} objcmd = "" positional = {static m_default_line_length = 80, m_caption = "Positional options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 5, capacity 8 = {{px = 0x371aac0, pn = { pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, {px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, { px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = {pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 5, capacity 64 = {0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} flags = 0 p = {bl = 0x1, ls = 0x7ffff7daf480, off = 4294960328, p = {_raw = , _off = 2231369728, _len = 1682666688}, p_off = 1} dpath = "/var/lib/ceph/osd/ceph-7/" op = "list" file = "" ls = std::vector of length 0, capacity 0 = {<error reading variable ls (Cannot access memory at address 0x1)> format = "json" pgid = {pgid = {m_pool = 0, m_seed = 0, m_preferred = -1}, shard = {id = 255 '\377', static NO_SHARD = { id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}} force = false head = false pd = {m_names = std::vector of length 4, capacity 4 = {"object", "objcmd", "arg1", "arg2"}, m_trailing = ""} formatter = 0x37d2000 fs_sharded_objects = <optimized out> type = "filestore" vm = <incomplete type> fs = 0x37d6000 ret = <optimized out> oldest_map = 4160723376, newest_map = 32767, weight = 6.9533558073559583e-310, compat_features = {compat = { mask = 140737488345616, names = std::map with 140737354048712 elements<error reading variable: Cannot access memory at address 0x70000001a>}, ro_compat = {mask = 85899345932, names = std::map with 2287792071848225636 elements<error reading variable: Cannot access memory at address 0xc7f8da00fb65c718>}, incompat = {mask = 7693550737134481880, names = std::map with 140737488347124 elements<error reading variable: Cannot access memory at address 0xfd70e90000002778>}}, mounted = 1, clean_thru = 0, last_map_marked_full = 4158321824} objcmd = "" positional = {static m_default_line_length = 80, m_caption = "Positional options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 5, capacity 8 = {{px = 0x371aac0, pn = { pi_ = 0x36f67e0}}, {px = 0x371aaf0, pn = {pi_ = 0x36f6840}}, {px = 0x371ab20, pn = {pi_ = 0x36f68c0}}, { px = 0x371ab80, pn = {pi_ = 0x36f6900}}, {px = 0x371abe0, pn = {pi_ = 0x36f6940}}}, belong_to_group = std::vector<bool> of length 5, capacity 64 = {0, 0, 0, 0, 0}, groups = std::vector of length 0, capacity 0} flags = 0 p = {bl = 0x1, ls = 0x7ffff7daf480, off = 4294960328, p = {_raw = , _off = 2231369728, _len = 1682666688}, p_off = 1} dpath = "/var/lib/ceph/osd/ceph-7/" op = "list" file = "" ls = std::vector of length 0, capacity 0 = {<error reading variable ls (Cannot access memory at address 0x1)> format = "json" pgid = {pgid = {m_pool = 0, m_seed = 0, m_preferred = -1}, shard = {id = 255 '\377', static NO_SHARD = { id = 255 '\377', static NO_SHARD = <same as static member of an already seen type>}}} force = false head = false pd = {m_names = std::vector of length 4, capacity 4 = {"object", "objcmd", "arg1", "arg2"}, m_trailing = ""} formatter = 0x37d2000 fs_sharded_objects = <optimized out> type = "filestore" vm = <incomplete type> fs = 0x37d6000 ret = <optimized out> ---Type <return> to continue, or q <return> to quit--- arg1 = "" Object is present in the bakcend and it also has few snapsets ===================================== [root@magna053 1.ff_head]# ll *obj1* -rw-r--r--. 1 root root 359 Sep 6 07:11 obj1__1_6CF8DEFF__1 -rw-r--r--. 1 root root 359 Sep 6 07:11 obj1__2_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__3_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__4_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__5_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__6_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__7_6CF8DEFF__1 -rw-r--r--. 1 root root 9889630 Sep 17 08:14 obj1__head_6CF8DEFF__1