Bug 1498611 - [RHCeph 3.0/ 12.2.1-9.el7cp] src/common/interval_map.h: 161: FAILED assert(len > 0)
Summary: [RHCeph 3.0/ 12.2.1-9.el7cp] src/common/interval_map.h: 161: FAILED assert(le...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RADOS
Version: 3.0
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: rc
: 3.0
Assignee: Josh Durgin
QA Contact: Vasu Kulkarni
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-04 18:44 UTC by Vasu Kulkarni
Modified: 2022-02-21 18:20 UTC (History)
6 users (show)

Fixed In Version: RHEL: ceph-12.2.1-10.el7cp Ubuntu: ceph_12.2.1-12redhat1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-05 23:46:42 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 21629 0 None None None 2017-10-04 18:46:38 UTC
Red Hat Product Errata RHBA-2017:3387 0 normal SHIPPED_LIVE Red Hat Ceph Storage 3.0 bug fix and enhancement update 2017-12-06 03:03:45 UTC

Description Vasu Kulkarni 2017-10-04 18:44:07 UTC 
Description of problem:

Crash seen during EC pool rbd fsx test

Test Description : rbd/librbd/{cache/none.yaml clusters/{fixed-3.yaml openstack.yaml} config/copy-on-read.yaml
    msgr-failures/few.yaml objectstore/filestore-xfs.yaml pool/ec-data-pool.yaml workloads/fsx.yaml}

2017-10-02T23:14:41.822 INFO:teuthology.orchestra.run.clara005.stdout:12665 write	0x2cd16b thru	0x2d8966	(0xb7fc bytes)
2017-10-02T23:14:41.855 INFO:teuthology.orchestra.run.clara005.stdout:12668 compareandwrite	0x68a2ef0 thru	0x68a2f6e	(0x7f bytes)
2017-10-02T23:14:41.861 INFO:tasks.ceph.osd.3.pluto009.stderr:/builddir/build/BUILD/ceph-12.2.1/src/common/interval_map.h: In function 'void interval_map<K, V, S>::insert(K, K, V&&) [with K = long unsigned int; V = ceph::buffer::list; S = bl_split_merge]' thread 7f1de111c700 time 2017-10-03 03:16:46.790002
2017-10-02T23:14:41.861 INFO:tasks.ceph.osd.3.pluto009.stderr:/builddir/build/BUILD/ceph-12.2.1/src/common/interval_map.h: 161: FAILED assert(len > 0)
2017-10-02T23:14:41.865 INFO:tasks.ceph.osd.3.pluto009.stderr: ceph version 12.2.1-9.el7cp (3972a2f60763dcf1be2e26457eee677515a2705d) luminous (stable)
2017-10-02T23:14:41.865 INFO:tasks.ceph.osd.3.pluto009.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x558207165f10]
2017-10-02T23:14:41.865 INFO:tasks.ceph.osd.3.pluto009.stderr: 2: (interval_map<unsigned long, ceph::buffer::list, bl_split_merge>::insert(unsigned long, unsigned long, ceph::buffer::list&&)+0x108) [0x558206ef43c8]
2017-10-02T23:14:41.865 INFO:tasks.ceph.osd.3.pluto009.stderr: 3: (CallClientContexts::finish(std::pair<RecoveryMessages*, ECBackend::read_result_t&>&)+0x2f7) [0x558206ef47f7]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 4: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x7f) [0x558206ecb58f]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0x1028) [0x558206ed2d18]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1af) [0x558206edd6df]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x558206de23c0]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5ae) [0x558206d4dfae]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9) [0x558206bde399]
2017-10-02T23:14:41.866 INFO:tasks.ceph.osd.3.pluto009.stderr: 10: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x558206e50787]
2017-10-02T23:14:41.867 INFO:tasks.ceph.osd.3.pluto009.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfce) [0x558206c099de]
2017-10-02T23:14:41.867 INFO:tasks.ceph.osd.3.pluto009.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839) [0x55820716ba29]
2017-10-02T23:14:41.867 INFO:tasks.ceph.osd.3.pluto009.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55820716d9c0]
2017-10-02T23:14:41.867 INFO:tasks.ceph.osd.3.pluto009.stderr: 14: (()+0x7e25) [0x7f1dfd9ece25]
2017-10-02T23:14:41.867 INFO:tasks.ceph.osd.3.pluto009.stderr: 15: (clone()+0x6d) [0x7f1dfcae034d]
2017-10-02T23:14:41.867 INFO:tasks.ceph.osd.3.pluto009.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-10-02T23:14:41.868 INFO:tasks.ceph.osd.3.pluto009.stderr:2017-10-03 03:16:46.794097 7f1de111c700 -1 /builddir/build/BUILD/ceph-12.2.1/src/common/interval_map.h: In function 'void interval_map<K, V, S>::insert(K, K, V&&) [with K = long unsigned int; V = ceph::buffer::list; S = bl_split_merge]' thread 7f1de111c700 time 2017-10-03 03:16:46.790002
2017-10-02T23:14:41.868 INFO:tasks.ceph.osd.3.pluto009.stderr:/builddir/build/BUILD/ceph-12.2.1/src/common/interval_map.h: 161: FAILED assert(len > 0)
2017-10-02T23:14:41.868 INFO:tasks.ceph.osd.3.pluto009.stderr:
2017-10-02T23:14:41.868 INFO:tasks.ceph.osd.3.pluto009.stderr: ceph version 12.2.1-9.el7cp (3972a2f60763dcf1be2e26457eee677515a2705d) luminous (stable)
2017-10-02T23:14:41.868 INFO:tasks.ceph.osd.3.pluto009.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x558207165f10]
2017-10-02T23:14:41.868 INFO:tasks.ceph.osd.3.pluto009.stderr: 2: (interval_map<unsigned long, ceph::buffer::list, bl_split_merge>::insert(unsigned long, unsigned long, ceph::buffer::list&&)+0x108) [0x558206ef43c8]
2017-10-02T23:14:41.869 INFO:tasks.ceph.osd.3.pluto009.stderr: 3: (CallClientContexts::finish(std::pair<RecoveryMessages*, ECBackend::read_result_t&>&)+0x2f7) [0x558206ef47f7]
2017-10-02T23:14:41.869 INFO:tasks.ceph.osd.3.pluto009.stderr: 4: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x7f) [0x558206ecb58f]
2017-10-02T23:14:41.869 INFO:tasks.ceph.osd.3.pluto009.stderr: 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0x1028) [0x558206ed2d18]
2017-10-02T23:14:41.869 INFO:tasks.ceph.osd.3.pluto009.stderr: 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1af) [0x558206edd6df]
2017-10-02T23:14:41.869 INFO:tasks.ceph.osd.3.pluto009.stderr: 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x558206de23c0]
2017-10-02T23:14:41.869 INFO:tasks.ceph.osd.3.pluto009.stderr: 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5ae) [0x558206d4dfae]
2017-10-02T23:14:41.870 INFO:tasks.ceph.osd.3.pluto009.stderr: 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9) [0x558206bde399]
2017-10-02T23:14:41.870 INFO:tasks.ceph.osd.3.pluto009.stderr: 10: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x558206e50787]
2017-10-02T23:14:41.870 INFO:tasks.ceph.osd.3.pluto009.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfce) [0x558206c099de]
2017-10-02T23:14:41.870 INFO:tasks.ceph.osd.3.pluto009.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839) [0x55820716ba29]
2017-10-02T23:14:41.870 INFO:tasks.ceph.osd.3.pluto009.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55820716d9c0]
2017-10-02T23:14:41.870 INFO:tasks.ceph.osd.3.pluto009.stderr: 14: (()+0x7e25) [0x7f1dfd9ece25]
2017-10-02T23:14:41.871 INFO:tasks.ceph.osd.3.pluto009.stderr: 15: (clone()+0x6d) [0x7f1dfcae034d]
2017-10-02T23:14:41.871 INFO:tasks.ceph.osd.3.pluto009.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-10-02T23:14:41.871 INFO:tasks.ceph.osd.3.pluto009.stderr:
2017-10-02T23:14:41.889 INFO:tasks.ceph.osd.3.pluto009.stderr:     0> 2017-10-03 03:16:46.794097 7f1de111c700 -1 /builddir/build/BUILD/ceph-12.2.1/src/common/interval_map.h: In function 'void interval_map<K, V, S>::insert(K, K, V&&) [with K = long unsigned int; V = ceph::buffer::list; S = bl_split_merge]' thread 7f1de111c700 time 2017-10-03 03:16:46.790002
2017-10-02T23:14:41.889 INFO:tasks.ceph.osd.3.pluto009.stderr:/builddir/build/BUILD/ceph-12.2.1/src/common/interval_map.h: 161: FAILED assert(len > 0)
2017-10-02T23:14:41.889 INFO:tasks.ceph.osd.3.pluto009.stderr:
2017-10-02T23:14:41.889 INFO:tasks.ceph.osd.3.pluto009.stderr: ceph version 12.2.1-9.el7cp (3972a2f60763dcf1be2e26457eee677515a2705d) luminous (stable)
2017-10-02T23:14:41.889 INFO:tasks.ceph.osd.3.pluto009.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x558207165f10]
2017-10-02T23:14:41.889 INFO:tasks.ceph.osd.3.pluto009.stderr: 2: (interval_map<unsigned long, ceph::buffer::list, bl_split_merge>::insert(unsigned long, unsigned long, ceph::buffer::list&&)+0x108) [0x558206ef43c8]
2017-10-02T23:14:41.890 INFO:tasks.ceph.osd.3.pluto009.stderr: 3: (CallClientContexts::finish(std::pair<RecoveryMessages*, ECBackend::read_result_t&>&)+0x2f7) [0x558206ef47f7]
2017-10-02T23:14:41.890 INFO:tasks.ceph.osd.3.pluto009.stderr: 4: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x7f) [0x558206ecb58f]
2017-10-02T23:14:41.890 INFO:tasks.ceph.osd.3.pluto009.stderr: 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0x1028) [0x558206ed2d18]
2017-10-02T23:14:41.890 INFO:tasks.ceph.osd.3.pluto009.stderr: 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1af) [0x558206edd6df]
2017-10-02T23:14:41.890 INFO:tasks.ceph.osd.3.pluto009.stderr: 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x558206de23c0]
2017-10-02T23:14:41.891 INFO:tasks.ceph.osd.3.pluto009.stderr: 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5ae) [0x558206d4dfae]
Full logs: http://magna002.ceph.redhat.com/vasu-2017-10-02_14:57:26-rbd-luminous---basic-multi/275702/teuthology.log
Comment 9 errata-xmlrpc 2017-12-05 23:46:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387
Note You need to log in before you can comment on or make changes to this bug.