Bug 1420530

Summary: [RHCeph 2.2/10.2.5-22.el7cp] test_rbd.TestExclusiveLock.test_break_lock ... ./common/WorkQueue.h: In function 'ThreadPool::PointerWQ<T>::~PointerWQ() [with T = librbd::AioImageRequest<librbd::ImageCtx>]'
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vasu Kulkarni <vakulkar>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Vasu Kulkarni <vakulkar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.2CC: ceph-eng-bugs, ceph-qe-bugs, hnallurv, kdreyer
Target Milestone: rc   
Target Release: 2.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.5-24.el7cp Ubuntu: ceph_10.2.5-17redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-14 15:49:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vasu Kulkarni 2017-02-08 21:55:45 UTC
Description of problem:

during librbd test, I see the following trace during ExclusiveLock

test_rbd.TestClone.test_resize_flatten_multi_level ... ok
test_rbd.TestClone.test_resize_io ... ok
test_rbd.TestClone.test_resize_stat ... ok
test_rbd.TestClone.test_stat ... ok
test_rbd.TestClone.test_unprotect_with_children ... ok
test_rbd.TestClone.test_unprotected ... ok
test_rbd.TestClone.test_with_params ... ok
test_rbd.TestClone.test_with_params2 ... ok
test_rbd.TestClone.test_with_params3 ... SKIP
test_rbd.TestClone.test_write ... ok
test_rbd.TestExclusiveLock.test_acquire_release_lock ... ok
test_rbd.TestExclusiveLock.test_break_lock ... ./common/WorkQueue.h: In function 'ThreadPool::PointerWQ<T>::~PointerWQ() [with T = librbd::AioImageRequest<librbd::ImageCtx>]' thread 7fdd76eeb740 time 2017-02-08 12:23:49.815328
./common/WorkQueue.h: 350: FAILED assert(m_processing == 0)
 ceph version 10.2.5-22.el7cp (5cec6848b914e87dd6178e559dedae8a37cc08a3)
 1: (()+0x2ab6c5) [0x7fdd541bd6c5]
 2: (()+0x524f8) [0x7fdd53f644f8]
 3: (()+0x876f5) [0x7fdd53f996f5]
 4: (()+0xa4daf) [0x7fdd53fb6daf]
 5: (()+0xb0e3d) [0x7fdd53fc2e3d]
 6: (rbd_close()+0x32) [0x7fdd53f785d2]
 7: (()+0x1b788) [0x7fdd5d89a788]
 8: (PyEval_EvalFrameEx()+0x6b7c) [0x7fdd769f487c]
 9: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 10: (()+0x70798) [0x7fdd76980798]
 11: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 12: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 13: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd]
 14: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 15: (()+0x7088d) [0x7fdd7698088d]
 16: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 17: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 18: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 19: (()+0x70798) [0x7fdd76980798]
 20: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 21: (()+0x5a8d5) [0x7fdd7696a8d5]
 22: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 23: (()+0xa2c27) [0x7fdd769b2c27]
 24: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 25: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036]
 26: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd]
 27: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 28: (()+0x7088d) [0x7fdd7698088d]
 29: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 30: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 31: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 32: (()+0x70798) [0x7fdd76980798]
 33: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 34: (()+0x5a8d5) [0x7fdd7696a8d5]
 35: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 36: (()+0xa2c27) [0x7fdd769b2c27]
 37: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 38: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036]
 39: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 40: (()+0x7088d) [0x7fdd7698088d]
 41: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 42: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 43: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 44: (()+0x70798) [0x7fdd76980798]
 45: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 46: (()+0x5a8d5) [0x7fdd7696a8d5]
 47: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 48: (()+0xa2c27) [0x7fdd769b2c27]
 49: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 50: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036]
 51: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 52: (()+0x7088d) [0x7fdd7698088d]
 53: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 54: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 55: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 56: (()+0x70798) [0x7fdd76980798]
 57: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 58: (()+0x5a8d5) [0x7fdd7696a8d5]
 59: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 60: (()+0xa2c27) [0x7fdd769b2c27]
 61: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 62: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036]
 63: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 64: (()+0x7088d) [0x7fdd7698088d]
 65: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 66: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 67: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 68: (()+0x70798) [0x7fdd76980798]
 69: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 70: (()+0x5a8d5) [0x7fdd7696a8d5]
 71: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 72: (()+0xa2c27) [0x7fdd769b2c27]
 73: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 74: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036]
 75: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd]
 76: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd]
 77: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 78: (()+0x7088d) [0x7fdd7698088d]
 79: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 80: (()+0x5a8d5) [0x7fdd7696a8d5]
 81: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 82: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd]
 83: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 84: (()+0x70798) [0x7fdd76980798]
 85: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 86: (()+0x5a8d5) [0x7fdd7696a8d5]
 87: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 88: (()+0xa2887) [0x7fdd769b2887]
 89: (()+0xa159f) [0x7fdd769b159f]
 90: (PyObject_Call()+0x43) [0x7fdd7695b8e3]
 91: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036]
 92: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d]
 93: (PyEval_EvalCode()+0x32) [0x7fdd769f6f42]
 94: (()+0x10037f) [0x7fdd76a1037f]
 95: (PyRun_FileExFlags()+0x7e) [0x7fdd76a1153e]
 96: (PyRun_SimpleFileExFlags()+0xe9) [0x7fdd76a127c9]
 97: (Py_Main()+0xc9f) [0x7fdd76a2391f]
 98: (__libc_start_main()+0xf5) [0x7fdd75c4bb35]
 99: /usr/bin/python2() [0x400721]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Version-Release number of selected component (if applicable):

10.2.5-22.el7cp

How reproducible:
Always

Comment 2 Vasu Kulkarni 2017-02-08 22:01:47 UTC
In some cases I have seen them SKIPPED but in many other places I see the traceback

full logs: http://magna002.ceph.redhat.com/vasu-2017-02-06_22:51:03-rbd-jewel---basic-multi/264462/teuthology.log

Comment 8 Vasu Kulkarni 2017-02-21 17:24:33 UTC
Verified in 10.2.5-26.el7cp

Comment 10 errata-xmlrpc 2017-03-14 15:49:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0514.html