Description of problem: during librbd test, I see the following trace during ExclusiveLock test_rbd.TestClone.test_resize_flatten_multi_level ... ok test_rbd.TestClone.test_resize_io ... ok test_rbd.TestClone.test_resize_stat ... ok test_rbd.TestClone.test_stat ... ok test_rbd.TestClone.test_unprotect_with_children ... ok test_rbd.TestClone.test_unprotected ... ok test_rbd.TestClone.test_with_params ... ok test_rbd.TestClone.test_with_params2 ... ok test_rbd.TestClone.test_with_params3 ... SKIP test_rbd.TestClone.test_write ... ok test_rbd.TestExclusiveLock.test_acquire_release_lock ... ok test_rbd.TestExclusiveLock.test_break_lock ... ./common/WorkQueue.h: In function 'ThreadPool::PointerWQ<T>::~PointerWQ() [with T = librbd::AioImageRequest<librbd::ImageCtx>]' thread 7fdd76eeb740 time 2017-02-08 12:23:49.815328 ./common/WorkQueue.h: 350: FAILED assert(m_processing == 0) ceph version 10.2.5-22.el7cp (5cec6848b914e87dd6178e559dedae8a37cc08a3) 1: (()+0x2ab6c5) [0x7fdd541bd6c5] 2: (()+0x524f8) [0x7fdd53f644f8] 3: (()+0x876f5) [0x7fdd53f996f5] 4: (()+0xa4daf) [0x7fdd53fb6daf] 5: (()+0xb0e3d) [0x7fdd53fc2e3d] 6: (rbd_close()+0x32) [0x7fdd53f785d2] 7: (()+0x1b788) [0x7fdd5d89a788] 8: (PyEval_EvalFrameEx()+0x6b7c) [0x7fdd769f487c] 9: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 10: (()+0x70798) [0x7fdd76980798] 11: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 12: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 13: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd] 14: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 15: (()+0x7088d) [0x7fdd7698088d] 16: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 17: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 18: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 19: (()+0x70798) [0x7fdd76980798] 20: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 21: (()+0x5a8d5) [0x7fdd7696a8d5] 22: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 23: (()+0xa2c27) [0x7fdd769b2c27] 24: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 25: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036] 26: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd] 27: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 28: (()+0x7088d) [0x7fdd7698088d] 29: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 30: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 31: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 32: (()+0x70798) [0x7fdd76980798] 33: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 34: (()+0x5a8d5) [0x7fdd7696a8d5] 35: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 36: (()+0xa2c27) [0x7fdd769b2c27] 37: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 38: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036] 39: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 40: (()+0x7088d) [0x7fdd7698088d] 41: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 42: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 43: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 44: (()+0x70798) [0x7fdd76980798] 45: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 46: (()+0x5a8d5) [0x7fdd7696a8d5] 47: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 48: (()+0xa2c27) [0x7fdd769b2c27] 49: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 50: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036] 51: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 52: (()+0x7088d) [0x7fdd7698088d] 53: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 54: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 55: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 56: (()+0x70798) [0x7fdd76980798] 57: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 58: (()+0x5a8d5) [0x7fdd7696a8d5] 59: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 60: (()+0xa2c27) [0x7fdd769b2c27] 61: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 62: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036] 63: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 64: (()+0x7088d) [0x7fdd7698088d] 65: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 66: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 67: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 68: (()+0x70798) [0x7fdd76980798] 69: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 70: (()+0x5a8d5) [0x7fdd7696a8d5] 71: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 72: (()+0xa2c27) [0x7fdd769b2c27] 73: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 74: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036] 75: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd] 76: (PyEval_EvalFrameEx()+0x67bd) [0x7fdd769f44bd] 77: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 78: (()+0x7088d) [0x7fdd7698088d] 79: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 80: (()+0x5a8d5) [0x7fdd7696a8d5] 81: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 82: (PyEval_EvalFrameEx()+0x17fd) [0x7fdd769ef4fd] 83: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 84: (()+0x70798) [0x7fdd76980798] 85: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 86: (()+0x5a8d5) [0x7fdd7696a8d5] 87: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 88: (()+0xa2887) [0x7fdd769b2887] 89: (()+0xa159f) [0x7fdd769b159f] 90: (PyObject_Call()+0x43) [0x7fdd7695b8e3] 91: (PyEval_EvalFrameEx()+0x2336) [0x7fdd769f0036] 92: (PyEval_EvalCodeEx()+0x7ed) [0x7fdd769f6e3d] 93: (PyEval_EvalCode()+0x32) [0x7fdd769f6f42] 94: (()+0x10037f) [0x7fdd76a1037f] 95: (PyRun_FileExFlags()+0x7e) [0x7fdd76a1153e] 96: (PyRun_SimpleFileExFlags()+0xe9) [0x7fdd76a127c9] 97: (Py_Main()+0xc9f) [0x7fdd76a2391f] 98: (__libc_start_main()+0xf5) [0x7fdd75c4bb35] 99: /usr/bin/python2() [0x400721] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Version-Release number of selected component (if applicable): 10.2.5-22.el7cp How reproducible: Always
In some cases I have seen them SKIPPED but in many other places I see the traceback full logs: http://magna002.ceph.redhat.com/vasu-2017-02-06_22:51:03-rbd-jewel---basic-multi/264462/teuthology.log
Verified in 10.2.5-26.el7cp
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0514.html