Description of problem: Having a setup with ceph backend with rhos16 Could not create a volume with muti-attach type... (it worked on rhos13) (overcloud) [stack@undercloud-0 ~]$ cinder type-show 44ff4b31-16e5-4ac1-ab57-57b0d9fdfcae +---------------------------------+----------------------------------------------------+ | Property | Value | +---------------------------------+----------------------------------------------------+ | description | None | | extra_specs | multiattach : <is> True | | | volume_backend_name : tripleo_ceph | | id | 44ff4b31-16e5-4ac1-ab57-57b0d9fdfcae | | is_public | True | | name | tempest-volume-typeTestVolumeMultiAttach-551572420 | | os-volume-type-access:is_public | True | | qos_specs_id | None | +---------------------------------+----------------------------------------------------+ When trying to create a volume - it fails. (overcloud) [stack@undercloud-0 ~]$ cinder list --all-tenants +--------------------------------------+----------------------------------+--------+------+------+----------------------------------------------------+----------+-------------+ | ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------------------------+--------+------+------+----------------------------------------------------+----------+-------------+ | b93cf462-2cdf-4aba-b7a2-d0aca9f725bd | 346ae2925a2c4fcd8fb162feedc9b8e7 | error | - | 1 | tempest-volume-typeTestVolumeMultiAttach-551572420 | false | | +--------------------------------------+----------------------------------+--------+------+------+----------------------------------------------------+----------+-------------+ From logs : vers/rbd.py:926 2019-11-14 08:57:38.034 22052 DEBUG cinder.volume.drivers.rbd [req-967dd415-8f2b-4edb-bd02-fe7945678dbe 22 4bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] connecting to openstaa ck@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/dd rivers/rbd.py:411 2019-11-14 08:57:38.173 22052 DEBUG cinder.volume.drivers.rbd [req-967dd415-8f2b-4edb-bd02-fe7945678dbe 22 4bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] connecting to openstaa ck@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/dd rivers/rbd.py:411 2019-11-14 08:57:38.326 22052 ERROR cinder.volume.drivers.rbd [req-967dd415-8f2b-4edb-bd02-fe7945678dbe 22 4bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] Error creating rbd imm age volume-057ca1dd-81ef-405c-b17f-d62c9cffc2df.: rbd.InvalidArgument: [errno 22] error updating featuress for image b'volume-057ca1dd-81ef-405c-b17f-d62c9cffc2df' 2019-11-14 08:57:39.252 7 INFO oslo_service.service [req-04fcde8a-f27b-4da2-8477-058cb7f95fc5 - - - - -] Child 22052 killed by signal 11 2019-11-14 08:57:39.258 7 DEBUG oslo_service.service [req-04fcde8a-f27b-4da2-8477-058cb7f95fc5 - - - - -]] Started child 22610 _start_child /usr/lib/python3.6/site-packages/oslo_service/service.py:579
Created attachment 1636165 [details] controller-0 logs
Stack trace of the segfault from running gdb on c-vol: #0 0x00007faadbaaaeb1 in void TracepointProvider::initialize<(anonymous namespace)::tracepoint_traits>(CephContext*) () from target:/lib64/librbd.so.1 #1 0x00007faadbab5276 in rbd_remove_with_progress () from target:/lib64/librbd.so.1 #2 0x00007faadc163327 in __pyx_pw_3rbd_3RBD_11remove () from target:/usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so #3 0x00007faaf4d64feb in _PyObject_FastCallDict () from target:/lib64/libpython3.6m.so.1.0 #4 0x00007faaf4d70790 in _PyObject_Call_Prepend () from target:/lib64/libpython3.6m.so.1.0 #5 0x00007faaf4d65623 in PyObject_Call () from target:/lib64/libpython3.6m.so.1.0 #6 0x00007faaf4e129f5 in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0 #7 0x00007faaf4deb0c8 in PyEval_EvalCodeEx () from target:/lib64/libpython3.6m.so.1.0 #8 0x00007faaf4dec0ee in function_call () from target:/lib64/libpython3.6m.so.1.0 #9 0x00007faaf4d65623 in PyObject_Call () from target:/lib64/libpython3.6m.so.1.0 #10 0x00007faaf4e129f5 in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0 #11 0x00007faaf4d9d61a in fast_function () from target:/lib64/libpython3.6m.so.1.0 #12 0x00007faaf4dd1e52 in call_function () from target:/lib64/libpython3.6m.so.1.0 #13 0x00007faaf4e10faa in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0 #14 0x00007faaf4d9d61a in fast_function () from target:/lib64/libpython3.6m.so.1.0 #15 0x00007faaf4dd1e52 in call_function () from target:/lib64/libpython3.6m.so.1.0 #16 0x00007faaf4e10faa in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0 #17 0x00007faaf4d6489a in _PyFunction_FastCallDict () from target:/lib64/libpython3.6m.so.1.0 #18 0x00007faaf4d6527e in _PyObject_FastCallDict () from target:/lib64/libpython3.6m.so.1.0 #19 0x00007faaf4d70790 in _PyObject_Call_Prepend () from target:/lib64/libpython3.6m.so.1.0 #20 0x00007faaf4d65623 in PyObject_Call () from target:/lib64/libpython3.6m.so.1.0 #21 0x00007faaf4e81902 in t_bootstrap () from target:/lib64/libpython3.6m.so.1.0 #22 0x00007faaf4e2edb4 in pythread_wrapper () from target:/lib64/libpython3.6m.so.1.0 #23 0x00007faaf4a4a2de in start_thread () from target:/lib64/libpthread.so.0 #24 0x00007faaf3ff1133 in clone () from target:/lib64/libc.so.6
More complete stack trace: Thread 12 "cinder-volume" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7faaa27fc700 (LWP 291728)] 0x00007faadbaaae82 in std::atomic_flag::test_and_set (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:176 176 /usr/include/c++/8/bits/atomic_base.h: No such file or directory. (gdb) bt #0 0x00007faadbaaae82 in std::atomic_flag::test_and_set (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:176 #1 ceph::version_1_0::spin_lock (lock=...) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/include/spinlock.h:50 #2 ceph::version_1_0::spinlock::lock (this=<optimized out>) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/include/spinlock.h:39 #3 std::lock_guard<ceph::version_1_0::spinlock>::lock_guard (__m=..., this=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:162 #4 CephContext::lookup_or_create_singleton_object<TracepointProvider::TypedSingleton<(TracepointProvider::Traits const&)(&(anonymous namespace)::tracepoint_traits)>, CephContext*&> ( drop_on_fork=false, name=..., this=0x707070707070707) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/common/ceph_context.h:174 #5 TracepointProvider::initialize<(TracepointProvider::Traits const&)(&(anonymous namespace)::tracepoint_traits)> (cct=0x707070707070707) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/common/TracepointProvider.h:59 #6 0x00007faadbab5276 in rbd_remove_with_progress (p=p@entry=0x7faa4c009860, name=name@entry=0x7faad3a3db90 "volume-7237a50b-8aad-4573-98bd-72d48dd35ebe", cb=cb@entry=0x7faadc1372d0 <__pyx_f_3rbd_no_op_progress_callback>, cbdata=cbdata@entry=0x0) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/librbd/librbd.cc:3160 #7 0x00007faadc163327 in __pyx_pf_3rbd_3RBD_10remove (__pyx_self=<optimized out>, __pyx_v_self=<optimized out>, __pyx_v_on_progress=<optimized out>, __pyx_v_name=b'volume-7237a50b-8aad-4573-98bd-72d48dd35ebe', __pyx_v_ioctx=<optimized out>) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/build/src/pybind/rbd3/pyrex/rbd.c:10013 #8 __pyx_pw_3rbd_3RBD_11remove (__pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/build/src/pybind/rbd3/pyrex/rbd.c:9832 #9 0x00007faaf4d64feb in _PyObject_FastCallDict (func=<cython_function_or_method at remote 0x7faadc4351b8>, args=0x7faaa27fb4e0, nargs=<optimized out>, kwargs={}) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2331 #10 0x00007faaf4d70790 in _PyObject_Call_Prepend (func=<cython_function_or_method at remote 0x7faadc4351b8>, obj=<optimized out>, args=(<rados.Ioctx at remote 0x7faad3af5f50>, 'volume-7237a50b-8aad-4573-98bd-72d48dd35ebe'), kwargs={}) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2373 #11 0x00007faaf4d65623 in PyObject_Call (func=<method at remote 0x7faad3a3bd88>, args=<optimized out>, kwargs=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2261 #12 0x00007faaf4e129f5 in do_call_core (kwdict={}, callargs=(<rados.Ioctx at remote 0x7faad3af5f50>, 'volume-7237a50b-8aad-4573-98bd-72d48dd35ebe'), func=<method at remote 0x7faad3a3bd88>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:5120 #13 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3404 #14 0x00007faaf4deb0c8 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=<optimized out>, kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3902 #15 PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4187 #16 0x00007faaf4dec0ee in function_call (func=func@entry=<function at remote 0x7faaeee11620>, arg=(), kw={}) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/funcobject.c:604 #17 0x00007faaf4d65623 in PyObject_Call (func=<function at remote 0x7faaeee11620>, args=<optimized out>, kwargs=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2261 #18 0x00007faaf4e129f5 in do_call_core (kwdict={}, callargs=(), func=<function at remote 0x7faaeee11620>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:5120 #19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3404 #20 0x00007faaf4d9d61a in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968 #21 fast_function (func=<optimized out>, stack=0x7faa7c000cd0, nargs=1, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968 #22 0x00007faaf4dd1e52 in call_function (pp_stack=0x7faaa27fbaa8, oparg=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4872 #23 0x00007faaf4e10faa in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3335 #24 0x00007faaf4d9d61a in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968 #25 fast_function (func=<optimized out>, stack=0x7faad3d1ad38, nargs=1, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968 #26 0x00007faaf4dd1e52 in call_function (pp_stack=0x7faaa27fbc58, oparg=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4872 #27 0x00007faaf4e10faa in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3335 #28 0x00007faaf4d6489a in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4933 #29 _PyFunction_FastCallDict (func=<optimized out>, args=0x7faaa27fbde0, nargs=1, kwargs=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:5035 #30 0x00007faaf4d6527e in _PyObject_FastCallDict (func=<function at remote 0x7faaf343db70>, args=0x7faaa27fbde0, nargs=<optimized out>, kwargs=0x0) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2310 #31 0x00007faaf4d70790 in _PyObject_Call_Prepend (func=<function at remote 0x7faaf343db70>, obj=<optimized out>, args=(), kwargs=0x0) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2373 #32 0x00007faaf4d65623 in PyObject_Call (func=<method at remote 0x7faa9f586cc8>, args=<optimized out>, kwargs=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2261 #33 0x00007faaf4e81902 in t_bootstrap (boot_raw=boot_raw@entry=0x7faad3d09da0) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Modules/_threadmodule.c:1000 #34 0x00007faaf4e2edb4 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/thread_pthread.h:205 --Type <RET> for more, q to quit, c to continue without paging-- #35 0x00007faaf4a4a2de in start_thread () from target:/lib64/libpthread.so.0 #36 0x00007faaf3ff1133 in clone () from target:/lib64/libc.so.6
New feature in 16 so not a blocker (was TP in 13).
*** Bug 1781774 has been marked as a duplicate of this bug. ***
This bugs currently prevents RBD multi attach to be supported on OSP 16. We expect the feature to be supported in 16.0.2
Hi, Installed latest code, rhos16. Run storage downstream with cinder multi-attach --> passed container cinder version = openstack-cinder-15.0.2-0.20200323201950.83cd511.el8ost.noarch Benny
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2161
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days