Bug 1772557

Summary:

[rhos16 ceph] Unable to create multi-attached volume

Product:

Red Hat OpenStack

Reporter:

bkopilov <bkopilov>

Component:

openstack-cinder

Assignee:

Jon Bernard <jobernar>

Status:

CLOSED ERRATA

QA Contact:

Tzach Shefi <tshefi>

Severity:

high

Docs Contact:

Chuck Copello <ccopello>

Priority:

high

Version:

16.0 (Train)

CC:

eharney, gcharot, jobernar, jparker, jvisser, pgrist, schhabdi, tshefi

Target Milestone:

Keywords:

Triaged

Target Release:

16.0 (Train on RHEL 8.1)

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

openstack-cinder-15.0.2-0.20200208000013.99321eb.el8ost

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-05-14 12:11:11 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1790753

Attachments:

Description	Flags
controller-0 logs	none

Description bkopilov 2019-11-14 15:35:43 UTC

Description of problem:

Having a setup with ceph backend with rhos16 
Could not create a volume with muti-attach type...  (it worked on rhos13)

(overcloud) [stack@undercloud-0 ~]$ cinder type-show 44ff4b31-16e5-4ac1-ab57-57b0d9fdfcae
+---------------------------------+----------------------------------------------------+
| Property                        | Value                                              |
+---------------------------------+----------------------------------------------------+
| description                     | None                                               |
| extra_specs                     | multiattach : <is> True                            |
|                                 | volume_backend_name : tripleo_ceph                 |
| id                              | 44ff4b31-16e5-4ac1-ab57-57b0d9fdfcae               |
| is_public                       | True                                               |
| name                            | tempest-volume-typeTestVolumeMultiAttach-551572420 |
| os-volume-type-access:is_public | True                                               |
| qos_specs_id                    | None                                               |
+---------------------------------+----------------------------------------------------+


When trying to create a volume - it fails.

(overcloud) [stack@undercloud-0 ~]$ cinder list --all-tenants
+--------------------------------------+----------------------------------+--------+------+------+----------------------------------------------------+----------+-------------+
| ID                                   | Tenant ID                        | Status | Name | Size | Volume Type                                        | Bootable | Attached to |
+--------------------------------------+----------------------------------+--------+------+------+----------------------------------------------------+----------+-------------+
| b93cf462-2cdf-4aba-b7a2-d0aca9f725bd | 346ae2925a2c4fcd8fb162feedc9b8e7 | error  | -    | 1    | tempest-volume-typeTestVolumeMultiAttach-551572420 | false    |             |
+--------------------------------------+----------------------------------+--------+------+------+----------------------------------------------------+----------+-------------+



From logs :
vers/rbd.py:926
2019-11-14 08:57:38.034 22052 DEBUG cinder.volume.drivers.rbd [req-967dd415-8f2b-4edb-bd02-fe7945678dbe 22
4bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] connecting to openstaa
ck@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/dd
rivers/rbd.py:411
2019-11-14 08:57:38.173 22052 DEBUG cinder.volume.drivers.rbd [req-967dd415-8f2b-4edb-bd02-fe7945678dbe 22
4bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] connecting to openstaa
ck@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). _do_conn /usr/lib/python3.6/site-packages/cinder/volume/dd
rivers/rbd.py:411
2019-11-14 08:57:38.326 22052 ERROR cinder.volume.drivers.rbd [req-967dd415-8f2b-4edb-bd02-fe7945678dbe 22
4bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] Error creating rbd imm
age volume-057ca1dd-81ef-405c-b17f-d62c9cffc2df.: rbd.InvalidArgument: [errno 22] error updating featuress
 for image b'volume-057ca1dd-81ef-405c-b17f-d62c9cffc2df'
2019-11-14 08:57:39.252 7 INFO oslo_service.service [req-04fcde8a-f27b-4da2-8477-058cb7f95fc5 - - - - -]  
Child 22052 killed by signal 11
2019-11-14 08:57:39.258 7 DEBUG oslo_service.service [req-04fcde8a-f27b-4da2-8477-058cb7f95fc5 - - - - -]]
 Started child 22610 _start_child /usr/lib/python3.6/site-packages/oslo_service/service.py:579

Comment 1 bkopilov 2019-11-14 16:15:28 UTC

Created attachment 1636165 [details]
controller-0 logs

Comment 2 Eric Harney 2019-11-14 16:46:06 UTC

Stack trace of the segfault from running gdb on c-vol:

#0  0x00007faadbaaaeb1 in void TracepointProvider::initialize<(anonymous namespace)::tracepoint_traits>(CephContext*) () from target:/lib64/librbd.so.1
#1  0x00007faadbab5276 in rbd_remove_with_progress () from target:/lib64/librbd.so.1
#2  0x00007faadc163327 in __pyx_pw_3rbd_3RBD_11remove () from target:/usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so
#3  0x00007faaf4d64feb in _PyObject_FastCallDict () from target:/lib64/libpython3.6m.so.1.0
#4  0x00007faaf4d70790 in _PyObject_Call_Prepend () from target:/lib64/libpython3.6m.so.1.0
#5  0x00007faaf4d65623 in PyObject_Call () from target:/lib64/libpython3.6m.so.1.0
#6  0x00007faaf4e129f5 in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0
#7  0x00007faaf4deb0c8 in PyEval_EvalCodeEx () from target:/lib64/libpython3.6m.so.1.0
#8  0x00007faaf4dec0ee in function_call () from target:/lib64/libpython3.6m.so.1.0
#9  0x00007faaf4d65623 in PyObject_Call () from target:/lib64/libpython3.6m.so.1.0
#10 0x00007faaf4e129f5 in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0
#11 0x00007faaf4d9d61a in fast_function () from target:/lib64/libpython3.6m.so.1.0
#12 0x00007faaf4dd1e52 in call_function () from target:/lib64/libpython3.6m.so.1.0
#13 0x00007faaf4e10faa in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0
#14 0x00007faaf4d9d61a in fast_function () from target:/lib64/libpython3.6m.so.1.0
#15 0x00007faaf4dd1e52 in call_function () from target:/lib64/libpython3.6m.so.1.0
#16 0x00007faaf4e10faa in _PyEval_EvalFrameDefault () from target:/lib64/libpython3.6m.so.1.0
#17 0x00007faaf4d6489a in _PyFunction_FastCallDict () from target:/lib64/libpython3.6m.so.1.0
#18 0x00007faaf4d6527e in _PyObject_FastCallDict () from target:/lib64/libpython3.6m.so.1.0
#19 0x00007faaf4d70790 in _PyObject_Call_Prepend () from target:/lib64/libpython3.6m.so.1.0
#20 0x00007faaf4d65623 in PyObject_Call () from target:/lib64/libpython3.6m.so.1.0
#21 0x00007faaf4e81902 in t_bootstrap () from target:/lib64/libpython3.6m.so.1.0
#22 0x00007faaf4e2edb4 in pythread_wrapper () from target:/lib64/libpython3.6m.so.1.0
#23 0x00007faaf4a4a2de in start_thread () from target:/lib64/libpthread.so.0
#24 0x00007faaf3ff1133 in clone () from target:/lib64/libc.so.6

Comment 3 Eric Harney 2019-11-14 16:55:02 UTC

More complete stack trace:

Thread 12 "cinder-volume" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7faaa27fc700 (LWP 291728)]
0x00007faadbaaae82 in std::atomic_flag::test_and_set (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:176
176	/usr/include/c++/8/bits/atomic_base.h: No such file or directory.
(gdb) bt
#0  0x00007faadbaaae82 in std::atomic_flag::test_and_set (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:176
#1  ceph::version_1_0::spin_lock (lock=...) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/include/spinlock.h:50
#2  ceph::version_1_0::spinlock::lock (this=<optimized out>) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/include/spinlock.h:39
#3  std::lock_guard<ceph::version_1_0::spinlock>::lock_guard (__m=..., this=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:162
#4  CephContext::lookup_or_create_singleton_object<TracepointProvider::TypedSingleton<(TracepointProvider::Traits const&)(&(anonymous namespace)::tracepoint_traits)>, CephContext*&> (
    drop_on_fork=false, name=..., this=0x707070707070707) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/common/ceph_context.h:174
#5  TracepointProvider::initialize<(TracepointProvider::Traits const&)(&(anonymous namespace)::tracepoint_traits)> (cct=0x707070707070707)
    at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/common/TracepointProvider.h:59
#6  0x00007faadbab5276 in rbd_remove_with_progress (p=p@entry=0x7faa4c009860, name=name@entry=0x7faad3a3db90 "volume-7237a50b-8aad-4573-98bd-72d48dd35ebe", 
    cb=cb@entry=0x7faadc1372d0 <__pyx_f_3rbd_no_op_progress_callback>, cbdata=cbdata@entry=0x0) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/src/librbd/librbd.cc:3160
#7  0x00007faadc163327 in __pyx_pf_3rbd_3RBD_10remove (__pyx_self=<optimized out>, __pyx_v_self=<optimized out>, __pyx_v_on_progress=<optimized out>, 
    __pyx_v_name=b'volume-7237a50b-8aad-4573-98bd-72d48dd35ebe', __pyx_v_ioctx=<optimized out>) at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/build/src/pybind/rbd3/pyrex/rbd.c:10013
#8  __pyx_pw_3rbd_3RBD_11remove (__pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>)
    at /usr/src/debug/ceph-14.2.4-42.el8cp.x86_64/build/src/pybind/rbd3/pyrex/rbd.c:9832
#9  0x00007faaf4d64feb in _PyObject_FastCallDict (func=<cython_function_or_method at remote 0x7faadc4351b8>, args=0x7faaa27fb4e0, nargs=<optimized out>, kwargs={})
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2331
#10 0x00007faaf4d70790 in _PyObject_Call_Prepend (func=<cython_function_or_method at remote 0x7faadc4351b8>, obj=<optimized out>, 
    args=(<rados.Ioctx at remote 0x7faad3af5f50>, 'volume-7237a50b-8aad-4573-98bd-72d48dd35ebe'), kwargs={}) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2373
#11 0x00007faaf4d65623 in PyObject_Call (func=<method at remote 0x7faad3a3bd88>, args=<optimized out>, kwargs=<optimized out>)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2261
#12 0x00007faaf4e129f5 in do_call_core (kwdict={}, callargs=(<rados.Ioctx at remote 0x7faad3af5f50>, 'volume-7237a50b-8aad-4573-98bd-72d48dd35ebe'), func=<method at remote 0x7faad3a3bd88>)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:5120
#13 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3404
#14 0x00007faaf4deb0c8 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=<optimized out>, 
    kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3902
#15 PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4187
#16 0x00007faaf4dec0ee in function_call (func=func@entry=<function at remote 0x7faaeee11620>, arg=(), kw={}) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/funcobject.c:604
#17 0x00007faaf4d65623 in PyObject_Call (func=<function at remote 0x7faaeee11620>, args=<optimized out>, kwargs=<optimized out>)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2261
#18 0x00007faaf4e129f5 in do_call_core (kwdict={}, callargs=(), func=<function at remote 0x7faaeee11620>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:5120
#19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3404
#20 0x00007faaf4d9d61a in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968
#21 fast_function (func=<optimized out>, stack=0x7faa7c000cd0, nargs=1, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968
#22 0x00007faaf4dd1e52 in call_function (pp_stack=0x7faaa27fbaa8, oparg=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4872
#23 0x00007faaf4e10faa in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3335
#24 0x00007faaf4d9d61a in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968
#25 fast_function (func=<optimized out>, stack=0x7faad3d1ad38, nargs=1, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4968
#26 0x00007faaf4dd1e52 in call_function (pp_stack=0x7faaa27fbc58, oparg=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4872
#27 0x00007faaf4e10faa in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:3335
#28 0x00007faaf4d6489a in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:4933
#29 _PyFunction_FastCallDict (func=<optimized out>, args=0x7faaa27fbde0, nargs=1, kwargs=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/ceval.c:5035
#30 0x00007faaf4d6527e in _PyObject_FastCallDict (func=<function at remote 0x7faaf343db70>, args=0x7faaa27fbde0, nargs=<optimized out>, kwargs=0x0)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2310
#31 0x00007faaf4d70790 in _PyObject_Call_Prepend (func=<function at remote 0x7faaf343db70>, obj=<optimized out>, args=(), kwargs=0x0)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2373
#32 0x00007faaf4d65623 in PyObject_Call (func=<method at remote 0x7faa9f586cc8>, args=<optimized out>, kwargs=<optimized out>)
    at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Objects/abstract.c:2261
#33 0x00007faaf4e81902 in t_bootstrap (boot_raw=boot_raw@entry=0x7faad3d09da0) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Modules/_threadmodule.c:1000
#34 0x00007faaf4e2edb4 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3-3.6.8-15.1.el8.x86_64/Python/thread_pthread.h:205
--Type <RET> for more, q to quit, c to continue without paging--
#35 0x00007faaf4a4a2de in start_thread () from target:/lib64/libpthread.so.0
#36 0x00007faaf3ff1133 in clone () from target:/lib64/libc.so.6

Comment 9 Gregory Charot 2019-11-28 11:15:07 UTC

New feature in 16 so not a blocker (was TP in 13).

Comment 12 Eric Harney 2019-12-10 15:00:15 UTC

*** Bug 1781774 has been marked as a duplicate of this bug. ***

Comment 18 Gregory Charot 2020-01-24 16:26:06 UTC

This bugs currently prevents RBD multi attach to be supported on OSP 16. We expect the feature to be supported in 16.0.2

Comment 22 bkopilov 2020-04-02 17:27:48 UTC

Hi, 
Installed latest code, rhos16.
Run storage downstream with cinder multi-attach  --> passed

container cinder version = openstack-cinder-15.0.2-0.20200323201950.83cd511.el8ost.noarch


Benny

Comment 28 errata-xmlrpc 2020-05-14 12:11:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2161

Comment 29 Red Hat Bugzilla 2023-09-14 05:46:10 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days