Created attachment 1028001 [details]
OSD  logs

Writing from multiple clients isn't supposed to work, but also should not deadlock/livelock. Since this is something that shouldn't be done in the first place, I don't think this should block the release. We should fix it at a lower priority.

Also, the flags show the object map became invalid somehow. That's a separate bug. Do you have the steps it took to become invalid, and can you reproduce it?

Josh,
w.r.t comment 3.

Mandatory exclusive lock is per-image and used to prevent the overlapping of writing into the same rbd image by using some locking technique.

Only one client holds the lock at a time, and only that client is allowed to do any kind of write to the image and other client will request the lock from the current lock holder.

So why writing from multiple client on the same image should not work ?

Please correct my understanding here, i may be totally wrong here.

(In reply to Josh Durgin from comment #4)
> Also, the flags show the object map became invalid somehow. That's a
> separate bug. Do you have the steps it took to become invalid, and can you
> reproduce it?

Josh,
I have reproduced and filed a new bug for it.

https://bugzilla.redhat.com/show_bug.cgi?id=1224174

Created attachment 1028916 [details]
deadlock backtrace

Federico, are you ok re-targeting this bz to 1.3.1?

Ken and Federico, before making the decision to fix in 1.3.1 can we please wait for Josh to reply to Tanay's comment 5?

w.r.t. comment 5, the use case for exclusive locking is not for two VMs sharing the same disk (nor three as another BZ tests).  The VMs won't expect the disks to be changing underneath them as caching would prevent one VM from seeing the changes of another.  If you needed to share a disk between two VMs for the purposes of using a clustered file system (GFS, OCFS), you would just use a standard (non-exclusive) RBD image.

This feature was designed to support the object map feature as well as future features such as RBD mirroring.  The only time one should expect the same disk to be opened by two different QEMU instances at the same time is in the live-migration case where QEMU temporarily has the image open on the source and destination nodes at the same time for a short period of time.

What Jason said addresses comment 5. To be more explicit, locking isn't meant to make it safe to use rbd from multiple clients at once. This will still fail with any local fs on top due to caching within the vm's kernel. Reads in one vm would get cached data that may have changed, and there's no way to invalidate the vm's cache if another vm wrote to the same disk.

This bug is still valid, but I wouldn't treat it as a blocker for 1.3.0, and would rather address it in z-stream or 1.3.1.

Hi Jason, what timeframe are we looking at for landing this fix?

commit 704c0e09479d435ecbb84693a786c60694732b19
Author: Jason Dillaman <dillaman>
Date:   Thu May 7 12:51:49 2015 -0400

    librbd: retry lock requests periodically until acquired
    
    If the exclusive lock owner acks the lock release request but crashes
    before it actually releases the lock, the requestor will wait forever.
    Therefore, after a certain timeout, retry the request again until it
    succeeds.
    
    Fixes: #11537
    Backport: hammer
    Signed-off-by: Jason Dillaman <dillaman>
    (cherry picked from commit 37c74e6e5274208d1b6efaf315afec03ea7eaa82)

commit dbaaed9cf99121d0c97c22a695a7a6e872a11f48
Author: Jason Dillaman <dillaman>
Date:   Thu May 7 12:35:36 2015 -0400

    librbd: don't hold owner_lock for write during flush
    
    The various IO callback codepaths will attempt to take
    the lock, which will result in deadlock since the flush
    cannot complete.
    
    Backport: hammer
    Fixes: #11537
    Signed-off-by: Jason Dillaman <dillaman>
    
    (cherry picked from commit 2b6d0633d5c89de3a557cdd72621dfc19f0540de)

# git tag --contains dbaaed9cf99121d0c97c22a695a7a6e872a11f48
v0.94.4

Thanks!

Jason,
I am still able to reproduce this Bug.

Ceph Build:
[ceph@cephqe9 ~]$ rpm -qa |grep ceph
ceph-common-0.94.5-4.el7cp.x86_64
ceph-osd-0.94.5-4.el7cp.x86_64
ceph-0.94.5-4.el7cp.x86_64
ceph-radosgw-0.94.5-4.el7cp.x86_64
ceph-selinux-0.94.5-4.el7cp.x86_64
[ceph@cephqe9 ~]$ rpm -qa |grep rados
librados2-0.94.5-4.el7cp.x86_64
ceph-radosgw-0.94.5-4.el7cp.x86_64
python-rados-0.94.5-4.el7cp.x86_64

Followed the exact steps mentioned in Comment #1

IO is getting halted on both the VMs ( i.e. Spare disk on which i am writing )

Every 0.1s: sudo rbd -p Tanay-RBD lock list testingClone_new9191                                                                                                 Fri Jan 29 23:04:56 2016

There is 1 exclusive lock on this image.
Locker        ID                   Address
client.269331 auto 139931622384896 10.70.44.52:0/2135169

Lock is still being held

The lock holder is stuck waiting for the cache to flush -- but I don't see any other threads blocking that operation from completing.

#0  0x00007f445827d6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f4455896753 in Wait (mutex=..., this=0x7f4433190440) at common/Cond.h:55
#2  librbd::ImageCtx::flush_cache (this=this@entry=0x7f445eaa2000) at librbd/ImageCtx.cc:675
#3  0x00007f44558b1a77 in librbd::_flush (ictx=0x7f445eaa2000) at librbd/internal.cc:3322
#4  0x00007f44558a5f5a in librbd::ImageWatcher::release_lock (this=this@entry=0x7f445e96ed80) at librbd/ImageWatcher.cc:393
#5  0x00007f44558a60be in librbd::ImageWatcher::notify_release_lock (this=0x7f445e96ed80) at librbd/ImageWatcher.cc:586
#6  0x00007f44558876ca in operator() (a0=<optimized out>, this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#7  FunctionContext::finish (this=<optimized out>, r=<optimized out>) at include/Context.h:461
#8  0x00007f445587f689 in Context::complete (this=0x7f4461ebc060, r=<optimized out>) at include/Context.h:65
#9  0x00007f445587f689 in Context::complete (this=0x7f4461ebdb30, r=<optimized out>) at include/Context.h:65
#10 0x00007f44558f99c8 in Finisher::finisher_thread_entry (this=0x7f445e9b77c0) at common/Finisher.cc:59
#11 0x00007f4458279dc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f44508f321d in clone () from /lib64/libc.so.6

Can you please re-run the test under the following conditions:

1) set SElinux to permissive mode -- your QEMU processes aren't creating the RBD log
2) add "admin socket = /var/log/libvirt/qemu/rbd-$pid.$cctid.asok" to your ceph.conf

Jason pointed out to me that ceph.conf has the following setting under [global]:

log file = /var/log/libvirt/qemu/rbd-$pid.$cctid.log

Since this is under [global], this causes all Ceph logs to go to this file, not just RBD logs, but OSD logs as well. And SELinux is denying the OSD from writing its logs there.

Please change the following:

1) Move this "log file" setting to a [client] setting, so that only the librbd client will write its logs to /var/log/libvirt/qemu.

2) Please create [osd/mon] sections for the settings that should apply only to Mons or only to OSDs.

It is ok to test with SELinux in permissive mode to see if the problem vanishes, but ideally we would make this work in enforcing mode.

Disregard SElinux comment, qemu user did not have permissions to write to /var/lib/libvirt/qemu

Marking this as Failed, as the same steps are causing Bug 1303728.
After discussion with Jason, the new crash/hang was different than this one 

So filed a new defect.

We need clear downstream documentation for setting up librbd with qemu and libvirt, doing logging, etc. Without clear instructions, QE cannot conclusively verify these librbd bugs that require configuring the logs and admin socket.

Setting this bug to depend on the docs one (bug 1305421)