Bug 1119226

Fixed in:
https://github.com/oVirt/pthreading/commit/b42f0acba4ad5a8fb971733fedd295e7d075afbc

Please provide exact steps for reproduction/verification. If not possible would be enough to check python code if it contains the hack with Lock/_Lock class?

Could you provide exact steps to reproduce this issue? I can't reproduce it with ISCSI-based VM disk (writing into it and it got extended without pause) on 0.1.3-1?

This happens when you have some io error when reading from the inbox lv
on the master domain. The current code will log an error and try again,
while the old code was checking if a lock is locked, and because the 
locked() method was not implemented by the lock in pthreading < 0.1.3-3,
the thread would exit.

So to reproduce, you have to cause an io error when accessing the inbox
lv on the master domain.

One option to do this is to replace /usr/bin/dd with a script simulating
errors. Check the dd.fake attachment for details.

Created attachment 919911 [details]
Fake dd simulating errors when accessing the inbox

ok, 0.1.3-3. (with older version no msg for mailbox lv io error were logged.)

simulating io issue from inbox lv:

IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
IOError: [Errno 5] _handleRequests._checkForMail - Could not read mailbox: /rhev/data-center/b7ee9232-7844-4d06-9f66-d347cb9e0f66/mastersd/dom_md/inbox
libvirtEventLoop::INFO::2014-07-22 14:38:13,548::vm::4574::vm.Vm::(_onIOError) vmId=`cb025f1d-e18f-44cc-96f8-8aab6c99de8a`::abnormal vm stop device virtio-disk0 error eother

the VM gets paused.

stop simulating io issue makes the VM change status to Up.

Verified based on #6.

seems like we need to sort out the Installation instruction (inline as comment on dd.fake)

What worked for me was :
...
chmod +x dd.fake
cd /usr/bin
mv /bin/dd /usr/bin/dd.real
ln -sf /usr/bin/dd.fake /bin/dd
...

You discovered difference between Fedora and RHEL :)

(In reply to Gal Amado from comment #8)
> seems like we need to sort out the Installation instruction (inline as
> comment on dd.fake)
> 
> What worked for me was :
> ...
> chmod +x dd.fake
> cd /usr/bin
> mv /bin/dd /usr/bin/dd.real

This leaves you fro a moment without a dd program, so if vdsm try to run dd now, it will fail.

This is why I was using ln, which create a hard link of dd - you have now 2 dd
programs.

> ln -sf /usr/bin/dd.fake /bin/dd

And this line replaces one of the real dd programs with a symbolic link atomically.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0975.html