Bug 1278920

A customer periodically takes a snapshot of the logical volume containing the root filesystem.  Several RHEL 6.7 systems have hung and had all applications become unresponsive when using lvremove to remove the snapshot.

A vmcore was captured, and lvremove was found to be stuck waiting for a page fault.  At the time lvremove triggered a page fault, it had already suspended the DM devices for the root logical volume and snapshot.  lvremove was deadlocked as the pagefault needed data from the root filesystem, but the root filesystem couldn't be read until lvremove finished its operations and resumed the root logical volume.


The issue appears to be a regression starting with the 2.02.118 releases of lvm2 for RHEL 6.7.  lvremove did not have any of its memory mlocked into physical memory.  Under a test with strace running, lvremove was found to be passing a length of 0 to all calls to mlock():

...
mlock(0x7fc06c125000, 0)                = 0
mlock(0x7fc06c33b000, 0)                = 0
mlock(0x7fc06c33c000, 0)                = 0
...

Tested older versions, including lvm2-2.02.111-2.el6_6.6, did not show this behavior.  Instead, mlock was being passed proper lengths for the regions to lock into memory with RHEL6.6 and older versions of lvm2.

...
mlock(0x400000, 1044480)                = 0
mlock(0x6fe000, 49152)                  = 0
mlock(0x70a000, 98304)                  = 0
...

The bug appears to be from a change to _maps_line() in lib/mm/memlock.c related to valgrind defines, specifically the code shortly before mlock() is called:

#ifdef HAVE_VALGRIND
        /*
         * Valgrind is continually eating memory while executing code
         * so we need to deactivate check of locked memory size
         */
#ifndef VALGRIND_POOL
        if (RUNNING_ON_VALGRIND)
#endif
                sz -= sz; /* = 0, but avoids getting warning about dead assigment */

#endif

With HAVE_VALGRIND defined and VALGRIND_POOL now defined from an option passed to ./configure in lvm2.spec, the "sz -=sz;" line is always invoked and sets a 0 size.   This 0 size is then passed to mlock(), breaking the use of mlock().  With lvremove not locked into memory, it can page fault in the middle of its critical section and deadlock itself and hang anything else needing the root filesystem.


Version-Release number of selected component (if applicable):
lvm2-2.02.118-3.el6_7.2.x86_64

How reproducible:
Deadlock is very random from needing a page fault at a critical time.

Steps to Reproduce:
1.  Create a snapshot of an logical volume
2.  Run "lvremove" under strace to remove the snapshot.
3.  strace data will show mlock() calls with a length parameter of 0 when the bug occurs.

Actual results:
lvremove can deadlock when removing a snapshot for a logical volume containing the root filesystem.

Expected results:
lvremove should remove the snapshot without risk of a deadlock.