Bug 821454

When taking a snapshot of a mirror (either dm or md) LVM2 can cause two instances of the mirror table to be live simultaneously, which can lead to corruption or crashes.  The sequence of ioctls that LVM2 issues needs to be fixed to avoid this.


+++ This bug was initially created as a clone of Bug #818371 +++

Description of problem:
SCENARIO - [write_to_snap_merge]
Create snaps of origin with fs data, verify data on snaps, change data on snaps, merge data back to origin, verify origin data
Making origin volume
** creating RAID1 origin **
Placing an ext filesystem on origin volume
mke2fs 1.41.12 (17-May-2010)
Mounting origin volume

Writing files to /mnt/origin
checkit starting with:
CREATE
Num files:          500
Random Seed:        8057
Verify XIOR Stream: /tmp/original.20616
Working dir:        /mnt/origin

Checking files on /mnt/origin
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/original.20616
Working dir:        /mnt/origin


Making 5 snapshots of the origin volume, mounting, and verifying original data
lvcreate -s /dev/snapper/origin -c 128 -n merge1 -L 2G

[MACHINE LOCKS UP AND REQUIRES A REBOOT]


Version-Release number of selected component (if applicable):
2.6.32-269.el6.x86_64
lvm2-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
lvm2-libs-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
lvm2-cluster-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
udev-147-2.41.el6    BUILT: Thu Mar  1 13:01:08 CST 2012
device-mapper-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
device-mapper-libs-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
device-mapper-event-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
device-mapper-event-libs-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
cmirror-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012


How reproducible:
Most of the time



--- Additional comment from jbrassow on 2012-05-08 09:36:38 EDT ---

I've added print statements to CTR, DTR, SUSPEND, and RESUME operations.  They print out the operation being performed and the memory position of the dm_target struct.  The print-outs seem to indicate that the ordering of the suspend/resume operations is wrong:
1) Table1 CTR
2) Table1 RESUME
3) Table2 CTR
4) Table2 RESUME
5) Snap CTR
6) Table1 SUSPEND
7) Table2 SUSPEND
8) Table2 RESUME
9) Snap RESUME
10) Table1 DTR

Table2 should probably not be resumed before Table1 is suspended.  Actual output below.


device-mapper: raid: RAID CTR - 0xffffc90014aa0040
device-mapper: raid: Superblocks created for new array
md/raid1:mdX: not clean -- starting background reconstruction
md/raid1:mdX: active with 2 out of 2 mirrors
Choosing daemon_sleep default (5 sec)
created bitmap (4 pages) for device mdX
device-mapper: raid: RAID RESUME, 0xffffc90014aa0040
mdX: bitmap file is out of date, doing full recovery
mdX: bitmap initialized from disk: read 1/1 pages, set 8192 of 8192 bits
md: resync of RAID array mdX
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) f.
md: using 128k window, over a total of 4194304k.
md: mdX: resync done.
[device-mapper: raid: RAID CTR - 0xffffc9001506b040
md/raid1:mdX: active with 2 out of 2 mirrors
created bitmap (4 pages) for device mdX
device-mapper: raid: RAID RESUME, 0xffffc9001506b040
mdX: bitmap initialized from disk: read 1/1 pages, set 1 of 8192 bits
device-mapper: snapshots: SNAP CTR
device-mapper: raid: RAID PRESUSPEND, 0xffffc90014aa0040
device-mapper: raid: RAID POSTSUSPEND, 0xffffc90014aa0040
device-mapper: raid: RAID PRESUSPEND, 0xffffc9001506b040
device-mapper: raid: RAID POSTSUSPEND, 0xffffc9001506b040
device-mapper: raid: RAID RESUME, 0xffffc9001506b040
device-mapper: snapshots: SNAP PRERESUME
device-mapper: snapshots: SNAP RESUME
device-mapper: raid: RAID DTR, 0xffffc90014aa0040
------------[ cut here ]------------
kernel BUG at kernel/timer.c:960!

--- Additional comment from jbrassow on 2012-05-08 11:40:18 EDT ---

Created attachment 583023 [details]
output of 'lvcreate -s' command showing possible bad ordering of suspend/resume

Here is the state of the machine before the command was issued (and the command issued to get the output attached here).

[root@bp-01 ~]# lvcreate --type raid1 -m 1 -L 500M -n lv vg
  Logical volume "lv" created
[root@bp-01 ~]# dmsetup table | grep vg-
vg-lv: 0 1024000 raid raid1 3 0 region_size 1024 2 253:3 253:4 253:5 253:6
vg-lv_rmeta_1: 0 8192 linear 8:33 2048
vg-lv_rmeta_0: 0 8192 linear 8:17 2048
vg-lv_rimage_1: 0 1024000 linear 8:33 10240
vg-lv_rimage_0: 0 1024000 linear 8:17 10240
[root@bp-01 ~]# lvcreate -s vg/lv -L 500M -n snap -vvvvv >& output.txt
[


--- Additional comment from agk on 2012-05-14 10:11:28 EDT ---

Ordering problem apparently also reproducible using original mirror implementation.

--- Additional comment from agk on 2012-05-14 10:18:23 EDT ---

Two separate problems:

1. userspace sending invalid sequence of operations;
- this needs fixing for 6.3

2. kernel crash when it receives the invalid sequence.
- if it's easy to detect the situation and return an error, we should do that - but otherwise it doesn't matter as it makes no sense for userspace to issue that sequence of ioctls.