Bug 801967

Summary: RAID1 down conversion with a specified device fails
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: agk, dwysocha, heinzm, jbrassow, mbroz, msnitzer, prajnoha, prockai, thornber, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.95-2.el6 Doc Type: Bug Fix
Doc Text:
no documentation necessary.
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 15:02:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2012-03-10 00:09:13 UTC
Description of problem:
# Mirror -> RAID1 -> RAID1 down conversion

[root@taft-01 ~]# lvs -a -o +devices
  LV              VG        Attr     LSize   Log       Copy%  Devices
  cent            centipede mwi-a-m- 500.00m cent_mlog 100.00 cent_mimage_0(0),cent_mimage_1(0),cent_mimage_2(0)
  [cent_mimage_0] centipede iwi-aom- 500.00m                  /dev/sdh2(0)
  [cent_mimage_1] centipede iwi-aom- 500.00m                  /dev/sdh1(0)
  [cent_mimage_2] centipede iwi-aom- 500.00m                  /dev/sdg2(0)
  [cent_mlog]     centipede lwi-aom-   4.00m                  /dev/sdd2(0)

[root@taft-01 ~]# lvconvert --type raid1 centipede/cent
[root@taft-01 ~]# lvs -a -o +devices
  LV              VG        Attr     LSize   Log Copy%  Devices
  cent            centipede rwi-a-m- 500.00m     100.00 cent_rimage_0(0),cent_rimage_1(0),cent_rimage_2(0)
  [cent_rimage_0] centipede iwi-aor- 500.00m            /dev/sdh2(0)
  [cent_rimage_1] centipede iwi-aor- 500.00m            /dev/sdh1(0)
  [cent_rimage_2] centipede iwi-aor- 500.00m            /dev/sdg2(0)
  [cent_rmeta_0]  centipede ewi-aor-   4.00m            /dev/sdh2(125)
  [cent_rmeta_1]  centipede ewi-aor-   4.00m            /dev/sdh1(125)
  [cent_rmeta_2]  centipede ewi-aor-   4.00m            /dev/sdg2(125)

[root@taft-01 ~]# lvconvert -m 1 centipede/cent /dev/sdh1
  device-mapper: rename ioctl on centipede-cent_rimage_2 failed: Device or resource busy
  Failed to rename centipede-cent_rimage_2 (253:5) to centipede-cent_rimage_1
  Failed to resume centipede/cent after committing changes
  libdevmapper exiting with 7 device(s) still suspended.

[root@taft-01 ~]# lvs -a -o +devices
[DEADLOCK]


taft-01 kernel: device-mapper: ioctl: Unable to change name on mapped device centipede-cent_rimage_2 to one that already exists: centipede-cent_rimage_1
taft-01 kernel: INFO: task lvs:2301 blocked for more than 120 seconds.
taft-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
taft-01 kernel: lvs           D 0000000000000003     0  2301   2035 0x00000080
taft-01 kernel: ffff880218e09b18 0000000000000082 ffff880218e09ad8 ffffffffa000422e
taft-01 kernel: ffff880218e09ae8 0000000008443e0a ffff880218e09b08 ffff8802152d1780
taft-01 kernel: ffff880215787078 ffff880218e09fd8 000000000000f4e8 ffff880215787078
taft-01 kernel: Call Trace:
taft-01 kernel: [<ffffffffa000422e>] ? dm_table_unplug_all+0x8e/0x100 [dm_mod]
taft-01 kernel: [<ffffffff8109b6a9>] ? ktime_get_ts+0xa9/0xe0
taft-01 kernel: [<ffffffff814ed153>] io_schedule+0x73/0xc0
taft-01 kernel: [<ffffffff811b18ce>] __blockdev_direct_IO_newtrunc+0x6fe/0xb90
taft-01 kernel: [<ffffffff81257f1d>] ? get_disk+0x7d/0xf0
taft-01 kernel: [<ffffffff811b1dbe>] __blockdev_direct_IO+0x5e/0xd0
taft-01 kernel: [<ffffffff811ae6c0>] ? blkdev_get_blocks+0x0/0xc0
taft-01 kernel: [<ffffffff8126ccca>] ? kobject_get+0x1a/0x30
taft-01 kernel: [<ffffffff811af527>] blkdev_direct_IO+0x57/0x60
taft-01 kernel: [<ffffffff811ae6c0>] ? blkdev_get_blocks+0x0/0xc0
taft-01 kernel: [<ffffffff8111277b>] generic_file_aio_read+0x6bb/0x700
taft-01 kernel: [<ffffffff81213741>] ? avc_has_perm+0x71/0x90
taft-01 kernel: [<ffffffff8120d23f>] ? security_inode_permission+0x1f/0x30
taft-01 kernel: [<ffffffff811762ba>] do_sync_read+0xfa/0x140
taft-01 kernel: [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40
taft-01 kernel: [<ffffffff811aeafc>] ? block_ioctl+0x3c/0x40
taft-01 kernel: [<ffffffff811891e2>] ? vfs_ioctl+0x22/0xa0
taft-01 kernel: [<ffffffff81218d2b>] ? selinux_file_permission+0xfb/0x150
taft-01 kernel: [<ffffffff8120c0d6>] ? security_file_permission+0x16/0x20
taft-01 kernel: [<ffffffff81176cb5>] vfs_read+0xb5/0x1a0
taft-01 kernel: [<ffffffff810d4582>] ? audit_syscall_entry+0x272/0x2a0
taft-01 kernel: [<ffffffff81176df1>] sys_read+0x51/0x90


Version-Release number of selected component (if applicable):
2.6.32-220.4.2.el6.x86_64
lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012


How reproducible:
Everytime

Comment 1 Jonathan Earl Brassow 2012-03-14 18:20:28 UTC
Works for me.  Is your kernel up-to-date?

[root@bp-01 LVM2]# lvcreate -m2 -L 500M -n lv vg
  Logical volume "lv" created
[root@bp-01 LVM2]# devices vg; dmsetup ls | grep vg-
  LV            Copy%  Devices                                     
  lv            100.00 lv_mimage_0(0),lv_mimage_1(0),lv_mimage_2(0)
  [lv_mimage_0]        /dev/sdb1(0)                                
  [lv_mimage_1]        /dev/sdc1(0)                                
  [lv_mimage_2]        /dev/sdd1(0)                                
  [lv_mlog]            /dev/sdi1(0)                                
vg-lv_mimage_2  (253:6)
vg-lv   (253:7)
vg-lv_mimage_1  (253:5)
vg-lv_mimage_0  (253:4)
vg-lv_mlog      (253:3)
[root@bp-01 LVM2]# lvconvert --type raid1 vg/lv
[root@bp-01 LVM2]# devices vg; dmsetup ls | grep vg-
  LV            Copy%  Devices                                     
  lv            100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
  [lv_rimage_0]        /dev/sdb1(0)                                
  [lv_rimage_1]        /dev/sdc1(0)                                
  [lv_rimage_2]        /dev/sdd1(0)                                
  [lv_rmeta_0]         /dev/sdb1(125)                              
  [lv_rmeta_1]         /dev/sdc1(125)                              
  [lv_rmeta_2]         /dev/sdd1(125)                              
vg-lv   (253:7)
vg-lv_rmeta_2   (253:9)
vg-lv_rmeta_1   (253:8)
vg-lv_rmeta_0   (253:3)
vg-lv_rimage_2  (253:6)
vg-lv_rimage_1  (253:5)
vg-lv_rimage_0  (253:4)
[root@bp-01 LVM2]# lvconvert -m1 vg/lv
[root@bp-01 LVM2]# devices vg; dmsetup ls | grep vg-
  LV            Copy%  Devices                      
  lv            100.00 lv_rimage_0(0),lv_rimage_1(0)
  [lv_rimage_0]        /dev/sdb1(0)                 
  [lv_rimage_1]        /dev/sdc1(0)                 
  [lv_rmeta_0]         /dev/sdb1(125)               
  [lv_rmeta_1]         /dev/sdc1(125)               
vg-lv   (253:7)
vg-lv_rmeta_1   (253:8)
vg-lv_rmeta_0   (253:3)
vg-lv_rimage_1  (253:5)
vg-lv_rimage_0  (253:4)

Comment 2 Corey Marthaler 2012-03-15 15:46:00 UTC
Still fails for me with the latest kern.

[root@taft-01 ~]# lvconvert -m 1 cent/lv /dev/sdc1
  device-mapper: rename ioctl on cent-lv_rimage_2 failed: Device or resource busy
  Failed to rename cent-lv_rimage_2 (253:6) to cent-lv_rimage_1
  Failed to resume cent/lv after committing changes
  libdevmapper exiting with 7 device(s) still suspended.

[root@taft-01 ~]# lvs -a -o +devices
[HANG]

2.6.32-251.el6.x86_64

lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012

Comment 4 Jonathan Earl Brassow 2012-03-15 20:34:48 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
no documentation necessary.

Comment 6 Corey Marthaler 2012-03-22 20:16:52 UTC
Fix verified in the latest errata rpms.


[root@taft-01 ~]# lvs -a -o +devices
 LV            VG     Attr     LSize   Copy%  Devices
 lv            taft   rwi-a-m- 500.00m 100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
 [lv_rimage_0] taft   iwi-aor- 500.00m        /dev/sdb1(1)
 [lv_rimage_1] taft   iwi-aor- 500.00m        /dev/sdc1(1)
 [lv_rimage_2] taft   iwi-aor- 500.00m        /dev/sdd1(1)
 [lv_rmeta_0]  taft   ewi-aor-   4.00m        /dev/sdb1(0)
 [lv_rmeta_1]  taft   ewi-aor-   4.00m        /dev/sdc1(0)
 [lv_rmeta_2]  taft   ewi-aor-   4.00m        /dev/sdd1(0)

[root@taft-01 ~]# lvconvert -m 1 taft/lv /dev/sdc1

[root@taft-01 ~]# lvs -a -o +devices
 LV            VG     Attr     LSize   Copy%  Devices
 lv            taft   rwi-a-m- 500.00m 100.00 lv_rimage_0(0),lv_rimage_1(0)
 [lv_rimage_0] taft   iwi-aor- 500.00m        /dev/sdb1(1)
 [lv_rimage_1] taft   iwi-aor- 500.00m        /dev/sdd1(1)
 [lv_rmeta_0]  taft   ewi-aor-   4.00m        /dev/sdb1(0)
 [lv_rmeta_1]  taft   ewi-aor-   4.00m        /dev/sdd1(0)



2.6.32-251.el6.x86_64
lvm2-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
lvm2-libs-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
lvm2-cluster-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
device-mapper-libs-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
device-mapper-event-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
device-mapper-event-libs-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
cmirror-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012

Comment 8 errata-xmlrpc 2012-06-20 15:02:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0962.html