Bug 214302 - device mapper panics when attempting big mirror creation/activation
device mapper panics when attempting big mirror creation/activation
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: lvm2 (Show other bugs)
5.0
All Linux
medium Severity high
: ---
: ---
Assigned To: LVM and device-mapper development team
Corey Marthaler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-11-06 18:01 EST by Nate Straz
Modified: 2011-05-03 04:48 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-03 04:48:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch to fix problem (3.44 KB, text/x-patch)
2006-11-10 11:53 EST, Jonathan Earl Brassow
no flags Details

  None (edit)
Description Nate Straz 2006-11-06 18:01:21 EST
Description of problem:

While running lvm_config on large storage array (38TB), I hit the following
panic while doing a minor number change.

Unable to handle kernel NULL pointer dereference at 0000000000000040 RIP: 
 [<ffffffff881cad2d>] :dm_mod:dispatch_io+0x8b/0x16c
PGD 7654c067 PUD 7654d067 PMD 0 
Oops: 0000 [1] SMP 
last sysfs file: /block/ram0/dev
CPU 2 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6
cpufreq_ondemand video sbs i2c_ec button battery asus_acpi acpi_memhotplug ac
parport_pc lp parport ide_cd cdrom k8_edac sg i2c_nforce2 i2c_core forcedeth
edac_mc serio_raw shpchp pcspkr dm_snapshot dm_zero dm_mirror dm_mod qla2xxx
scsi_transport_fc sata_nv libata sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 6873, comm: lvchange Not tainted 2.6.18-1.2745.el5 #1
RIP: 0010:[<ffffffff881cad2d>]  [<ffffffff881cad2d>] :dm_mod:dispatch_io+0x8b/0x16c
RSP: 0018:ffff810077dedb78  EFLAGS: 00010206
RAX: 00000000000017c5 RBX: 0000000000000000 RCX: 0000000000000001
RDX: ffff810128416f40 RSI: 00000000040ce4d0 RDI: 0000000000000000
RBP: 00000000000017c5 R08: ffff810077dedc08 R09: 0000000000000001
R10: 0000000000000001 R11: ffffffff80042d6a R12: ffff810077dedc48
R13: ffff81012f6af4e8 R14: ffff810077dedc08 R15: 0000000000000010
FS:  00002aaaaaabb7e0(0000) GS:ffff81012fd279c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000040 CR3: 000000007609d000 CR4: 00000000000006e0
Process lvchange (pid: 6873, threadinfo ffff810077dec000, task ffff81007e5bc100)
Stack:  ffff81007f9f78e8 0000000000000001 ffffffff881cabfc ffffffff881cabc2
 ffff810000000000 ffffc2000029d000 0000000000000292 ffffffff8013d0d0
 0000000000000096 0000000000000292 000000000000000b ffff810077dedca0
Call Trace:
 [<ffffffff881cb01e>] :dm_mod:sync_io+0x82/0xeb
 [<ffffffff881cb0bf>] :dm_mod:dm_io_sync_vm+0x38/0x46
 [<ffffffff881e0a23>] :dm_mirror:disk_resume+0x41/0x1ea
 [<ffffffff881e0e83>] :dm_mirror:mirror_resume+0x17/0x4b
 [<ffffffff881c7b88>] :dm_mod:dm_table_resume_targets+0x5b/0x72
 [<ffffffff881c71c9>] :dm_mod:dm_resume+0x62/0x10d
 [<ffffffff881ca1e4>] :dm_mod:dev_suspend+0x167/0x18d
 [<ffffffff881ca9e2>] :dm_mod:ctl_ioctl+0x217/0x262
 [<ffffffff8003fe20>] do_ioctl+0x55/0x6b
 [<ffffffff8002fbad>] vfs_ioctl+0x248/0x261
 [<ffffffff8004a55c>] sys_ioctl+0x59/0x78
 [<ffffffff8005b641>] tracesys+0xd1/0xdc
DWARF2 unwinder stuck at tracesys+0xd1/0xdc
Leftover inexact backtrace:


Code: 48 8b 53 40 48 29 e8 48 89 03 49 8b 45 00 48 c7 43 48 4b af 
RIP  [<ffffffff881cad2d>] :dm_mod:dispatch_io+0x8b/0x16c
 RSP <ffff810077dedb78>
CR2: 0000000000000040
 <0>Kernel panic - not syncing: Fatal exception

output from the test program:
...
[lvm_single_config] creating lvm devices...
[lvm_single_config] make PV out of all avail partitions on marathon-01
[lvm_single_config] make VG out of all avail partitions on marathon-01
[lvm_single_config] activate VG on marathon-01
[lvm_single_config] make needed LVs on marathon-01
[lvm_single_config] lvcreate -l 3337336 -n linear_5_29630 linear_5_2963
[lvm_single_config] 
[lvm_single_config] deactivating volume group
[lvm_single_config] VOLUME MIRROR CONVERTION to 1 legged mirror on marathon-01
[lvm_single_config] lvconvert -m 1 /dev/linear_5_2963/linear_5_29630
[lvm_single_config] --- ALTERATION ITERATION 1 ---------------------------
[lvm_single_config] 
[lvm_single_config] VOLUME REDUCTION from 3337336 to 39957 on marathon-01
[lvm_single_config] VOLUME EXTENSION from 39957 to 141256 on marathon-01
[lvm_single_config] 
[lvm_single_config] VOLUME MINOR DEV NUM CHANGE to 168 on marathon-01
[lvm_single_config] verifying new minor num on marathon-01
[lvm_single_config] 
[lvm_single_config] 
[lvm_single_config] deactivating volume group
[lvm_single_config] EXPORTING VOLUME GROUP on marathon-01
[lvm_single_config] attempting to activate on marathon-01
[lvm_single_config]   Volume group "linear_5_2963" is exported
[lvm_single_config] IMPORTING VOLUME GROUP on marathon-01
[lvm_single_config] --- ALTERATION ITERATION 2 ---------------------------
[lvm_single_config] 
[lvm_single_config] VOLUME REDUCTION from 141256 to 82757 on marathon-01
[lvm_single_config] VOLUME EXTENSION from 82757 to 853143 on marathon-01
[lvm_single_config] 
[lvm_single_config] VOLUME MINOR DEV NUM CHANGE to 77 on marathon-01
[lvm_single_config] verifying new minor num on marathon-01
[lvm_single_config] 
[lvm_single_config] 
[lvm_single_config] deactivating volume group
[lvm_single_config] EXPORTING VOLUME GROUP on marathon-01
[lvm_single_config] attempting to activate on marathon-01
[lvm_single_config]   Volume group "linear_5_2963" is exported
[lvm_single_config] IMPORTING VOLUME GROUP on marathon-01
[lvm_single_config] --- ALTERATION ITERATION 3 ---------------------------
[lvm_single_config] 
[lvm_single_config] VOLUME REDUCTION from 853143 to 470628 on marathon-01
[lvm_single_config] VOLUME EXTENSION from 470628 to 3113990 on marathon-01
[lvm_single_config] 
[lvm_single_config] VOLUME MINOR DEV NUM CHANGE to 27 on marathon-01
[lvm_single_config] Didn't receive heartbeat for 120 seconds
[lvm_single_config] lvchange failed:
[lvm_single_config] qarsh root@marathon-01 lvchange -ay -f -My --major 255
--minor 27 /dev/linear_5_2963/linear_5_29630


Version-Release number of selected component (if applicable):


How reproducible:
unknown

Steps to Reproduce:
1. lvm_config -s 31897 -o marathon-01 -l /local/nstraz/svn/sts-trunk/sts-root -r
/usr/tests/sts -i 15

Vol_config files:
<volconfig name="random vol config">
        <vol name="linear_1_8060" vm="lvm" type="linear" legs="1"/>
</volconfig>
<volconfig name="random vol config">
        <vol name="linear_5_2963" vm="lvm" type="linear" legs="5" convert="1"/>
</volconfig>
  
Actual results:

See Above

Expected results:

No Panic.

Additional info:
Comment 1 Nate Straz 2006-11-06 18:11:19 EST
Once I get into this state, I hit it every time the system reboots.  Good thing
this is SAN storage. :)
Comment 2 Alasdair Kergon 2006-11-06 18:45:48 EST
Version numbers?

If your lvm2 package is not lvm2-2.02.12-7.el5 then we'd like to see it
reproduced with that version.

But before destroying the data, grab a copy of the contents of the mirror log LV
(it'll only be small) and attach it to this bug together with a copy of the
current metadata for the volume group.

Also attach a copy of the *actual* lvm2 commands run, not just the descriptive text.
Comment 3 Nate Straz 2006-11-06 19:13:09 EST
This was really easy to reproduce with lvm2-2.02.12-7.el5.

The last commands run was:
lvcreate -l 2503003 -n mirror_1_44330 -m 1  mirror_1_4433

Here is the metadata for the VG.

# vgdisplay -v
    Finding all volume groups
    Finding volume group "mirror_1_4433"
  --- Volume group ---
  VG Name               mirror_1_4433
  System ID             
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               0
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               38.19 TB
  PE Size               4.00 MB
  Total PE              10012012
  Alloc PE / Size       5006007 / 19.10 TB
  Free  PE / Size       5006005 / 19.10 TB
  VG UUID               UWVORV-KvdA-x9PU-asFw-5k1X-co3Y-Ib8Fe1
   
  --- Logical volume ---
  LV Name                /dev/mirror_1_4433/mirror_1_44330
  VG Name                mirror_1_4433
  LV UUID                HmBV4F-ICsa-bdsw-O550-o3hK-hw66-G33Pue
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                9.55 TB
  Current LE             2503003
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
   
  --- Logical volume ---
  LV Name                /dev/mirror_1_4433/mirror_1_44330_mlog
  VG Name                mirror_1_4433
  LV UUID                t07aP7-3KbP-fyh0-A3Pv-zO5O-lGwX-7kfLZO
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                4.00 MB
  Current LE             1
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
   
  --- Logical volume ---
  LV Name                /dev/mirror_1_4433/mirror_1_44330_mimage_0
  VG Name                mirror_1_4433
  LV UUID                wJuP91-sKHG-RVho-w3Ue-phM2-pDYO-RrYIB9
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                9.55 TB
  Current LE             2503003
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
   
  --- Logical volume ---
  LV Name                /dev/mirror_1_4433/mirror_1_44330_mimage_1
  VG Name                mirror_1_4433
  LV UUID                FDmRHG-ZNNj-V4o3-8bGw-hCt2-cGuh-WiAf11
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                9.55 TB
  Current LE             2503003
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
   
  --- Physical volumes ---
  PV Name               /dev/sdb1     
  PV UUID               RZMqde-KvZU-QlCE-1v2D-Z3QS-B2yo-svW8XP
  PV Status             allocatable
  Total PE / Free PE    2503003 / 0
   
  PV Name               /dev/sdc1     
  PV UUID               Bdt3iO-fIvc-YEmR-35hX-bVg1-zVfP-v2xy5U
  PV Status             allocatable
  Total PE / Free PE    2503003 / 0
   
  PV Name               /dev/sdd1     
  PV UUID               O33vWD-yec6-0cGX-M6o3-0hms-b6GW-9KVv1Q
  PV Status             allocatable
  Total PE / Free PE    2503003 / 2503003
   
  PV Name               /dev/sde1     
  PV UUID               JHWROw-yi0u-EgJW-87M6-2KAv-dyJY-UKMczp
  PV Status             allocatable
  Total PE / Free PE    2503003 / 2503002
   
Activating the VG in order to get at the mirror log volume caused a second node
to panic.  Any ideas on grabbing it?
Comment 4 Milan Broz 2006-11-08 10:45:08 EST
Please, if it is possible, can you attach log of last lvm command which cause
this with logging enabled ?

To enable full logging add these parameters
<command> -vvvv --config "log { activation = 1 }"

(or alternatively you can enable this in /etc/lvm.conf)


Comment 5 Corey Marthaler 2006-11-08 11:52:35 EST
There isn't much needed to recreate this bug, it just comes down to attempting
to create a big mirror.


[root@marathon-03 ~]# pvcreate  /dev/sd[bcde]1
  Physical volume "/dev/sdb1" successfully created
  Physical volume "/dev/sdc1" successfully created
  Physical volume "/dev/sdd1" successfully created
  Physical volume "/dev/sde1" successfully created
[root@marathon-03 ~]# vgcreate vg /dev/sd[bcde]1
  Volume group "vg" successfully created
[root@marathon-03 ~]# pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/sdb1  vg   lvm2 a-   9.55T 9.55T
  /dev/sdc1  vg   lvm2 a-   9.55T 9.55T
  /dev/sdd1  vg   lvm2 a-   9.55T 9.55T
  /dev/sde1  vg   lvm2 a-   9.55T 9.55T
[root@marathon-03 ~]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  vg     4   0   0 wz--n- 38.19T 38.19T
[root@marathon-03 ~]# lvcreate -l 2503003 -vvv -n mirror -m 1  vg
        Processing: lvcreate -l 2503003 -vvv -n mirror -m 1 vg
        O_DIRECT will be used
      Setting global/locking_type to 1
      File-based locking selected.
      Setting global/locking_dir to /var/lock/lvm
      Getting target version for mirror
        dm version   OF   [16384]
        dm versions   OF   [16384]
      Getting target version for mirror
        dm versions   OF   [16384]
      Setting activation/mirror_region_size to 512
      Locking /var/lock/lvm/V_vg WB
    Finding volume group "vg"
        /dev/ramdisk: Not using O_DIRECT
        Opened /dev/ramdisk RW
        /dev/ramdisk: block size is 4096 bytes
      /dev/ramdisk: No label detected
        Closed /dev/ramdisk
        Opened /dev/root RW O_DIRECT
        /dev/root: block size is 4096 bytes
      /dev/root: No label detected
        Closed /dev/root
        /dev/ram: Not using O_DIRECT
        Opened /dev/ram RW
        /dev/ram: block size is 4096 bytes
      /dev/ram: No label detected
        Closed /dev/ram
        Opened /dev/VolGroup00/LogVol01 RW O_DIRECT
        /dev/VolGroup00/LogVol01: block size is 4096 bytes
      /dev/VolGroup00/LogVol01: No label detected
        Closed /dev/VolGroup00/LogVol01
        /dev/ram2: Not using O_DIRECT
        Opened /dev/ram2 RW
        /dev/ram2: block size is 4096 bytes
      /dev/ram2: No label detected
        Closed /dev/ram2
        /dev/ram3: Not using O_DIRECT
        Opened /dev/ram3 RW
        /dev/ram3: block size is 4096 bytes
      /dev/ram3: No label detected
        Closed /dev/ram3
        /dev/ram4: Not using O_DIRECT
        Opened /dev/ram4 RW
        /dev/ram4: block size is 4096 bytes
      /dev/ram4: No label detected
        Closed /dev/ram4
        /dev/ram5: Not using O_DIRECT
        Opened /dev/ram5 RW
        /dev/ram5: block size is 4096 bytes
      /dev/ram5: No label detected
        Closed /dev/ram5
        /dev/ram6: Not using O_DIRECT
        Opened /dev/ram6 RW
        /dev/ram6: block size is 4096 bytes
      /dev/ram6: No label detected
        Closed /dev/ram6
        /dev/ram7: Not using O_DIRECT
        Opened /dev/ram7 RW
        /dev/ram7: block size is 4096 bytes
      /dev/ram7: No label detected
        Closed /dev/ram7
        /dev/ram8: Not using O_DIRECT
        Opened /dev/ram8 RW
        /dev/ram8: block size is 4096 bytes
      /dev/ram8: No label detected
        Closed /dev/ram8
        /dev/ram9: Not using O_DIRECT
        Opened /dev/ram9 RW
        /dev/ram9: block size is 4096 bytes
      /dev/ram9: No label detected
        Closed /dev/ram9
        /dev/ram10: Not using O_DIRECT
        Opened /dev/ram10 RW
        /dev/ram10: block size is 4096 bytes
      /dev/ram10: No label detected
        Closed /dev/ram10
        /dev/ram11: Not using O_DIRECT
        Opened /dev/ram11 RW
        /dev/ram11: block size is 4096 bytes
      /dev/ram11: No label detected
        Closed /dev/ram11
        /dev/ram12: Not using O_DIRECT
        Opened /dev/ram12 RW
        /dev/ram12: block size is 4096 bytes
      /dev/ram12: No label detected
        Closed /dev/ram12
        /dev/ram13: Not using O_DIRECT
        Opened /dev/ram13 RW
        /dev/ram13: block size is 4096 bytes
      /dev/ram13: No label detected
        Closed /dev/ram13
        /dev/ram14: Not using O_DIRECT
        Opened /dev/ram14 RW
        /dev/ram14: block size is 4096 bytes
      /dev/ram14: No label detected
        Closed /dev/ram14
        /dev/ram15: Not using O_DIRECT
        Opened /dev/ram15 RW
        /dev/ram15: block size is 4096 bytes
      /dev/ram15: No label detected
        Closed /dev/ram15
        Opened /dev/sdb1 RW O_DIRECT
        /dev/sdb1: block size is 512 bytes
      /dev/sdb1: lvm2 label detected
        lvmcache: /dev/sdb1: now orphaned
        /dev/sdb1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        lvmcache: /dev/sdb1: now in VG vg
        lvmcache: /dev/sdb1: setting vg VGID to rLAp1zTVoUvewcC4kQFaahesJjulf05m
        lvmcache: /dev/sdb1: VG vg: Set creation host to marathon-03.
        Opened /dev/sdc1 RW O_DIRECT
        /dev/sdc1: block size is 512 bytes
      /dev/sdc1: lvm2 label detected
        lvmcache: /dev/sdc1: now orphaned
        /dev/sdc1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        lvmcache: /dev/sdc1: now in VG vg (rLAp1zTVoUvewcC4kQFaahesJjulf05m)
        Opened /dev/sdd1 RW O_DIRECT
        /dev/sdd1: block size is 512 bytes
      /dev/sdd1: lvm2 label detected
        lvmcache: /dev/sdd1: now orphaned
        /dev/sdd1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        lvmcache: /dev/sdd1: now in VG vg (rLAp1zTVoUvewcC4kQFaahesJjulf05m)
        Opened /dev/sde1 RW O_DIRECT
        /dev/sde1: block size is 512 bytes
      /dev/sde1: lvm2 label detected
        lvmcache: /dev/sde1: now orphaned
        /dev/sde1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        lvmcache: /dev/sde1: now in VG vg (rLAp1zTVoUvewcC4kQFaahesJjulf05m)
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (1) from /dev/sdb1 at 4608 size 1086
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (1) from /dev/sdc1 at 4608 size 1086
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (1) from /dev/sdd1 at 4608 size 1086
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 4608 size 1086 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (1) from /dev/sde1 at 4608 size 1086
        /dev/sdb1 0:      0 2503003: NULL(0:0)
        /dev/sdc1 0:      0 2503003: NULL(0:0)
        /dev/sdd1 0:      0 2503003: NULL(0:0)
        /dev/sde1 0:      0 2503003: NULL(0:0)
    Archiving volume group "vg" metadata (seqno 1).
        Allowing allocation on /dev/sdb1 start PE 0 length 2503003
        Allowing allocation on /dev/sdc1 start PE 0 length 2503003
        Allowing allocation on /dev/sdd1 start PE 0 length 2503003
        Allowing allocation on /dev/sde1 start PE 0 length 2503003
    Creating logical volume mirror_mlog
        /dev/sdb1 0:      0 2503003: NULL(0:0)
        /dev/sdc1 0:      0 2503003: NULL(0:0)
        /dev/sdd1 0:      0 2503003: NULL(0:0)
        /dev/sde1 0:      0      1: mirror_mlog(0:0)
        /dev/sde1 1:      1 2503002: NULL(0:0)
        Writing vg metadata to /dev/sdb1 at 6144 len 1377
        Writing vg metadata to /dev/sdc1 at 6144 len 1377
        Writing vg metadata to /dev/sdd1 at 6144 len 1377
        Writing vg metadata to /dev/sde1 at 6144 len 1377
        Pre-Committing vg metadata (2) to /dev/sdb1 header at 4096
        Pre-Committing vg metadata (2) to /dev/sdc1 header at 4096
        Pre-Committing vg metadata (2) to /dev/sdd1 header at 4096
        Pre-Committing vg metadata (2) to /dev/sde1 header at 4096
    Creating volume group backup "/etc/lvm/backup/vg" (seqno 2).
        Writing vg metadata to /etc/lvm/backup/.lvm_marathon-03_2791_893496924
        Committing vg metadata (2)
        Renaming /etc/lvm/backup/vg.tmp to /etc/lvm/backup/vg
        Committing vg metadata (2) to /dev/sdb1 header at 4096
        Committing vg metadata (2) to /dev/sdc1 header at 4096
        Committing vg metadata (2) to /dev/sdd1 header at 4096
        Committing vg metadata (2) to /dev/sde1 header at 4096
        Locking LV
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw (R)
      Finding volume group for uuid
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdb1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdc1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdd1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sde1 at 6144 size 1377
    Found volume group "vg"
        Getting device info for vg-mirror_mlog
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw NF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw NF   [16384]
        dm info vg-mirror_mlog  NF   [16384]
      Locking memory
        memlock_count inc to 1
      Setting activation/missing_stripe_filler to /dev/ioerror
        Getting device info for vg-mirror_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw OF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw OF   [16384]
        dm info vg-mirror_mlog  OF   [16384]
        Getting device info for vg-mirror_mlog-real
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real OF   [16384]
        dm info vg-mirror_mlog-real  OF   [16384]
        Getting device info for vg-mirror_mlog-cow
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow OF   [16384]
        dm info vg-mirror_mlog-cow  OF   [16384]
        Getting device info for vg-mirror_mlog_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog OF   [16384]
        dm info vg-mirror_mlog_mlog  OF   [16384]
      Getting target version for linear
        dm versions   OF   [16384]
      Getting target version for striped
        dm versions   OF   [16384]
    Creating vg-mirror_mlog
        dm create vg-mirror_mlog
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw NF   [16384]
    Loading vg-mirror_mlog table
        Adding target: 0 8192 linear 8:65 384
        dm table   (253:2) OF   [16384]
        dm reload   (253:2) NF   [16384]
    Resuming vg-mirror_mlog (253:2)
        dm resume   (253:2) NF   [16384]
        Getting device info for vg-mirror_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw OF   [16384]
        dm deps   (253:2) OF   [16384]
        Getting device info for vg-mirror_mlog-real
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real OF   [16384]
        dm info vg-mirror_mlog-real  OF   [16384]
        Getting device info for vg-mirror_mlog-cow
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow OF   [16384]
        dm info vg-mirror_mlog-cow  OF   [16384]
        Getting device info for vg-mirror_mlog_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog OF   [16384]
        dm info vg-mirror_mlog_mlog  OF   [16384]
      Unlocking memory
        memlock_count dec to 0
      Creating directory /dev/vg
      Linking /dev/vg/mirror_mlog -> /dev/mapper/vg-mirror_mlog
    Clearing start of logical volume "mirror_mlog"
        /dev/vg/mirror_mlog: Added to device cache
        Opened /dev/vg/mirror_mlog RW O_DIRECT
        Wiping /dev/vg/mirror_mlog at sector 0 length 8192 sectors
        /dev/vg/mirror_mlog: block size is 4096 bytes
        Closed /dev/vg/mirror_mlog
    Writing log header to device, mirror_mlog
        Opened /dev/vg/mirror_mlog RW O_DIRECT
        /dev/vg/mirror_mlog: block size is 4096 bytes
        Closed /dev/vg/mirror_mlog
        Locking LV
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw (NL)
      Finding volume group for uuid
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdb1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdc1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdd1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sde1 at 6144 size 1377
    Found volume group "vg"
        Getting device info for vg-mirror_mlog
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw OF   [16384]
      Locking memory
        memlock_count inc to 1
        Getting device info for vg-mirror_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw OF   [16384]
        dm deps   (253:2) OF   [16384]
        Getting device info for vg-mirror_mlog-real
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-real OF   [16384]
        dm info vg-mirror_mlog-real  OF   [16384]
        Getting device info for vg-mirror_mlog-cow
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-cow OF   [16384]
        dm info vg-mirror_mlog-cow  OF   [16384]
        Getting device info for vg-mirror_mlog_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw-_mlog OF   [16384]
        dm info vg-mirror_mlog_mlog  OF   [16384]
        dm info   (253:2) OF   [16384]
    Removing vg-mirror_mlog (253:2)
        dm remove   (253:2) NF   [16384]
      Unlocking memory
        memlock_count dec to 0
      Removing link /dev/vg/mirror_mlog
      Removing directory /dev/vg
        Unlocking LV
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw
      Finding volume group for uuid
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdb1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdc1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sdd1 at 6144 size 1377
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 6144 size 1377 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (2) from /dev/sde1 at 6144 size 1377
    Found volume group "vg"
        Getting device info for vg-mirror_mlog
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw NF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw NF   [16384]
        dm info vg-mirror_mlog  NF   [16384]
    Creating logical volume mirror
    Creating logical volume mirror_mimage_0
    Creating logical volume mirror_mimage_1
        /dev/sdb1 0:      0 2503003: mirror_mimage_0(0:0)
        /dev/sdc1 0:      0 2503003: mirror_mimage_1(0:0)
        /dev/sdd1 0:      0 2503003: NULL(0:0)
        /dev/sde1 0:      0      1: mirror_mlog(0:0)
        /dev/sde1 1:      1 2503002: NULL(0:0)
        Writing vg metadata to /dev/sdb1 at 7680 len 2140
        Writing vg metadata to /dev/sdc1 at 7680 len 2140
        Writing vg metadata to /dev/sdd1 at 7680 len 2140
        Writing vg metadata to /dev/sde1 at 7680 len 2140
        Pre-Committing vg metadata (3) to /dev/sdb1 header at 4096
        Pre-Committing vg metadata (3) to /dev/sdc1 header at 4096
        Pre-Committing vg metadata (3) to /dev/sdd1 header at 4096
        Pre-Committing vg metadata (3) to /dev/sde1 header at 4096
    Creating volume group backup "/etc/lvm/backup/vg" (seqno 3).
        Writing vg metadata to /etc/lvm/backup/.lvm_marathon-03_2791_253372966
        Committing vg metadata (3)
        Renaming /etc/lvm/backup/vg.tmp to /etc/lvm/backup/vg
        Committing vg metadata (3) to /dev/sdb1 header at 4096
        Committing vg metadata (3) to /dev/sdc1 header at 4096
        Committing vg metadata (3) to /dev/sdd1 header at 4096
        Committing vg metadata (3) to /dev/sde1 header at 4096
        Locking LV
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b (R)
      Finding volume group for uuid
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (3) from /dev/sdb1 at 7680 size 2140
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (3) from /dev/sdc1 at 7680 size 2140
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (3) from /dev/sdd1 at 7680 size 2140
      /dev/sdb1: lvm2 label detected
        /dev/sdb1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdc1: lvm2 label detected
        /dev/sdc1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sdd1: lvm2 label detected
        /dev/sdd1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
      /dev/sde1: lvm2 label detected
        /dev/sde1: Found metadata at 7680 size 2140 for vg
(rLAp1z-TVoU-vewc-C4kQ-Faah-esJj-ulf05m)
        Read vg metadata (3) from /dev/sde1 at 7680 size 2140
    Found volume group "vg"
        Getting device info for vg-mirror
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b NF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b NF   [16384]
        dm info vg-mirror  NF   [16384]
      Locking memory
        memlock_count inc to 1
        Getting device info for vg-mirror
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b OF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b OF   [16384]
        dm info vg-mirror  OF   [16384]
        Getting device info for vg-mirror-real
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-real]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-real OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-real OF   [16384]
        dm info vg-mirror-real  OF   [16384]
        Getting device info for vg-mirror-cow
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-cow]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-cow OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-cow OF   [16384]
        dm info vg-mirror-cow  OF   [16384]
        Getting device info for vg-mirror_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-_mlog]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-_mlog OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b-_mlog OF   [16384]
        dm info vg-mirror_mlog  OF   [16384]
        Getting device info for vg-mirror_mimage_0
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0 OF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0 OF   [16384]
        dm info vg-mirror_mimage_0  OF   [16384]
        Getting device info for vg-mirror_mimage_0-real
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-real]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-real OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-real OF   [16384]
        dm info vg-mirror_mimage_0-real  OF   [16384]
        Getting device info for vg-mirror_mimage_0-cow
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-cow]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-cow OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-cow OF   [16384]
        dm info vg-mirror_mimage_0-cow  OF   [16384]
        Getting device info for vg-mirror_mimage_0_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-_mlog]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-_mlog OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0-_mlog OF   [16384]
        dm info vg-mirror_mimage_0_mlog  OF   [16384]
        Getting device info for vg-mirror_mimage_1
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1 OF   [16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1 OF   [16384]
        dm info vg-mirror_mimage_1  OF   [16384]
        Getting device info for vg-mirror_mimage_1-real
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-real]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-real OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-real OF   [16384]
        dm info vg-mirror_mimage_1-real  OF   [16384]
        Getting device info for vg-mirror_mimage_1-cow
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-cow]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-cow OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-cow OF   [16384]
        dm info vg-mirror_mimage_1-cow  OF   [16384]
        Getting device info for vg-mirror_mimage_1_mlog
[LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-_mlog]
        dm info 
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-_mlog OF  
[16384]
        dm info 
rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1-_mlog OF   [16384]
        dm info vg-mirror_mimage_1_mlog  OF   [16384]
      Setting activation/mirror_region_size to 512
    Creating vg-mirror_mlog
        dm create vg-mirror_mlog
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mJSwhChA8JupWO49E6RTvJHcIK2sz9zhw NF   [16384]
    Loading vg-mirror_mlog table
        Adding target: 0 8192 linear 8:65 384
        dm table   (253:2) OF   [16384]
        dm reload   (253:2) NF   [16384]
    Resuming  (253:2)
        dm resume   (253:2) NF   [16384]
    Creating vg-mirror_mimage_0
        dm create vg-mirror_mimage_0
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mq9pwVi61aH1pwhJIyXRa7jVfXgce0pJ0 NF   [16384]
    Loading vg-mirror_mimage_0 table
        Adding target: 0 20504600576 linear 8:17 384
        dm table   (253:3) OF   [16384]
        dm reload   (253:3) NF   [16384]
    Resuming  (253:3)
        dm resume   (253:3) NF   [16384]
    Creating vg-mirror_mimage_1
        dm create vg-mirror_mimage_1
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mN9qBD5jHh5rGN7hxLh7jQomFaSFDlyv1 NF   [16384]
    Loading vg-mirror_mimage_1 table
        Adding target: 0 20504600576 linear 8:33 384
        dm table   (253:4) OF   [16384]
        dm reload   (253:4) NF   [16384]
    Resuming  (253:4)
        dm resume   (253:4) NF   [16384]
    Creating vg-mirror
        dm create vg-mirror
LVM-rLAp1zTVoUvewcC4kQFaahesJjulf05mzoN1Gda86yVKsAhsQZt7Aaz0hKANQ09b NF   [16384]
    Loading vg-mirror table
        Adding target: 0 20504600576 mirror disk 3 253:2 1024 block_on_error 2
253:3 0 253:4 0
        dm table   (253:5) OF   [16384]
        dm reload   (253:5) NF   [16384]
    Resuming vg-mirror (253:5)
        dm resume   (253:5) NF   [16384]


[PANIC!]
Comment 6 Jonathan Earl Brassow 2006-11-10 11:53:56 EST
Created attachment 140907 [details]
Patch to fix problem

This patch also fixes 214604
Comment 7 Jonathan Earl Brassow 2006-11-10 11:58:19 EST
'assigning' to agk to include patch in packages and then move to 'post'.
Comment 8 Alasdair Kergon 2006-12-05 12:54:10 EST
improved region size logic was included in a recent lvm2 build - is this fixed?
Comment 9 Nate Straz 2006-12-20 18:58:13 EST
No, this is not fixed.  I hit it with lvm2-2.02.16-3.el5.
Comment 11 Alasdair Kergon 2011-02-11 15:47:14 EST
Let's take fresh logs then, please, from the current RHEL5 package, and do a fresh investigation.
Comment 12 Milan Broz 2011-05-03 04:48:13 EDT
BUg reported in 2006, no fresh info for more that a month, closing.

Please reopen if you can reproduce it with recent system.

Note You need to log in before you can comment on or make changes to this bug.