Description of problem: I'm trying to set up a nested LVM configuration with DRBD, see for example http://www.drbd.org/users-guide-emb/s-nested-lvm.html . Creating logical volumes on top of a DRBD device fails. Version-Release number of selected component (if applicable): lvm2-2.02.48-2.fc11.i586 kernel-2.6.30.8-64.fc11.i586 How reproducible: stable, happens always. Steps to Reproduce: 1. set up a DRBD device 2. create a physical volume on it: # pvcreate /dev/drbd0 Physical volume "/dev/drbd0" successfully created 3. create a volume group on it: # vgcreate replicated /dev/drbd0 Volume group "replicated" successfully created # vgscan Reading all physical volumes. This may take a while... Found volume group "replicated" using metadata type lvm2 Found volume group "vg0" using metadata type lvm2 Found volume group "vg1" using metadata type lvm2 # vgchange -a y replicated 0 logical volume(s) in volume group "replicated" now active 4. try to create a logical volum: # lvcreate --name foo --size 32G replicated device-mapper: reload ioctl failed: Invalid argument Aborting. Failed to activate new LV to wipe the start of it. Actual results: LV not created. Error messages on stdout: device-mapper: reload ioctl failed: Invalid argument Aborting. Failed to activate new LV to wipe the start of it. Syslog messages: # Oct 24 21:44:01 gemini kernel: device-mapper: table: 253:11: linear: dm-linear: Device lookup failed Oct 24 21:44:01 gemini kernel: device-mapper: ioctl: error adding target to table Expected results: LV created, no error messages. Additional info: I am pretty sure that used to work in older versions (probably F9 or F10), but unfortunately I don't have any records of this setup any more. There are similar bug reports from other distributions, so this seems to be a generic (kernel?) issue, see for example http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=533436
As per the Debian bug, run the tool with -vvvv, look for the line similar to this one: Adding target: 0 8388608 linear 252:2 384 Take the '252:2' there as major:minor and investigate the state of that device. Report all its attributes - use 'blockdev', find it in /sys/block etc. Check if anything has it open. Check its size etc. etc. Also test with udev disabled in case that's interfering.
Created attachment 366101 [details] Running pvcreate, vgcreate and lvcreate with "-vvvv" + additional info
OK, I attached the requested additional information - the only thing I cannot do right now is testing without udev - I don't know how to disable udev (and still have a resonably "normal" Fedora 11 running). Please advice if really needed - although I don't see where udev would interfere. [Note this is from another system, actually the peer; problem is the very same though.] Please let me know if any additional information is requqired. Wolfgang Denk, wd
Please can you also attach output (tarball) from lvmdump? (from some system where is /dev/drbd* device present and lvm fails)
Created attachment 366106 [details] Output of "lvmdump -a -m"
Thanks. There is something strange: - you create pvcreate /dev/drbd0 (/dev/drbd0 is major:minor 147:0) - but in metadata PV as /dev/dm-2 (which is 253:2 and in fact it is vg0/drbd0 volume mapped to 8:3 - which is /dev/sda3) So now we need to find where this confusion happens.
seems one thing still missing in lvmdump - please can you paste output of these files? cat /proc/devices /proc/partitions Thanks.
-> cat /proc/devices Character devices: 1 mem 4 /dev/vc/0 4 tty 4 ttyS 5 /dev/tty 5 /dev/console 5 /dev/ptmx 7 vcs 9 st 10 misc 13 input 14 sound 21 sg 29 fb 86 ch 99 ppdev 116 alsa 128 ptm 136 pts 162 raw 180 usb 188 ttyUSB 189 usb_device 202 cpu/msr 203 cpu/cpuid 206 osst 226 drm 248 firewire 249 hidraw 250 usb_endpoint 251 usbmon 252 bsg 253 pcmcia 254 rtc Block devices: 1 ramdisk 2 fd 259 blkext 7 loop 8 sd 9 md 11 sr 65 sd 66 sd 67 sd 68 sd 69 sd 70 sd 71 sd 128 sd 129 sd 130 sd 131 sd 132 sd 133 sd 134 sd 135 sd 147 drbd 253 device-mapper 254 mdp -> cat /proc/partitions major minor #blocks name 8 0 244198584 sda 8 1 128488 sda1 8 2 4192965 sda2 8 3 239874547 sda3 8 16 390711384 sdb 8 17 390708801 sdb1 253 0 33554432 dm-0 253 1 83886080 dm-1 253 3 390705152 dm-3 253 2 100663296 dm-2 147 0 100660188 drbd0
We should probably add some especial case for DRBD similar to MDinto lvm code. But for the time being, you can blacklist underlying device for DRBD, see lvm.conf filter setting . When you create new PV on /dev/drbdX device, lvm should see only PV on /dev/drbd and not the underlying device. (run vgscan ; pvs and you see something like this: # pvs PV VG Fmt Attr PSize PFree /dev/drbd0 vg_test lvm2 a- 1020.00M 620.00M and no duplicate warnings like Found duplicate PV TQrcpi4AAEzMGDHYY1cbwsByUkpLuEjj: using /dev/sdf not /dev/drbd0) Then DRBD PV works as expected. (Your configuration is more complicated, because you are using LV for backend DRBD device - but it still should work with proper filter setting.) Maybe for testing, explicitly name your PVs in lvm.conf. For your config it can be: filter = [ "a/drbd/", "a/sda/", "a/sdb/", "r/.*/" ] (or similar rule). Then run vgscan to refresh device cache and try again.
I'm afraid I don't get what you mean, or it doesn't work. I don't see any "Found duplicate PV" warnings: # diff /etc/lvm/lvm.conf.ORIG /etc/lvm/lvm.conf 53c53,55 < filter = [ "a/.*/" ] --- > #filter = [ "a/.*/" ] > # For DEBD setup, accept only DRBD and "SCSI" disks > filter = [ "a/drbd/", "a/sda/", "a/sdb/", "r/.*/" ] # vgscan Reading all physical volumes. This may take a while... Found volume group "vg1" using metadata type lvm2 Found volume group "vg0" using metadata type lvm2 # pvs PV VG Fmt Attr PSize PFree /dev/sda3 vg0 lvm2 a- 228.75G 20.75G /dev/sdb1 vg1 lvm2 a- 372.61G 0 # pvcreate /dev/drbd0 Physical volume "/dev/drbd0" successfully created # vgcreate replicated /dev/drbd0 Volume group "replicated" successfully created # vgscan Reading all physical volumes. This may take a while... Found volume group "vg1" using metadata type lvm2 Found volume group "vg0" using metadata type lvm2 Found volume group "replicated" using metadata type lvm2 # pvs PV VG Fmt Attr PSize PFree /dev/dm-2 replicated lvm2 a- 96.00G 96.00G /dev/sda3 vg0 lvm2 a- 228.75G 20.75G /dev/sdb1 vg1 lvm2 a- 372.61G 0 # lvcreate --name Mail --size 32G replicated device-mapper: reload ioctl failed: Invalid argument Aborting. Failed to activate new LV to wipe the start of it. Oct 27 13:10:13 nyx kernel: device-mapper: table: 253:4: linear: dm-linear: Device lookup failed Oct 27 13:10:13 nyx kernel: device-mapper: ioctl: error adding target to table As far as I can tell nothing has changed. Did I miss something?
(In reply to comment #10) > # pvs > PV VG Fmt Attr PSize PFree > /dev/dm-2 replicated lvm2 a- 96.00G 96.00G this is stil wrong, it uses directly LV instead of drbd device. Can you try filter = [ "a|^/dev/drbd|", "a/sda/", "a/sdb/", "r/.*/" ] I forgot you have drbd in LV name :)
Indeed - this gets it working: # diff /etc/lvm/lvm.conf.ORIG /etc/lvm/lvm.conf 53c53,55 < filter = [ "a/.*/" ] --- > #filter = [ "a/.*/" ] > # For DEBD setup, accept only DRBD and "SCSI" disks > filter = [ "a|^/dev/drbd|", "a/sda/", "a/sdb/", "r/.*/" ] # vgscan Reading all physical volumes. This may take a while... Found volume group "vg1" using metadata type lvm2 Found volume group "vg0" using metadata type lvm2 # pvs PV VG Fmt Attr PSize PFree /dev/sda3 vg0 lvm2 a- 228.75G 20.75G /dev/sdb1 vg1 lvm2 a- 372.61G 0 # pvcreate /dev/drbd0 Physical volume "/dev/drbd0" successfully created # vgcreate replicated /dev/drbd0 Volume group "replicated" successfully created # pvs PV VG Fmt Attr PSize PFree /dev/drbd0 replicated lvm2 a- 96.00G 96.00G /dev/sda3 vg0 lvm2 a- 228.75G 20.75G /dev/sdb1 vg1 lvm2 a- 372.61G 0 # lvcreate --name Mail --size 32G replicated Logical volume "Mail" created Thanks a lot!
Patch to automatically prefer DRBD top device similar to MD mirror images is here https://www.redhat.com/archives/lvm-devel/2009-October/msg00229.html
The code is for some time in rawhide and workaround (filters) exists for older systems, so I am closing this as fixed in rawhide/F13.