Bug 896052 - creating libvirt storage pools fails for lvm volume groups with striped volumes
creating libvirt storage pools fails for lvm volume groups with striped volumes
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt (Show other bugs)
5.8
All Linux
urgent Severity high
: rc
: ---
Assigned To: Osier Yang
Virtualization Bugs
: Patch, ZStream
Depends On: 727474
Blocks: 836232 903600
  Show dependency treegraph
 
Reported: 2013-01-16 09:32 EST by J.H.M. Dassen (Ray)
Modified: 2013-09-23 07:01 EDT (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
If an LVM volume group contains a striped LVM volume, the output of the "device" field separates the multiple device paths using the comma separator. Previously, the libvirt library also used the lvs command with the comma separator, which caused regular expressions in the libvirt code to parse the output of lvs incorrectly when used on a striped LVM volume. Consequently, creation of a logical storage pool in libvirt failed if the used LVM volume group contained the striped LVM volume. Also, libvirt did not have the correct mechanism to generate multiple device XML elements for the multiple device paths of striped LVM volume. With this update, libvirt has been modified to use lvs with the "#" separator; also, the library can now parse the multiple device paths of striped LVM volumes and generate relevant XML code. Users can now create logical storage pools with striped LVM volumes and generate appropriate XML code as expected.
Story Points: ---
Clone Of: 727474
Environment:
Last Closed: 2013-09-23 07:01:49 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Proposed patch (10.71 KB, patch)
2013-01-16 09:36 EST, J.H.M. Dassen (Ray)
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Legacy) 65146 None None None Never

  None (edit)
Description J.H.M. Dassen (Ray) 2013-01-16 09:32:55 EST
+++ This bug was initially created as a clone of Bug #727474 +++

Description of problem:

When I try to define a lvm volume group as storage pool in libvirt this fails as soon as this volume group contains at least one volume which is using striping. Defining storage pools that do not use striping works as expected.

We want to use striped lvs to maximize the io performance of our vms.

Version-Release number of selected component (if applicable):

> cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.1 (Santiago)
> rpm -q libvirt
libvirt-0.8.7-18.el6.x86_64
> rpm -q lvm2
lvm2-2.02.83-3.el6.x86_64
> rpm -q udev
udev-147-2.35.el6.x86_64

How reproducible:
always

Steps to Reproduce:

> pvcreate /dev/sdc1
  Physical volume "/dev/sdc1" successfully created
> pvcreate /dev/sdd1
  Physical volume "/dev/sdd1" successfully created
> vgcreate vg_ssd /dev/sdc1 /dev/sdd1
  Volume group "vg_ssd" successfully created
> lvcreate --size 40GB --name test_nostripes vg_ssd
  Logical volume "vmtest" created
> vgchange -a y vg_ssd
  1 logical volume(s) in volume group "vg_ssd" now active
> virsh pool-create-as vg_ssd logical --target /dev/vg_ssd
Pool vg_ssd created
> virsh pool-info vg_ssd
Name:           vg_ssd
UUID:           ea3f0222-9b31-054e-c411-fa57f0d7e4b4
State:          running
Persistent:     no
Autostart:      no
Capacity:       558.91 GB
Allocation:     0.00 
Available:      558.91 GB

-> Now we have created a working storage pool from a lvm vg that does not use any striping.

> virsh pool-destroy vg_ssd
Pool vg_ssd destroyed
> lvcreate --size 40GB --stripes 2 --stripesize 8kb --name test_stripes vg_ssd
  Logical volume "vmtest" created
> vgchange -a y vg_ssd
  2 logical volume(s) in volume group "vg_ssd" now active
> stat /dev/vg_ssd/test_stripes 
  File: `/dev/vg_ssd/test_stripes' -> `../dm-3'
  Size: 7               Blocks: 0          IO Block: 4096   symbolic link
Device: 5h/5d   Inode: 25160       Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-08-02 10:30:58.616055653 +0200
Modify: 2011-08-02 10:30:58.609053092 +0200
Change: 2011-08-02 10:30:58.609053092 +0200


> virsh -d 5 pool-create-as vg_ssd logical --target /dev/vg_ssd
pool-create-as: name(optdata): vg_ssd
pool-create-as: type(optdata): logical
pool-create-as: target(optdata): /dev/vg_ssd
error: Failed to create pool vg_ssd
error: internal error lvs command failed

-> As soon as we have a lv with striping within the vg, defining the storage pool fails.

If I try to first define the storage pool and add the striped lv later on, the storage pool can't be activated.

Output into messages log (with full libvirt debugging enabled):

[...]
libvirtd: 10:41:20.917: 4657: debug : remoteDispatchClientRequest:373 : prog=536903814 ver=1 type=0 status=0 serial=3 proc=76
libvirtd: 10:41:20.917: 4657: debug : virStoragePoolCreateXML:7986 : conn=0x7f3778000a00, xmlDesc=<pool type='logical'>#012  <name>vg_ssd</name>#012  <target>#012    <path>/dev/vg_ssd</path>#012  </target>#012</pool>#012
libvirtd: 10:41:20.917: 4656: debug : virEventCleanupTimeouts:484 : Cleanup 3
libvirtd: 10:41:20.917: 4656: debug : virEventCleanupHandles:532 : Cleanup 7
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=0 w=1, f=5 e=1 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=1 w=2, f=7 e=1 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=2 w=3, f=13 e=1 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=3 w=4, f=16 e=1 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=4 w=5, f=12 e=25 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=5 w=6, f=11 e=25 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventMakePollFDs:362 : Prepare n=6 w=7, f=17 e=1 d=0
libvirtd: 10:41:20.917: 4656: debug : virEventCalculateTimeout:302 : Calculate expiry of 3 timers
libvirtd: 10:41:20.917: 4656: debug : virEventCalculateTimeout:332 : Timeout at 0 due in -1 ms
libvirtd: 10:41:20.917: 4656: debug : virEventRunOnce:594 : Poll on 7 handles 0x7f37880011e0 timeout -1
libvirtd: 10:41:20.918: 4657: debug : virRunWithHook:833 : /sbin/vgchange -ay vg_ssd
libvirtd: 10:41:21.244: 4657: debug : virRunWithHook:849 : Command stdout:   2 logical volume(s) in volume group "vg_ssd" now active#012
libvirtd: 10:41:21.246: 4657: debug : virRunWithHook:833 : /sbin/udevadm settle
libvirtd: 10:41:21.291: 4657: debug : virExecWithHook:725 : /sbin/lvs --separator , --noheadings --units b --unbuffered --nosuffix --options lv_name,origin,uuid,devices,seg_size,vg_extent_size vg_ssd
libvirtd: 10:41:21.421: 4657: error : virStorageBackendVolOpenCheckMode:1025 : cannot open volume '/dev/vg_ssd/test_stripes,': No such file or directory
libvirtd: 10:41:21.421: 4657: error : virStorageBackendLogicalFindLVs:219 : internal error lvs command failed
libvirtd: 10:41:21.421: 4657: debug : virRunWithHook:833 : /sbin/vgchange -an vg_ssd
libvirtd: 10:41:21.536: 4656: debug : virEventRunOnce:603 : Poll got 1 event(s)
libvirtd: 10:41:21.536: 4656: debug : virEventDispatchTimeouts:394 : Dispatch 3
libvirtd: 10:41:21.536: 4656: debug : virEventDispatchHandles:439 : Dispatch 7
libvirtd: 10:41:21.537: 4656: debug : virEventDispatchHandles:453 : i=0 w=1
libvirtd: 10:41:21.537: 4656: debug : virEventDispatchHandles:453 : i=1 w=2
libvirtd: 10:41:21.537: 4656: debug : virEventDispatchHandles:453 : i=2 w=3
libvirtd: 10:41:21.537: 4656: debug : virEventDispatchHandles:453 : i=3 w=4
libvirtd: 10:41:21.537: 4656: debug : virEventDispatchHandles:466 : Dispatch n=3 f=16 w=4 e=1 (nil)
libvirtd: 10:41:21.541: 4656: debug : udevEventHandleCallback:1458 : udev action: 'remove'
libvirtd: 10:41:21.541: 4656: debug : udevRemoveOneDevice:1213 : Failed to find device to remove that has udev name '/sys/devices/virtual/block/dm-2'
[...]

As you can see from the stat call above, the dir /dev/vg_ssd/test_stripes did exist before the call to pool-create-as. So the entry in the log seems just to be a side-effect, the /dev/vg_ssd dir is gone after the failed call because the vg has become inactive.

Running the lvs call that libvirt does yields no unexpected results:
> /sbin/lvs --separator , --noheadings --units b --unbuffered --nosuffix --options lv_name,origin,uuid,devices,seg_size,vg_extent_size vg_ssd; echo $?
  test_nostripes,,N8cT8q-H5oH-hYcw-P85y-HcwH-Ms1r-Z830xi,/dev/sdc1(0),42949672960,4194304
  test_stripes,,fSLSZH-zAS2-yAIb-n4mV-Al9u-HA3V-oo9K1B,/dev/sdc1(10240),/dev/sdd1(0),42949672960,4194304
0

So maybe the dir is removed by udev and udevadm settle does not wait for it to be recreated?

--- Additional comment from Gerd v. Egidy on 2011-08-02 05:04:11 EDT ---

I just took a deep look at the output of the lvs command. Counting commas...

When you change it into this it becomes more obvious:

> /sbin/lvs --separator "#" --noheadings --units b --unbuffered --nosuffix --options lv_name,origin,uuid,devices,seg_size,vg_extent_size vg_ssd
  test_nostripes##N8cT8q-H5oH-hYcw-P85y-HcwH-Ms1r-Z830xi#/dev/sdc1(0)#42949672960#4194304
  test_stripes##fSLSZH-zAS2-yAIb-n4mV-Al9u-HA3V-oo9K1B#/dev/sdc1(10240),/dev/sdd1(0)#42949672960#4194304

Seems like lvs always writes the devices as comma separated list. Most probably libvirt does not correctly handle the extra comma introduced by the extra device.
Comment 1 J.H.M. Dassen (Ray) 2013-01-16 09:36:18 EST
Created attachment 679629 [details]
Proposed patch

libvirt-storage-Do-not-use-comma-as-seperator-for-lvs-output.patch from RHEL6's libvirt-0_9_4-17_el6 applies cleanly (with some offsets) to RHEL5's current libvirt-0.8.2-29.el5.
Comment 5 zhe peng 2013-01-17 05:23:58 EST
can reproduce on libvirt-0.8.2-29.el5

Note You need to log in before you can comment on or make changes to this bug.