Bug 921387 - libvirtd crashes when deleting logical storage pool
libvirtd crashes when deleting logical storage pool
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.4
x86_64 Unspecified
unspecified Severity high
: rc
: ---
Assigned To: Martin Kletzander
Virtualization Bugs
: Upstream
Depends On:
Blocks: 921480
  Show dependency treegraph
 
Reported: 2013-03-14 01:44 EDT by Brian Kruger
Modified: 2013-11-21 03:50 EST (History)
10 users (show)

See Also:
Fixed In Version: libvirt-0.10.2-19.el6
Doc Type: Bug Fix
Doc Text:
Cause: There was invalid use-after-free in logical storage backed. Consequence: Sometimes, when deleting logical storage pool, the daemon crashed. Fix: The pointer was cleared after free'd. Result: Deleting logical volumes works correctly.
Story Points: ---
Clone Of:
: 921480 (view as bug list)
Environment:
Last Closed: 2013-11-21 03:50:34 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Output from when it crashes when trying to remove storage pool. (64.06 KB, text/plain)
2013-03-14 01:44 EDT, Brian Kruger
no flags Details
Comment (129.27 KB, text/plain)
2013-03-26 14:54 EDT, Brian Kruger
no flags Details

  None (edit)
Description Brian Kruger 2013-03-14 01:44:16 EDT
Created attachment 709876 [details]
Output from when it crashes when trying to remove storage pool.

Description of problem:

libvirt lvm storage pool usage seems to be broken.  virt-install can't find the lvm path and libvirtd also seems to crash when you try to undefine it.

Version-Release number of selected component (if applicable):

0.10.2-18.el6.x86_64

How reproducible:
95%

Steps to Reproduce:

 (we have an auto-install that uses /home for a majority of space)

Setup: 
1. cd /tmp
2. yes | sudo lvreduce --size $SIZE --resizefs --force /dev/sys/home
3. sudo lvcreate sys -n vg001 -l 100%FREE 
4. sudo vgcreate vg001 /dev/sys/vg001;
5. sudo service libvirtd start
6. sudo /usr/bin/virsh pool-define-as --name vg001 --type logical --target /dev/vg001
7. sudo /usr/bin/virsh pool-start vg001
8. sudo /usr/bin/virsh pool-autostart vg001
9. sudo /usr/bin/virt-install --name testhost --ram 4096 --disk pool=vg001,size=60 --boot network,hd,menu=on --graphics none --network bridge=br0,mac=52:54:00:34:51:4A   --vcpus 4 --os-variant=rhel6

Remove pool: 
1. sudo /usr/bin/virsh pool-destroy vg001
2. sudo /usr/bin/virsh pool-delete vg001
3 <this is where libvirtd seems to crash>
  
Actual results:

Attempt to create virtual machine (virt-install):

2013-03-13 22:22:30.017+0000: 6988: error : virCommandWait:2345 : internal error Child process (/sbin/lvcreate --name testhost.img -L 0K --virtualsize 62914560K vg001) unexpected exit status 5:   Unable to create new logical volume with no extents

2013-03-13 22:22:30.039+0000: 6988: error : virCommandWait:2345 : internal error Child process (/sbin/lvremove -f vg001/testhost.img) unexpected exit status 5:   One or more specified logical volume(s) not found.

2013-03-13 22:22:30.058+0000: 6988: error : virCommandWait:2345 : internal error Child process (/sbin/lvchange -aln vg001/testhost.img) unexpected exit status 5:   One or more specified logical volume(s) not found.
 --

Deleting Volume pool:

<see attached debug log file>

Expected results:

It should just work as it has with previous versions.

Additional info:
This maybe similar to: https://bugzilla.redhat.com/show_bug.cgi?id=872162 
However, if you follow command line from:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/chap-Virtualization_Administration_Guide-Storage_Pools-Storage_Pools.html#sect-Virtualization-Storage_Pools-Creating-LVM

this is still the result.

Had to downgrade back to 0.9.10 to make this work correctly

Also dies if defining pool this way:
sudo virsh pool-define-as vg001 logical - - /dev/mapper/sys-vg001 vg001 /dev/vg001
sudo virsh pool-start vg001

(didn't figure it'd make a difference, but tried anyway)

Here's notes of what was seen.

Starting install...
ERROR    internal error Child process (/sbin/lvchange -aln vg001/testhost.img) unexpected exit status 5:   One or more specified logical volume(s) not found.

XML dump:

<pool type='logical'>
  <name>vg001</name>
  <uuid>820d51e9-42ea-cda8-ba3b-d5b08ee29082</uuid>
  <capacity unit='bytes'>512460062720</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>512460062720</available>
  <source>
    <device path='/dev/mapper/sys-vg001'/>
    <name>vg001</name>
    <format type='lvm2'/>
  </source>
  <target>
    <path>/dev/vg001</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>
Comment 1 Jiri Denemark 2013-03-14 05:52:55 EDT
There are two issues here. First one is that virt-install tries to create a sparse logical volume (which is not a good idea since it won't work out of the box) and I'll clone this bug to virt-install to solve it. The other issue is a libvirt bug causing libvirtd to crash when you try to delete the pool.
Comment 2 Martin Kletzander 2013-03-22 11:32:04 EDT
Hello,

step 4 doesn't make sense for me.  Are you trying to create a VG on an LV and missed 'pvcreate /dev/sys/vg001' command between steps 3 and 4?  Thanks for the clarification.
Comment 3 Brian Kruger 2013-03-22 21:32:55 EDT

With OS install, we have it automated to allocate X amount to each slice (/, /var, /home, etc...) on install. /home is done last and the remaineder is done then.

The script I have, resizes /home down to $SIZE (50 gigs) and leaves the rest for allocation to a LVM slice for VM Guest usage.   No pvcreate is done since /home was already part of a LVM and space was just given back and can be used for whatever from what I understand. (another slice, snapshots, etc).

This procedure does work with 0.9.* and below.  If you have an alternate suggestion I would be happy to try it.
Comment 4 Martin Kletzander 2013-03-26 11:25:35 EDT
Could you try this with libvirt-0.10.2-18.el6_4.2 ?  Libvirt shouldn't fail, however, the volume will be only one extent in size.  If this is true, the fix is on its way.  But if it still fails, there is another problem.  Thanks for checking that.
Comment 5 Brian Kruger 2013-03-26 14:54:00 EDT
Created attachment 915683 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Comment 6 Martin Kletzander 2013-03-29 06:50:34 EDT
Moving to POST:

commit 2d73f2120f2073bff9e311626f6b6febcfc1e290
Author: Martin Kletzander <mkletzan@redhat.com>
Date:   Thu Mar 28 16:36:50 2013 +0100

    storage: Avoid double virCommandFree in virStorageBackendLogicalDeletePool
Comment 15 chhu 2013-07-11 04:21:19 EDT
According to comment1, try to verify the issue: 
"libvirt bug causing libvirtd to crash when you try to delete the pool."

For the issue: "virt-install tries to create a sparse logical volume (which is not a good idea since it won't work out of the box)", will be verified in the cloned bug ?

Reproduce with libvirt-0.10.2-18.el6.x86_64.

Verified with the packages:
libvirt-0.10.2-19.el6.x86_64

Test steps:
1. cd /tmp
2. #yes| sudo lvreduce --size 30G --resizefs --force /dev/mapper/vg_intelq8200novt86-lv_home

3. #lvcreate vg_intelq8200novt86 -n vg001 -l 100%FREE
  Logical volume "vg001" created

4. #vgcreate vg001 /dev/vg_intelq8200novt86/vg001 
  Volume group "vg001" successfully created

5. #service libvirtd restart

6. #virsh pool-define-as --name vg001 --type logical --target /dev/vg001
   Pool vg001 defined

7. #virsh pool-start vg001
   Pool vg001 started

8. #virsh pool-autostart vg001
   Pool vg001 marked as autostarted

   # virsh pool-list --all 
   Name                 State      Autostart 
   -----------------------------------------
   default              active     yes       
   vg001                active     yes

9. #virt-install --name testhost --ram 1024 --disk pool=vg001,size=10 --boot network,hd,menu=on --graphics none --network network=default,mac=52:54:00:15:15:4A --vcpus 2 --os-variant=rhel6 --location http://rhel6-url
----------------------------------------------------------------------
Starting install...
Retrieving file .treeinfo...     | 3.5 kB     00:00 ... 
Retrieving file file vmlinuz...  | 7.7 MB     00:00 ... 
Retrieving file initrd.img...    |  60 MB     00:00 ... 
Allocating 'testhost1.img'       |  10 GB     00:00     
Creating domain...               |    0 B     00:00     
Connected to domain testhost1
Escape character is ^]

Didn't met error below:
---------------------------------------------------------------------
internal error Child process (/sbin/lvchange -aln vg001/testhost.img) unexpected exit status 5:   One or more specified logical volume(s) not found.
--------------------------------------------------------------------
Then stop the installation programe.

Remove pool: 
1. # virsh pool-destroy vg001
   Pool vg001 destroyed

2. # virsh pool-delete vg001 
   Pool vg001 deleted

3. # service libvirtd status
libvirtd (pid  9430) is running...

4. check the log:/var/log/libvirt/libvirtd.log, no libvirtd segfaults.

5. # virsh pool-undefine vg001
Pool vg001 has been undefined

Test results:
After running pool-delete, libvirtd is still running, without crash. Command work well.
Comment 16 Martin Kletzander 2013-07-11 05:47:55 EDT
Yes, the virt-install part will be verified in the virt-install bug.
Comment 18 errata-xmlrpc 2013-11-21 03:50:34 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html

Note You need to log in before you can comment on or make changes to this bug.