Red Hat Bugzilla – Bug 702260
Libvirt can't remove logical volume because it doesn't deactivate it first
Last modified: 2012-06-20 02:27:59 EDT
Description of problem: SSIA related to bug #591390 Version-Release number of selected component (if applicable): libvirt-0.8.7-18.el6.x86_64 libvirt-python-0.8.7-18.el6.x86_64 libvirt-client-0.8.7-18.el6.x86_64 How reproducible: always Steps to Reproduce: 1. create a new logical volume using virsh 2. attempt to delete logical volume created in previous step 3. Actual results: Deletion fails with this error: error:Failed to delete vol lv-win2k8 error:internal error '/sbin/lvremove -f /path/to/lv' exited with non-zero status 5 and signal 0: Can't remove open logical volume "lv" Expected results: LV gets successfully deleted Additional info: 'lvremove -f /path/to/lv' also fails when invoked from root shell, but when one runs 'lvchange -an /path/to/lv' and then 'lvremove /path/to/lv', the LV gets successfully removed. Full CLI output: ---------- 8< ------------------------ [root@dhcp-29-7 ~]# virsh Vítejte na virsh, interaktivním virtualizačním terminálu. Napište: 'help' pro nápovědu k příkazům 'quit' pro skončení virsh # vol-list root_vg Jméno Path ----------------------------------------- home /dev/root_vg/home q-winxp /dev/root_vg/q-winxp rawhide /dev/root_vg/rawhide rhel6 /dev/root_vg/rhel6 swap /dev/root_vg/swap virsh # vol-create-as root_vg lv-win2k8 25G --allocation 0 --format raw Vol lv-win2k8 created virsh # vol-list root_vg Jméno Path ----------------------------------------- home /dev/root_vg/home lv-win2k8 /dev/root_vg/lv-win2k8 q-winxp /dev/root_vg/q-winxp rawhide /dev/root_vg/rawhide rhel6 /dev/root_vg/rhel6 swap /dev/root_vg/swap virsh # vol-delete --pool root_vg lv-win2k8 chyba:Failed to delete vol lv-win2k8 chyba:vnitřní chyba '/sbin/lvremove -f /dev/root_vg/lv-win2k8' exited with non-zero status 5 and signal 0: Can't remove open logical volume "lv-win2k8" virsh # [root@dhcp-29-7 ~]# lvremove -f /dev/root_vg/lv-win2k8 Can't remove open logical volume "lv-win2k8" [root@dhcp-29-7 ~]# lvchange -an /dev/root_vg/lv-win2k8 [root@dhcp-29-7 ~]# lvremove /dev/root_vg/lv-win2k8 Logical volume "lv-win2k8" successfully removed
One more CLI example, this time LV name is q-rhel in the same [root@dhcp-29-7 ~]# virsh vol-delete --pool root_vg q-rhel chyba:Failed to delete vol q-rhel chyba:vnitřní chyba '/sbin/lvremove -f /dev/root_vg/q-rhel' exited with non-zero status 5 and signal 0: Can't remove open logical volume "q-rhel" [root@dhcp-29-7 ~]# lvchange -an /dev/root_vg/q-rhel [root@dhcp-29-7 ~]# virsh vol-delete --pool root_vg q-rhel Vol q-rhel deleted
This looks like a bug / race condition in the lvremove command to me. It is non-deterministic when I run it [root@lettuce ~]# virsh vol-delete --pool HostVG foo error: Failed to delete vol foo error: internal error '/sbin/lvremove -f /dev/HostVG/foo' exited with non-zero status 5 and signal 0: LV HostVG/foo in use: not deactivating Unable to deactivate logical volume "foo" [root@lettuce ~]# virsh vol-delete --pool HostVG foo error: Failed to delete vol foo error: internal error '/sbin/lvremove -f /dev/HostVG/foo' exited with non-zero status 5 and signal 0: LV HostVG/foo in use: not deactivating Unable to deactivate logical volume "foo" [root@lettuce ~]# virsh vol-delete --pool HostVG foo Vol foo deleted libvirt explicitly uses the '-f' argument to ensure active volumes are removed " -f, --force Remove active logical volumes without confirmation."
(In reply to comment #2) > This looks like a bug / race condition in the lvremove command to me. It is > non-deterministic when I run it > It seems so, but doing a 'lvchange -an $LV' seems to be pretty viable workaround (numbers printed are numbers of retries before LV got removed), -f option isn't needed at all: [root@dhcp-29-7 ~]# for I in {1..20};do virsh vol-create-as root_vg lv-example 25G --allocation 0 --format raw >/dev/null ; i=0; until lvremove -f /dev/root_vg/lv-example >/dev/null 2>&1; do i=$(($i+1)); done ; echo $i ; virsh pool-refresh root_vg >/dev/null;done 19 9 4 1 2 4 0 10 7 19 2 1 13 0 0 1 6 14 2 3 [root@dhcp-29-7 ~]# for I in {1..20};do virsh vol-create-as root_vg lv-example 25G --allocation 0 --format raw >/dev/null ; i=0; until { lvchange -an /dev/root_vg/lv-example; lvremove /dev/root_vg/lv-example >/dev/null 2>&1; } ; do i=$(($i+1)); done ; echo $i ; virsh pool-refresh root_vg >/dev/null;done 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
This is race of lvremove, see BZ https://bugzilla.redhat.com/show_bug.cgi?id=570359, patch was applied in lvm upstream, https://www.redhat.com/archives/lvm-devel/2011-May/msg00025.html, but the patch can't fix the problem drastically, it just can minimize the possiability to reproduce the problem.
Both lvremove and "lvchange -an" invoke function "deactivate_lv(cmd, lv))" to deactive the lv, so using "lvchange -an" won't work for problem as <a href="https://bugzilla.redhat.com/show_bug.cgi?id=702260#c2">#c2</a>. Using "lvremove" will work for David's case though, as lvremove do checking before deactive the lv to ensure the lv is not referred to by another existing LVs. <quote> if (lv_info(cmd, lv, 0, &info, 1, 0)) { if (info.open_count) { log_error("Can't remove open logical volume \"%s\"", lv->name); return 0; } if (lv_is_active(lv) && (force == PROMPT) && lv_is_visible(lv) && yes_no_prompt("Do you really want to remove active " "%slogical volume %s? [y/n]: ", vg_is_clustered(vg) ? "clustered " : "", lv->name) == 'n') { log_error("Logical volume %s not removed", lv->name); return 0; } } /* Skipped codes here */ if (!deactivate_lv(cmd, lv)) { log_error("Unable to deactivate logical volume \"%s\"", lv->name); return 0; } </quote> I'm wondering if libvirt should use "lvchange -an" before "lvremove", as it at least could fix David's problem.
Patch posted to upstream http://www.redhat.com/archives/libvir-list/2011-September/msg00802.html
patch committed to upstream.
Verify this bug with: libvirt-0.9.9-1.el6.x86_64 1) add vg in the libvirt pool # virsh pool-dumpxml vg_inteli7124 <pool type='logical'> <name>vg_inteli7124</name> <uuid>794ac3ee-3a62-d96b-db6d-4c4857269cbb</uuid> <capacity>499579355136</capacity> <allocation>284264759296</allocation> <available>215314595840</available> <source> <device path='/dev/sda2'/> <name>vg_inteli7124</name> <format type='lvm2'/> </source> <target> <path>/dev/vg_inteli7124</path> <permissions> <mode>0700</mode> <owner>-1</owner> <group>-1</group> </permissions> </target> </pool> 2) add volume # virsh vol-create-as --pool vg_inteli7124 --name lv_test --capacity 100M Vol lv_test created 3) check vol list # virsh vol-list vg_inteli7124 Name Path ----------------------------------------- lv_home /dev/vg_inteli7124/lv_home lv_root /dev/vg_inteli7124/lv_root lv_swap /dev/vg_inteli7124/lv_swap lv_test /dev/vg_inteli7124/lv_test 4) remove the volume #virsh vol-delete --pool vg_inteli7124 lv_test Vol lv_test deleted #virsh vol-list vg_inteli7124 Name Path ----------------------------------------- lv_home /dev/vg_inteli7124/lv_home lv_root /dev/vg_inteli7124/lv_root lv_swap /dev/vg_inteli7124/lv_swap
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: No documentaion needed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0748.html