Bug 702260

Summary: Libvirt can't remove logical volume because it doesn't deactivate it first
Product: Red Hat Enterprise Linux 6 Reporter: David Jaša <djasa>
Component: libvirtAssignee: Osier Yang <jyang>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2CC: berrange, dallan, dyuan, mzhan, prajnoha, rwu, scenek, whuang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.9.9-1.el6 Doc Type: Bug Fix
Doc Text:
No documentaion needed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 06:27:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 570359    
Bug Blocks:    

Description David Jaša 2011-05-05 08:11:03 UTC
Description of problem:
SSIA
related to bug #591390

Version-Release number of selected component (if applicable):
libvirt-0.8.7-18.el6.x86_64
libvirt-python-0.8.7-18.el6.x86_64
libvirt-client-0.8.7-18.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. create a new logical volume using virsh
2. attempt to delete logical volume created in previous step
3.
  
Actual results:
Deletion fails with this error:
error:Failed to delete vol lv-win2k8
error:internal error '/sbin/lvremove -f /path/to/lv' exited with non-zero status 5 and signal 0:   Can't remove open logical volume "lv"

Expected results:
LV gets successfully deleted

Additional info: 
'lvremove -f /path/to/lv' also fails when invoked from root shell, but when one runs 'lvchange -an /path/to/lv' and then 'lvremove /path/to/lv', the LV gets successfully removed. Full CLI output:

---------- 8< ------------------------

[root@dhcp-29-7 ~]# virsh
Vítejte na virsh, interaktivním virtualizačním terminálu.

Napište: 'help' pro nápovědu k příkazům
       'quit' pro skončení

virsh # vol-list root_vg
Jméno               Path                                    
-----------------------------------------
home                 /dev/root_vg/home                       
q-winxp              /dev/root_vg/q-winxp                    
rawhide              /dev/root_vg/rawhide                    
rhel6                /dev/root_vg/rhel6                      
swap                 /dev/root_vg/swap                       

virsh # vol-create-as root_vg lv-win2k8 25G --allocation 0 --format raw
Vol lv-win2k8 created

virsh # vol-list root_vg
Jméno               Path                                    
-----------------------------------------
home                 /dev/root_vg/home                       
lv-win2k8            /dev/root_vg/lv-win2k8                  
q-winxp              /dev/root_vg/q-winxp                    
rawhide              /dev/root_vg/rawhide                    
rhel6                /dev/root_vg/rhel6                      
swap                 /dev/root_vg/swap                       

virsh # vol-delete --pool root_vg lv-win2k8
chyba:Failed to delete vol lv-win2k8
chyba:vnitřní chyba '/sbin/lvremove -f /dev/root_vg/lv-win2k8' exited with non-zero status 5 and signal 0:   Can't remove open logical volume "lv-win2k8"


virsh # 
[root@dhcp-29-7 ~]# lvremove -f /dev/root_vg/lv-win2k8 
  Can't remove open logical volume "lv-win2k8"
[root@dhcp-29-7 ~]# lvchange -an /dev/root_vg/lv-win2k8 
[root@dhcp-29-7 ~]# lvremove /dev/root_vg/lv-win2k8 
  Logical volume "lv-win2k8" successfully removed

Comment 1 David Jaša 2011-05-05 08:35:52 UTC
One more CLI example, this time LV name is q-rhel in the same 



[root@dhcp-29-7 ~]# virsh vol-delete --pool root_vg q-rhel
chyba:Failed to delete vol q-rhel
chyba:vnitřní chyba '/sbin/lvremove -f /dev/root_vg/q-rhel' exited with non-zero status 5 and signal 0:   Can't remove open logical volume "q-rhel"


[root@dhcp-29-7 ~]# lvchange -an /dev/root_vg/q-rhel 
[root@dhcp-29-7 ~]# virsh vol-delete --pool root_vg q-rhel
Vol q-rhel deleted

Comment 2 Daniel Berrangé 2011-05-05 09:06:02 UTC
This looks like a  bug / race condition in the lvremove command to me. It is non-deterministic when I run it 

[root@lettuce ~]# virsh vol-delete --pool HostVG foo
error: Failed to delete vol foo
error: internal error '/sbin/lvremove -f /dev/HostVG/foo' exited with non-zero status 5 and signal 0:   LV HostVG/foo in use: not deactivating
  Unable to deactivate logical volume "foo"


[root@lettuce ~]# virsh vol-delete --pool HostVG foo
error: Failed to delete vol foo
error: internal error '/sbin/lvremove -f /dev/HostVG/foo' exited with non-zero status 5 and signal 0:   LV HostVG/foo in use: not deactivating
  Unable to deactivate logical volume "foo"


[root@lettuce ~]# virsh vol-delete --pool HostVG foo
Vol foo deleted

libvirt explicitly uses the '-f' argument to ensure active volumes are removed

 "  -f, --force
      Remove active logical volumes without confirmation."

Comment 3 David Jaša 2011-05-05 10:04:44 UTC
(In reply to comment #2)
> This looks like a  bug / race condition in the lvremove command to me. It is
> non-deterministic when I run it 
> 

It seems so, but doing a 'lvchange -an $LV' seems to be pretty viable workaround (numbers printed are numbers of retries before LV got removed), -f option isn't needed at all:


[root@dhcp-29-7 ~]# for I in {1..20};do virsh vol-create-as root_vg lv-example 25G --allocation 0 --format raw >/dev/null ; i=0; until lvremove -f /dev/root_vg/lv-example >/dev/null 2>&1; do i=$(($i+1)); done ; echo $i ; virsh pool-refresh root_vg >/dev/null;done
19
9
4
1
2
4
0
10
7
19
2
1
13
0
0
1
6
14
2
3
[root@dhcp-29-7 ~]# for I in {1..20};do virsh vol-create-as root_vg lv-example 25G --allocation 0 --format raw >/dev/null ; i=0; until { lvchange -an /dev/root_vg/lv-example; lvremove /dev/root_vg/lv-example >/dev/null 2>&1; } ; do i=$(($i+1)); done ; echo $i ; virsh pool-refresh root_vg >/dev/null;done
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Comment 4 Osier Yang 2011-06-08 06:28:25 UTC
This is race of lvremove, see BZ https://bugzilla.redhat.com/show_bug.cgi?id=570359, patch was applied in lvm upstream, https://www.redhat.com/archives/lvm-devel/2011-May/msg00025.html, but the patch can't fix the problem drastically, it just can minimize the possiability to reproduce the problem.

Comment 5 Osier Yang 2011-06-08 07:03:50 UTC
Both lvremove and "lvchange -an" invoke function "deactivate_lv(cmd, lv))" to deactive the lv, so using "lvchange -an" won't work for problem as <a href="https://bugzilla.redhat.com/show_bug.cgi?id=702260#c2">#c2</a>.

Using "lvremove" will work for David's case though, as lvremove do checking before deactive the lv to ensure the lv is not referred to by another existing LVs.

<quote>

        if (lv_info(cmd, lv, 0, &info, 1, 0)) {
                if (info.open_count) {
                        log_error("Can't remove open logical volume \"%s\"",
                                  lv->name);
                        return 0;
                }

                if (lv_is_active(lv) && (force == PROMPT) &&
                    lv_is_visible(lv) &&
                    yes_no_prompt("Do you really want to remove active "
                                  "%slogical volume %s? [y/n]: ",
                                  vg_is_clustered(vg) ? "clustered " : "",
                                  lv->name) == 'n') {
                        log_error("Logical volume %s not removed", lv->name);
                        return 0;
                }
        }
        
        /* Skipped codes here */

        if (!deactivate_lv(cmd, lv)) {
                log_error("Unable to deactivate logical volume \"%s\"",
                          lv->name);
                return 0;
        }
</quote>

I'm wondering if libvirt should use "lvchange -an" before "lvremove", as it at least could fix David's problem.

Comment 7 Osier Yang 2011-09-21 10:14:29 UTC
Patch posted to upstream

http://www.redhat.com/archives/libvir-list/2011-September/msg00802.html

Comment 8 Osier Yang 2011-09-22 00:21:37 UTC
patch committed to upstream.

Comment 12 Huang Wenlong 2012-01-10 03:03:27 UTC
Verify this bug with:
libvirt-0.9.9-1.el6.x86_64


1) add vg in the libvirt pool 

# virsh pool-dumpxml vg_inteli7124
<pool type='logical'>
  <name>vg_inteli7124</name>
  <uuid>794ac3ee-3a62-d96b-db6d-4c4857269cbb</uuid>
  <capacity>499579355136</capacity>
  <allocation>284264759296</allocation>
  <available>215314595840</available>
  <source>
    <device path='/dev/sda2'/>
    <name>vg_inteli7124</name>
    <format type='lvm2'/>
  </source>
  <target>
    <path>/dev/vg_inteli7124</path>
    <permissions>
      <mode>0700</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>

2) add volume 
# virsh vol-create-as --pool vg_inteli7124 --name lv_test --capacity 100M
Vol lv_test created

3)  check vol list 
# virsh vol-list  vg_inteli7124
Name                 Path                                    
-----------------------------------------
lv_home              /dev/vg_inteli7124/lv_home              
lv_root              /dev/vg_inteli7124/lv_root              
lv_swap              /dev/vg_inteli7124/lv_swap              
lv_test              /dev/vg_inteli7124/lv_test

4)  remove the volume 
#virsh vol-delete --pool vg_inteli7124 lv_test
Vol lv_test deleted

#virsh vol-list vg_inteli7124
Name                 Path                                    
-----------------------------------------
lv_home              /dev/vg_inteli7124/lv_home              
lv_root              /dev/vg_inteli7124/lv_root              
lv_swap              /dev/vg_inteli7124/lv_swap

Comment 13 Osier Yang 2012-05-04 10:05:10 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No documentaion needed.

Comment 15 errata-xmlrpc 2012-06-20 06:27:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html