Bug 822015

Summary: libvirt can not create live snapshot( disk-only) if guest's image has backing file in the NFS server
Product: Red Hat Enterprise Linux 6 Reporter: Huang Wenlong <whuang>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.3CC: acathrow, cwei, dallan, dyasny, dyuan, eblake, gsun, mzhan, rwu, weizhan
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-31 02:55:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
audit log when create snapshot failed
none
libvirtd debug log snapshot create failed when backing file is in the NFS
none
create snapshot successfully when backing file is in local disk none

Description Huang Wenlong 2012-05-16 05:18:59 UTC
Description of problem:
libvirt can not create live snapshot( disk-only)  if guest's image has backing file in the NFS server , selinux will block it 

Version-Release number of selected component (if applicable):
libvirt-0.9.10-20.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.292.el6.x86_64
selinux-policy-3.7.19-150.el6.noarch


How reproducible:
100%

Steps to Reproduce:

1.NFS server   *(rw,sync,no_root_squash)

mount $NFS   /mnt/mig -vers=3

#mount
...
10.66.5.12:/mnt/data/images/ on /mnt/mig type nfs (rw,vers=3,addr=10.66.5.12)
...

2.install a guest in the NFS server named mig-qcow2.img

#qemu-img info /mnt/mig/mig-qcow2.img
image: /mnt/mig/mig-qcow2.img
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 3.4G
cluster_size: 65536

3. create a image which backing file is /mnt/mig/mig-qcow2.img

#qemu-img create -f qcow2 -b /mnt/mig/mig-qcow2.img  /var/lib/libvirt/images/q.img  8G

# qemu-img info /var/lib/libvirt/images/q.img
image: /var/lib/libvirt/images/q.img
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 31M
cluster_size: 65536
backing file: /mnt/mig/mig-qcow2.img (actual path: /mnt/mig/mig-qcow2.img)

4.  start a guest  with  q.img
# virsh dumpxml q
...
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/libvirt/images/q.img'>
<seclabel relabel='no'/>
</source>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
...

# virsh domstate q
running


5. try to create live snapshot
# virsh snapshot-create q --disk-only
error: internal error unable to execute QEMU command 'transaction': Could not open '/var/lib/libvirt/images/q.1337057407

tail -f /var/log/libvirt/libvirtd.log
2012-05-15 04:50:07.331+0000: 12026: error : qemuMonitorJSONCheckError:338 : internal error unable to execute QEMU command 'transaction': Could not open '/var/lib/libvirt/images/q.1337057407'


I will attach audit.log and debug libvirtd.log

  
Actual results:
can not create live disk snapshot 

Expected results:
create live disk snapshot successfully 

Additional info:
if the image's backing file is in local rather than NFS .live disk snapshot create successfully

Comment 1 Huang Wenlong 2012-05-16 05:21:51 UTC
Created attachment 584846 [details]
audit log when create snapshot failed

Comment 3 Huang Wenlong 2012-05-16 05:23:20 UTC
Created attachment 584847 [details]
libvirtd debug log snapshot create failed when backing file is in the NFS

Comment 4 Huang Wenlong 2012-05-16 05:26:27 UTC
Created attachment 584848 [details]
create snapshot successfully when backing file is in local disk

Comment 7 weizhang 2012-07-26 09:16:41 UTC
When I do save/restore with an image with backing file on nfs, it also have problem
error: Failed to restore domain from /tmp/demo.save1
error: internal error Process exited while reading console log output: char device redirected to /dev/pts/18
qemu-kvm: -drive file=/var/lib/libvirt/images/rhel6u3-mig.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /var/lib/libvirt/images/rhel6u3-mig.qcow2: Permission denied

are they the same cause?

Comment 8 Peter Krempa 2013-01-30 12:47:28 UTC
Hi Wenlong,
it seems that the snapshot creation was denied by selinux. Could you try to reproduce this bug and check the value of:

# getsebool virt_use_nfs

If it is off while being reproduced, please try to enable that feature and try to reproduce it again.

Thanks

Comment 9 Huang Wenlong 2013-01-31 02:47:35 UTC
(In reply to comment #8)
> Hi Wenlong,
> it seems that the snapshot creation was denied by selinux. Could you try to
> reproduce this bug and check the value of:
> 
> # getsebool virt_use_nfs
> 
> If it is off while being reproduced, please try to enable that feature and
> try to reproduce it again.
> 
> Thanks

Hi. peter 

I can not reproduce this bug with 
libvirt-0.10.2-18.el6.x86_64
selinux-policy-3.7.19-195.el6.noarch

it can create disk-only snapshot now without error  , when I enabled virt_use_nfs ,
but the behavior is not the same when I report this bug ,now when I just disabled virsh_use_nfs , the guest even can not be started ,it will pop up selinux error 
qemu-kvm: 
-drive file=/var/lib/libvirt/images/q.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none: could not open disk image /var/lib/libvirt/images/q.img: Permission denied

Comment 10 Eric Blake 2013-01-31 02:55:12 UTC
(In reply to comment #9)
> it can create disk-only snapshot now without error  , when I enabled
> virt_use_nfs ,
> but the behavior is not the same when I report this bug ,now when I just
> disabled virsh_use_nfs , the guest even can not be started ,it will pop up
> selinux error 
> qemu-kvm: 
> -drive
> file=/var/lib/libvirt/images/q.img,if=none,id=drive-ide0-0-0,format=qcow2,
> cache=none: could not open disk image /var/lib/libvirt/images/q.img:
> Permission denied

This is expected.  Disabling virt_use_nfs prevents the use of any disk images on NFS (at least, until bug 731134 is solved, which is still waiting on some upstream qemu design).  As such, I'm closing this as not a bug.

Comment 11 Huang Wenlong 2013-01-31 03:14:59 UTC
Hi,Eric 

When I downgrade libvirt to 0.9.10-20 then I still can reproduce this bug ,so I do not think it should be closed by NOTABUG , it is a real bug with libvirt-0.9.10-20.el6.x86_64  but it has been fixed by newer version 
I am sure the whole test process the virt_use_nfs is enabled

downgrade version libvirt  with new selniux-policy 
libvirt-0.9.10-20.el6.x86_64
selinux-policy-3.7.19-195.el6.noarch

Comment 12 Eric Blake 2013-01-31 03:26:46 UTC
What bug? If you are using NFS for disk images, then your testing must be done with virt_use_nfs enabled.  Any failures with virt_use_nfs disabled are expected, as a feature of sVirt.  You still haven't demonstrated that there is a bug.

Comment 13 Huang Wenlong 2013-01-31 03:31:06 UTC
(In reply to comment #12)
> What bug? If you are using NFS for disk images, then your testing must be
> done with virt_use_nfs enabled.  Any failures with virt_use_nfs disabled are
> expected, as a feature of sVirt.  You still haven't demonstrated that there
> is a bug.

when I use libvirt-0.9.10-20.el6.x86_64 and selinux-policy-3.7.19-195.el6.noarch 
set  virt_use_nfs enabled 

1) start a domain  with disk image's backing file is on the NFS 

2) try to create disk-only snapshot  it will report a AVC error in selinux  

# virsh start b15
Domain b15 started

[root@intel-q9400-4-1 ~]# virsh snapshot-create-as b15 disk-only-nfs-3 --disk-only
error: internal error unable to execute QEMU command 'transaction': Could not open '/var/lib/libvirt/images/q.disk-only-nfs-3': Permission denied

# getsebool -a |grep virt_use_nfs
virt_use_nfs --> on

Comment 14 Eric Blake 2013-01-31 03:43:17 UTC
Okay, I took time to re-read the entire bug.  I had closed things based on comment 9 stating that RHEL 6.4 libvirt works as expected according to the virt_use_nfs bool settings.  But the original report is about RHEL 6.3 libvirt not behaving nicely, even when virt_use_nfs is specified, and asking whether there is something that needs to be backported to 6.3.z to pick up the fix that already lives in the 6.4 build.  So my close action may have been premature.  On the other hand, there was a LOT of changes to the code base between 6.3 and 6.4 in regards to snapshot management, so I'm not sure it would be easy to identify what to backport.  Is this bug severe enough to need a z-stream fix and thus reopen it?

Comment 15 Huang Wenlong 2013-01-31 04:29:18 UTC
(In reply to comment #14)
> Okay, I took time to re-read the entire bug.  I had closed things based on
> comment 9 stating that RHEL 6.4 libvirt works as expected according to the
> virt_use_nfs bool settings.  But the original report is about RHEL 6.3
> libvirt not behaving nicely, even when virt_use_nfs is specified, and asking
> whether there is something that needs to be backported to 6.3.z to pick up
> the fix that already lives in the 6.4 build.  So my close action may have
> been premature.  On the other hand, there was a LOT of changes to the code
> base between 6.3 and 6.4 in regards to snapshot management, so I'm not sure
> it would be easy to identify what to backport.  Is this bug severe enough to
> need a z-stream fix and thus reopen it?

I think it is a libvirt svirt bug not snapshot, the new image context is incorrect in old libvirt 
type=AVC msg=audit(1359602249.767:80375): avc:  denied  { getattr } for  pid=6164 comm="qemu-kvm" path="/var/lib/libvirt/images/q.disk-only-nfs-3" dev=sda1 ino=4488 scontext=system_u:system_r:svirt_t:s0:c492,c770 tcontext=unconfined_u:object_r:virt_image_t:s0 tclass=file

it can reproduce this bug with libvirt-0.9.10-21.el6_3.8.x86_64 and virt_use_nfs=1 
it is not very severe for z-stream , maybe it can be closed by "wontfix" for 63z and "currentrelease" for 6.4
thanks 

Wenlong

Comment 16 Eric Blake 2013-01-31 04:37:06 UTC
Okay, I'm changing the closed reason to current release, and targeting the bug to 6.4 instead of 6.5.

Comment 17 Huang Wenlong 2013-01-31 04:39:04 UTC
(In reply to comment #16)
> Okay, I'm changing the closed reason to current release, and targeting the
> bug to 6.4 instead of 6.5.

Thank you very much :>