Description of problem: When running Fedora in SELinux "Enforcing" mode libvirt can't start VMs backed by chain of images. Images created via: $ qemu-img create -f qcow2 vm1-base.qcow2 10 $ qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-stage1.qcow2 $ qemu-img create -f qcow2 -b vm1-stage1.qcow2 vm1-active.qcow2 using vm1-active.qcow2 yeilds failure to start up: # virsh start vm1 error: Failed to start domain vm1 error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 (label charserial0) qemu-system-x86_64: -drive file=/var/lib/libvirt/images/vm1-active.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /var/lib/libvirt/images/vm1-active.qcow2: Permission denied troubleshooting reveals: # grep qemu-system-x86 /var/log/audit/audit.log | audit2allow #============= svirt_t ============== allow svirt_t virt_image_t:file read; Raw Audit Messages type=AVC msg=audit(1387241248.723:73): avc: denied { read } for pid=3332 comm="qemu-system-x86" name="vm1-base.qcow2" dev="dm-7" ino=524302 scontext=system_u:system_r:svirt_t:s0:c782,c976 tcontext=system_u:object_r:virt_image_t:s0 tclass=file type=SYSCALL msg=audit(1387241248.723:73): arch=x86_64 syscall=open success=no exit=EACCES a0=7fffd69ce220 a1=80800 a2=0 a3=62696c2f7261762f items=0 ppid=1 pid=3332 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 ses=4294967295 tty=(none) comm=qemu-system-x86 exe=/usr/bin/qemu-system-x86_64 subj=system_u:system_r:svirt_t:s0:c782,c976 key=(null) From talking to Daniel Walsh it sounds like libvirt is supposed to relabel images it'll be using for the runtime which it doesn't do with vm1-base.qcow2 SELinux labels on files are all virt_image_t. I have tried virt_content_t with no success. Version-Release number of selected component (if applicable): libvirt-1.0.5.7-2.fc19.x86_64 selinux-policy-targeted-3.12.1-74.15.fc19.noarch How reproducible: Always Steps to Reproduce: 1. create original image: $ qemu-img create -f qcow2 vm1-base.qcow2 10 2. install OS onto the image.... 3. shutdown VM 4. create 2 snapshot layers: $ qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-stage1.qcow2 $ qemu-img create -f qcow2 -b vm1-stage1.qcow2 vm1-active.qcow2 5. modify VM to use vm1-active.qcow2 instead of vm1-base.qcow2 6. attempt to start VM Actual results: # virsh start vm1 error: Failed to start domain vm1 error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 (label charserial0) qemu-system-x86_64: -drive file=/var/lib/libvirt/images/vm1-active.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /var/lib/libvirt/images/vm1-active.qcow2: Permission denied Expected results: # virsh start vm1 Domain vm1 started Additional info: I have been running single snapshot scenario for a while without much trouble, however yesterday I had to switch to dual-snapshot which coincided with system updates (and bump of selinux-policy-targeted). However I have attempted to launch single snapshot vm with the same error. (in other words - I have invoked "qemu-img create -b ..." only once and then used resulting snapshot for backing storage for an image. I have also force-relabeled *everything* in /var/lib/libvirt/images with svirt_image_t and got functioning setup again.
(In reply to Dmitry S. Makovey from comment #0) > Description of problem: > > When running Fedora in SELinux "Enforcing" mode libvirt can't start VMs > backed by chain of images. Images created via: > > $ qemu-img create -f qcow2 vm1-base.qcow2 10 > $ qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-stage1.qcow2 > $ qemu-img create -f qcow2 -b vm1-stage1.qcow2 vm1-active.qcow2 There's your bug. You forgot to pass -o backing_fmt=qcow2, and your /etc/libvirt/qemu.conf does not override allow_disk_format_probing to turn on the (insecure) probing. Fix your files to properly specify the backing format, and sVirt will no longer get in your way. (Meanwhile, libvirt REALLY ought to provide a way to dump its notion of the backing chain, including highlighting user errors like this)
I'm not sure I understand implications here. 1. It was working up until yesterday with no problem. 2. I don't see how -o backing format would help since qemu-img already properly recognises all items in chain: # qemu-img info --backing-chain vm1-active.qcow2 image: vm1-active.qcow2 file format: qcow2 virtual size: 7.0G (7516192768 bytes) disk size: 8.2M cluster_size: 65536 backing file: /var/lib/libvirt/images/vm1-stage.qcow2 image: /var/lib/libvirt/images/rns-stage.qcow2 file format: qcow2 virtual size: 7.0G (7516192768 bytes) disk size: 67M cluster_size: 65536 backing file: vm1-base.qcow2 (actual path: /var/lib/libvirt/images/vm1-base.qcow2) image: /var/lib/libvirt/images/vm1-base.qcow2 file format: qcow2 virtual size: 7.0G (7516192768 bytes) disk size: 1.6G cluster_size: 65536 (note - edited filenames to be consistent with above post, however everything else is as reported) 3. why should allow_disk_format_probing be involved if VM definition clearly states format is qcow2 : # grep -C 3 rns /etc/libvirt/qemu/vm1.xml <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1-active.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> each of the above points suggests it's a bug someplace, but I fail to see how it is not a bug.
err. replace "rns" in point 3 with "vm1" sorry for the typo
(In reply to Dmitry S. Makovey from comment #2) > I'm not sure I understand implications here. > > 1. It was working up until yesterday with no problem. You must have done something differently up until yesterday. If libvirt creates the files, then it uses -o backing_fmt. If you create the files by hand, then you must do the same, or run with SELinux permissive instead of enforcing, or tweak your /etc/libvirt/qemu.conf to allow libvirt probing (and acknowledge the CVE risk if probing guesses wrong). > 2. I don't see how -o backing format would help since qemu-img already > properly recognises all items in chain: qemu-img probes automatically. As long as SELinux is not involved, this is okay; but it is risky. If you pass a raw image to a malicious guest, the guest can write qcow2 metadata into that image, such that a future probe will think the image is qcow2 instead of raw. The only way to prevent this is to avoid automatic probing. Therefore, libvirt's use of SELinux enforces that if probing would be required to open a backing file correctly, then libvirt intentionally fails to label the file, thus preventing qemu from being able to probe (and possibly probe incorrectly), unless you tell libvirt that probing is okay. But if you use -o backing_fmt consistently (the way libvirt does for all qcow2 files it creates), then there is no need to probe in the first place. > # qemu-img info --backing-chain vm1-active.qcow2 > image: vm1-active.qcow2 > file format: qcow2 > virtual size: 7.0G (7516192768 bytes) > disk size: 8.2M > cluster_size: 65536 > backing file: /var/lib/libvirt/images/vm1-stage.qcow2 You are missing the 'backing format:' line, which meant that qemu-img probed. And as I explained, probing under sVirt is a security risk, hence it is denied by default. This is intentional on libvirt's part. > 3. why should allow_disk_format_probing be involved if VM definition clearly > states format is qcow2 : > > # grep -C 3 rns /etc/libvirt/qemu/vm1.xml > <emulator>/usr/bin/qemu-kvm</emulator> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> That's the format of the top file in the chain. But the VM definition doesn't state ANYTHING about the format of the second file in the chain (yes, we SHOULD enhance libvirt to allow someone to specify their entire chain in the VM XML, instead of relying on metadata scraping from the files, but that's a much bigger project). So the only way libvirt knows whether the second file is raw or qcow2 is by reading the metadata from the top file in the chain; and if you tell libvirt the probing is disabled but the metadata lacks the backing_fmt information, libvirt assumes that the backing file is raw, as the only safe default course of action. > each of the above points suggests it's a bug someplace, but I fail to see > how it is not a bug. The bug is in your failure to set qcow2 metadata correctly, and not in libvirt.
ok. I got your points. I will try to use backing_fmt in the future. However I have images that already are in this state so I'll be resorting to fake-labeling them with svirt_image_t. In the meantime I've discovered that my single-snapshot scenario is indeed 2-snapshot AND I have discovered that rolling it back to single-snapshot state works whereas 2-snapshot fails: # virsh start devstack-f Domain devstack-f started /var/lib/libvirt/images # qemu-img info --backing-chain devstack-f.qcow2 image: devstack-f.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 3.2G cluster_size: 65536 backing file: f19-minimal.qcow2 image: f19-minimal.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 1.6G cluster_size: 65536 note the absence of "backing format:" in above scenario. Also note that if I try to re-create it from scratch "backing_fmt" becomes an issue: # mv devstack-f.qcow2 devstack-f-foo.qcow2 # qemu-img create -f qcow2 -b f19-minimal.qcow2 devstack-f.qcow2 Formatting 'devstack-f.qcow2', fmt=qcow2 size=10737418240 backing_file='f19-minimal.qcow2' encryption=off cluster_size=65536 lazy_refcounts=off # virsh start devstack-f error: Failed to start domain devstack-f error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/7 (label charserial0) qemu-system-x86_64: -drive file=/var/lib/libvirt/images/devstack-f.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /var/lib/libvirt/images/devstack-f.qcow2: Permission denied To sum up: * I've confirmed that adding -o backing_fmt=qcow2 does help * I also apologize for inaccurate original info. * I went through my history file and not a single time did I use backing_fmt in the past while everything was working, always was using (successfully): # qemu-img create -b f19-minimal.qcow2 -f qcow2 devstack-f.qcow2 So there must've been a recent change (my last history entry of this kind where everything worked is from around Nov 19, 2013 give-or-take a day or two, and I just have noticed that there was an update on Nov 19 with lots of qemu-1.4.2-13 and libvirt-1.0.5.7-1 updates if that helps)
(In reply to Dmitry S. Makovey from comment #5) > ok. I got your points. I will try to use backing_fmt in the future. However > I have images that already are in this state so I'll be resorting to > fake-labeling them with svirt_image_t. No need to do that; qemu-img is your friend. Given your file: # qemu-img info --backing-chain vm1-active.qcow2 image: vm1-active.qcow2 file format: qcow2 virtual size: 7.0G (7516192768 bytes) disk size: 8.2M cluster_size: 65536 backing file: /var/lib/libvirt/images/vm1-stage.qcow2 Use: qemu-img rebase -f qcow2 -u -b /var/lib/libvirt/images/vm1-stage.qcow2 -F qcow2 vm1-active.qcow2 and you'll have effectively added in the missing backing_fmt option. > To sum up: > * I've confirmed that adding -o backing_fmt=qcow2 does help > * I also apologize for inaccurate original info. > * I went through my history file and not a single time did I use backing_fmt > in the past while everything was working, always was using (successfully): For only a single level deep, the fact that libvirt treats the backing file as raw but qemu treats it as qcow2 via probing is not a violation of sVirt. The problem happens when the backing chain is 3 deep, but libvirt stopped labelling at 2 deep because libvirt treated a file as raw even though qemu would treat it as qcow2. > > # qemu-img create -b f19-minimal.qcow2 -f qcow2 devstack-f.qcow2 > > So there must've been a recent change (my last history entry of this kind > where everything worked is from around Nov 19, 2013 give-or-take a day or > two, and I just have noticed that there was an update on Nov 19 with lots of > qemu-1.4.2-13 and libvirt-1.0.5.7-1 updates if that helps) That doesn't tell me what you upgraded from; and while we have patched bugs related to libvirt incorrectly parsing backing chains, I don't see any particular patches related to backing chains directly in 1.0.5.7 in relation to 1.0.5.6.