Bug 1044217 - SELinux prevents VM startup with QCOW2 snapshot images
Summary: SELinux prevents VM startup with QCOW2 snapshot images
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 19
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-17 22:34 UTC by Dmitry S. Makovey
Modified: 2013-12-18 03:16 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-17 22:41:57 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Dmitry S. Makovey 2013-12-17 22:34:06 UTC
Description of problem:

When running Fedora in SELinux "Enforcing" mode libvirt can't start VMs backed by chain of images. Images created via:

$ qemu-img create -f qcow2 vm1-base.qcow2 10
$ qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-stage1.qcow2
$ qemu-img create -f qcow2 -b vm1-stage1.qcow2 vm1-active.qcow2

using vm1-active.qcow2 yeilds failure to start up:

# virsh start vm1
error: Failed to start domain vm1
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 (label charserial0)
qemu-system-x86_64: -drive file=/var/lib/libvirt/images/vm1-active.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /var/lib/libvirt/images/vm1-active.qcow2: Permission denied

troubleshooting reveals:

# grep qemu-system-x86 /var/log/audit/audit.log | audit2allow

#============= svirt_t ==============
allow svirt_t virt_image_t:file read;

Raw Audit Messages
type=AVC msg=audit(1387241248.723:73): avc:  denied  { read } for  pid=3332 comm="qemu-system-x86" name="vm1-base.qcow2" dev="dm-7" ino=524302 scontext=system_u:system_r:svirt_t:s0:c782,c976 tcontext=system_u:object_r:virt_image_t:s0 tclass=file

type=SYSCALL msg=audit(1387241248.723:73): arch=x86_64 syscall=open success=no exit=EACCES a0=7fffd69ce220 a1=80800 a2=0 a3=62696c2f7261762f items=0 ppid=1 pid=3332 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 ses=4294967295 tty=(none) comm=qemu-system-x86 exe=/usr/bin/qemu-system-x86_64 subj=system_u:system_r:svirt_t:s0:c782,c976 key=(null)

From talking to Daniel Walsh it sounds like libvirt is supposed to relabel images it'll be using for the runtime which it doesn't do with vm1-base.qcow2 

SELinux labels on files are all virt_image_t. I have tried virt_content_t with no success.

Version-Release number of selected component (if applicable):

libvirt-1.0.5.7-2.fc19.x86_64
selinux-policy-targeted-3.12.1-74.15.fc19.noarch

How reproducible:

Always

Steps to Reproduce:
1. create original image:

$ qemu-img create -f qcow2 vm1-base.qcow2 10

2. install OS onto the image....
3. shutdown VM
4. create 2 snapshot layers:

$ qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-stage1.qcow2
$ qemu-img create -f qcow2 -b vm1-stage1.qcow2 vm1-active.qcow2

5. modify VM to use vm1-active.qcow2 instead of vm1-base.qcow2
6. attempt to start VM

Actual results:

# virsh start vm1
error: Failed to start domain vm1
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 (label charserial0)
qemu-system-x86_64: -drive file=/var/lib/libvirt/images/vm1-active.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /var/lib/libvirt/images/vm1-active.qcow2: Permission denied


Expected results:

# virsh start vm1
Domain vm1 started

Additional info:

I have been running single snapshot scenario for a while without much trouble, however yesterday I had to switch to dual-snapshot which coincided with system updates (and bump of selinux-policy-targeted). 

However I have attempted to launch single snapshot vm with the same error. (in other words - I have invoked "qemu-img create -b ..." only once and then used resulting snapshot for backing storage for an image.

I have also force-relabeled *everything* in /var/lib/libvirt/images with svirt_image_t and got functioning setup again.

Comment 1 Eric Blake 2013-12-17 22:41:45 UTC
(In reply to Dmitry S. Makovey from comment #0)
> Description of problem:
> 
> When running Fedora in SELinux "Enforcing" mode libvirt can't start VMs
> backed by chain of images. Images created via:
> 
> $ qemu-img create -f qcow2 vm1-base.qcow2 10
> $ qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-stage1.qcow2
> $ qemu-img create -f qcow2 -b vm1-stage1.qcow2 vm1-active.qcow2

There's your bug.  You forgot to pass -o backing_fmt=qcow2, and your /etc/libvirt/qemu.conf does not override allow_disk_format_probing to turn on the (insecure) probing.

Fix your files to properly specify the backing format, and sVirt will no longer get in your way.  (Meanwhile, libvirt REALLY ought to provide a way to dump its notion of the backing chain, including highlighting user errors like this)

Comment 2 Dmitry S. Makovey 2013-12-17 22:51:12 UTC
I'm not sure I understand implications here.

1. It was working up until yesterday with no problem.
2. I don't see how -o backing format would help since qemu-img already properly recognises all items in chain: 

# qemu-img info --backing-chain vm1-active.qcow2
image: vm1-active.qcow2
file format: qcow2
virtual size: 7.0G (7516192768 bytes)
disk size: 8.2M
cluster_size: 65536
backing file: /var/lib/libvirt/images/vm1-stage.qcow2

image: /var/lib/libvirt/images/rns-stage.qcow2
file format: qcow2
virtual size: 7.0G (7516192768 bytes)
disk size: 67M
cluster_size: 65536
backing file: vm1-base.qcow2 (actual path: /var/lib/libvirt/images/vm1-base.qcow2)

image: /var/lib/libvirt/images/vm1-base.qcow2
file format: qcow2
virtual size: 7.0G (7516192768 bytes)
disk size: 1.6G
cluster_size: 65536

(note - edited filenames to be consistent with above post, however everything else is as reported)

3. why should allow_disk_format_probing be involved if VM definition clearly states format is qcow2 :

# grep -C 3 rns /etc/libvirt/qemu/vm1.xml 
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1-active.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

each of the above points suggests it's a bug someplace, but I fail to see how it is not a bug.

Comment 3 Dmitry S. Makovey 2013-12-17 22:52:07 UTC
err. replace "rns" in point 3 with "vm1" sorry for the typo

Comment 4 Eric Blake 2013-12-17 23:08:33 UTC
(In reply to Dmitry S. Makovey from comment #2)
> I'm not sure I understand implications here.
> 
> 1. It was working up until yesterday with no problem.

You must have done something differently up until yesterday.  If libvirt creates the files, then it uses -o backing_fmt.  If you create the files by hand, then you must do the same, or run with SELinux permissive instead of enforcing, or tweak your /etc/libvirt/qemu.conf to allow libvirt probing (and acknowledge the CVE risk if probing guesses wrong).

> 2. I don't see how -o backing format would help since qemu-img already
> properly recognises all items in chain: 

qemu-img probes automatically.  As long as SELinux is not involved, this is okay; but it is risky.  If you pass a raw image to a malicious guest, the guest can write qcow2 metadata into that image, such that a future probe will think the image is qcow2 instead of raw.  The only way to prevent this is to avoid automatic probing.

Therefore, libvirt's use of SELinux enforces that if probing would be required to open a backing file correctly, then libvirt intentionally fails to label the file, thus preventing qemu from being able to probe (and possibly probe incorrectly), unless you tell libvirt that probing is okay.  But if you use -o backing_fmt consistently (the way libvirt does for all qcow2 files it creates), then there is no need to probe in the first place.

> # qemu-img info --backing-chain vm1-active.qcow2
> image: vm1-active.qcow2
> file format: qcow2
> virtual size: 7.0G (7516192768 bytes)
> disk size: 8.2M
> cluster_size: 65536
> backing file: /var/lib/libvirt/images/vm1-stage.qcow2

You are missing the 'backing format:' line, which meant that qemu-img probed.  And as I explained, probing under sVirt is a security risk, hence it is denied by default.  This is intentional on libvirt's part.

> 3. why should allow_disk_format_probing be involved if VM definition clearly
> states format is qcow2 :
> 
> # grep -C 3 rns /etc/libvirt/qemu/vm1.xml 
>     <emulator>/usr/bin/qemu-kvm</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2'/>

That's the format of the top file in the chain.  But the VM definition doesn't state ANYTHING about the format of the second file in the chain (yes, we SHOULD enhance libvirt to allow someone to specify their entire chain in the VM XML, instead of relying on metadata scraping from the files, but that's a much bigger project).  So the only way libvirt knows whether the second file is raw or qcow2 is by reading the metadata from the top file in the chain; and if you tell libvirt the probing is disabled but the metadata lacks the backing_fmt information, libvirt assumes that the backing file is raw, as the only safe default course of action.

> each of the above points suggests it's a bug someplace, but I fail to see
> how it is not a bug.

The bug is in your failure to set qcow2 metadata correctly, and not in libvirt.

Comment 5 Dmitry S. Makovey 2013-12-18 01:06:56 UTC
ok. I got your points. I will try to use backing_fmt in the future. However I have images that already are in this state so I'll be resorting to fake-labeling them with svirt_image_t.

In the meantime I've discovered that my single-snapshot scenario is indeed 2-snapshot AND I have discovered that rolling it back to single-snapshot state works whereas 2-snapshot fails:

# virsh start devstack-f
Domain devstack-f started

/var/lib/libvirt/images # qemu-img info --backing-chain devstack-f.qcow2 
image: devstack-f.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 3.2G
cluster_size: 65536
backing file: f19-minimal.qcow2

image: f19-minimal.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.6G
cluster_size: 65536

note the absence of "backing format:" in above scenario.

Also note that if I try to re-create it from scratch "backing_fmt" becomes an issue:

# mv devstack-f.qcow2 devstack-f-foo.qcow2
# qemu-img create -f qcow2 -b f19-minimal.qcow2 devstack-f.qcow2
Formatting 'devstack-f.qcow2', fmt=qcow2 size=10737418240 backing_file='f19-minimal.qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 
# virsh start devstack-f
error: Failed to start domain devstack-f
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/7 (label charserial0)
qemu-system-x86_64: -drive file=/var/lib/libvirt/images/devstack-f.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /var/lib/libvirt/images/devstack-f.qcow2: Permission denied

To sum up: 
* I've confirmed that adding -o backing_fmt=qcow2 does help
* I also apologize for inaccurate original info. 
* I went through my history file and not a single time did I use backing_fmt in the past while everything was working, always was using (successfully):

# qemu-img create -b f19-minimal.qcow2 -f qcow2 devstack-f.qcow2

So there must've been a recent change (my last history entry of this kind where everything worked is from around Nov 19, 2013 give-or-take a day or two, and I just have noticed that there was an update on Nov 19 with lots of qemu-1.4.2-13 and libvirt-1.0.5.7-1 updates if that helps)

Comment 6 Eric Blake 2013-12-18 03:16:05 UTC
(In reply to Dmitry S. Makovey from comment #5)
> ok. I got your points. I will try to use backing_fmt in the future. However
> I have images that already are in this state so I'll be resorting to
> fake-labeling them with svirt_image_t.

No need to do that; qemu-img is your friend.

Given your file:
# qemu-img info --backing-chain vm1-active.qcow2
image: vm1-active.qcow2
file format: qcow2
virtual size: 7.0G (7516192768 bytes)
disk size: 8.2M
cluster_size: 65536
backing file: /var/lib/libvirt/images/vm1-stage.qcow2

Use:
qemu-img rebase -f qcow2 -u -b /var/lib/libvirt/images/vm1-stage.qcow2 -F qcow2 vm1-active.qcow2

and you'll have effectively added in the missing backing_fmt option.


> To sum up: 
> * I've confirmed that adding -o backing_fmt=qcow2 does help
> * I also apologize for inaccurate original info. 
> * I went through my history file and not a single time did I use backing_fmt
> in the past while everything was working, always was using (successfully):

For only a single level deep, the fact that libvirt treats the backing file as raw but qemu treats it as qcow2 via probing is not a violation of sVirt.  The problem happens when the backing chain is 3 deep, but libvirt stopped labelling at 2 deep because libvirt treated a file as raw even though qemu would treat it as qcow2.

> 
> # qemu-img create -b f19-minimal.qcow2 -f qcow2 devstack-f.qcow2
> 
> So there must've been a recent change (my last history entry of this kind
> where everything worked is from around Nov 19, 2013 give-or-take a day or
> two, and I just have noticed that there was an update on Nov 19 with lots of
> qemu-1.4.2-13 and libvirt-1.0.5.7-1 updates if that helps)

That doesn't tell me what you upgraded from; and while we have patched bugs related to libvirt incorrectly parsing backing chains, I don't see any particular patches related to backing chains directly in 1.0.5.7 in relation to 1.0.5.6.


Note You need to log in before you can comment on or make changes to this bug.