Bug 1361592 - image backing file cannot be accessed when adding device to running domain due to incomplete apparmor profile
Summary: image backing file cannot be accessed when adding device to running domain du...
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-29 13:10 UTC by silvan
Modified: 2018-08-15 17:58 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)

Description silvan 2016-07-29 13:10:59 UTC
Description of problem:
When trying to attach an image as a device to a running vm the attachment fails if the image file has a backing file (e.g. from an external snapshot). Afaics the apparmor profile of the VM is extended only for the image file itself but not it's backing file. Libvirt fails to attach when trying to access the images backing file afterwards. The libvirt log shows:

------libvirt.log excerpt-------
2016-07-27 10:31:23.135+0000: 16633: debug : qemuDomainPCIAddressReserveAddr:2114 : Reserving PCI slot 0000:00:05.0 (multifunction='off')
2016-07-27 10:31:23.193+0000: 16633: debug : qemuSetupDiskPathAllow:60 : Process path /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d480
5690e81b for disk
2016-07-27 10:31:23.194+0000: 16633: debug : qemuSetupDiskPathAllow:60 : Process path /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.2f3c2d0e-5315-48f3-9fc7-7e39
2c7958e1 for disk
2016-07-27 10:31:23.194+0000: 16633: debug : qemuSetupDiskPathAllow:60 : Process path /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0 for disk
2016-07-27 10:31:23.197+0000: 16633: debug : virCommandRunAsync:2282 : About to run /usr/lib/libvirt/virt-aa-helper -p 0 -r -u libvirt-4df0bca7-5e5c-47b8-9c27-d40277351fb0 -f /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b
2016-07-27 10:31:23.197+0000: 16633: debug : virCommandRunAsync:2285 : Command result 0, with PID 10895
2016-07-27 10:31:23.521+0000: 16633: debug : virCommandRun:2142 : Result status 0, stdout: '' stderr: ''
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:01 already in use
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:02 already in use
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:03 already in use
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:04 already in use
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainPCIAddressGetNextSlot:2313 : Found free PCI slot 0000:00:05
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainPCIAddressReserveAddr:2114 : Reserving PCI slot 0000:00:05.0 (multifunction='off')
2016-07-27 10:31:23.521+0000: 16633: debug : qemuDomainObjEnterMonitorInternal:1278 : Entering monitor (mon=0x7fc904181380 vm=0x7fc8f80117a0 name=instance-00000015)
2016-07-27 10:31:23.521+0000: 16633: debug : qemuMonitorAddDrive:3011 : mon=0x7fc904181380 drive=file=/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f30
0-4927-9231-d4805690e81b,if=none,id=drive-virtio-disk1,format=qcow2,serial=605eb107-baf5-4a10-a2b9-60a3a77f2fc0,cache=none,aio=native
2016-07-27 10:31:23.521+0000: 16633: debug : qemuMonitorJSONAddDrive:3126 : drive_add command not found, trying HMP
2016-07-27 10:31:23.521+0000: 16633: debug : qemuMonitorJSONCommandWithFd:264 : Send command '{"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=/mnt/quobyte-volume/abfa1002557a
b2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b,if=none,id=drive-virtio-disk1,format=qcow2,serial=605eb107-baf5-4a10-a2b9-60a3a77f2fc0,cache=none,aio=native"
},"id":"libvirt-16"}' for write with FD -1
2016-07-27 10:31:23.521+0000: 16633: debug : qemuMonitorSend:959 : QEMU_MONITOR_SEND_MSG: mon=0x7fc904181380 msg={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=/mnt/quobyte-
volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b,if=none,id=drive-virtio-disk1,format=qcow2,serial=605eb107-baf5-4a10-a2b9-60a3a77f2fc0,cache=none,aio=native"},"id":"libvirt-16"}
 fd=-1
2016-07-27 10:31:23.521+0000: 16628: debug : qemuMonitorIOWrite:504 : QEMU_MONITOR_IO_WRITE: mon=0x7fc904181380 buf={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b,if=none,id=drive-virtio-disk1,format=qcow2,serial=605eb107-baf5-4a10-a2b9-60a3a77f2fc0,cache=none,aio=native"},"id":"libvirt-16"}
 len=350 ret=350 errno=11
2016-07-27 10:31:23.632+0000: 16628: debug : qemuMonitorIOProcess:396 : QEMU_MONITOR_IO_PROCESS: mon=0x7fc904181380 buf={"return": "could not open disk image /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b: Could not open backing file: Could not open '/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.2f3c2d0e-5315-48f3-9fc7-7e392c7958e1': Permission denied\r\n", "id": "libvirt-16"}
 len=399
2016-07-27 10:31:23.632+0000: 16628: debug : qemuMonitorJSONIOProcessLine:157 : Line [{"return": "could not open disk image /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b: Could not open backing file: Could not open '/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.2f3c2d0e-5315-48f3-9fc7-7e392c7958e1': Permission denied\r\n", "id": "libvirt-16"}]
------end of excerpt----------

(image file: /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0.1a65bc67-f300-4927-9231-d4805690e81b)
(backing file: /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-605eb107-baf5-4a10-a2b9-60a3a77f2fc0)

I do not know if there should be two calls to the virt-aa-helper (one for the image itself and one for it's backing file) or if virt-aa-helper should analyse the image and deduct the requirement of adding the images backing file by itself.

Attaching the image works if the vm is shut down and started afterwards. In this latter case the apparmor profile of the VM contains the files of the image as well as it's backing file and everything is fine.

Things also work if the image has no backing file, the image file device is added to the profile ok and the device is added in the vm (this is from a different test run therefore different filename & timestamps):

-------------libvirt.log excerpt---------------
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainObjEnterMonitorInternal:1278 : Entering monitor (mon=0x7fa6c800c1f0 vm=0x7fa6dc003a30 name=instance-0000000a)  
2016-07-29 12:53:51.645+0000: 6958: debug : qemuMonitorAddDrive:3011 : mon=0x7fa6c800c1f0 drive=file=/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-48535ec1-31be-422a-8285-da24fbf5cc44,if=none,id=drive-virtio-disk1,format=qcow2,serial=48535ec1-31be-422a-8285-da24fbf5cc44,cache=none,aio=native
2016-07-29 12:53:51.645+0000: 6958: debug : qemuMonitorJSONAddDrive:3126 : drive_add command not found, trying HMP  
2016-07-29 12:53:51.645+0000: 6958: debug : qemuMonitorJSONCommandWithFd:264 : Send command '{"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-48535ec1-31be-422a-8285-da24fbf5cc44,if=none,id=drive-virtio-disk1,format=qcow2,serial=48535ec1-31be-422a-8285-da24fbf5cc44,cache=none,aio=native"},"id":"libvirt-16"}' for write with FD -1  
2016-07-29 12:53:51.645+0000: 6958: debug : qemuMonitorSend:959 : QEMU_MONITOR_SEND_MSG: mon=0x7fa6c800c1f0 msg={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-48535ec1-31be-422a-8285-da24fbf5cc44,if=none,id=drive-virtio-disk1,format=qcow2,serial=48535ec1-31be-422a-8285-da24fbf5cc44,cache=none,aio=native"},"id":"libvirt-16"}
 fd=-1  
2016-07-29 12:53:51.648+0000: 6955: debug : qemuMonitorIOWrite:504 : QEMU_MONITOR_IO_WRITE: mon=0x7fa6c800c1f0 buf={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-48535ec1-31be-422a-8285-da24fbf5cc44,if=none,id=drive-virtio-disk1,format=qcow2,serial=48535ec1-31be-422a-8285-da24fbf5cc44,cache=none,aio=native"},"id":"libvirt-16"}
 len=313 ret=313 errno=11
2016-07-29 12:53:51.671+0000: 6955: debug : qemuMonitorIOProcess:396 : QEMU_MONITOR_IO_PROCESS: mon=0x7fa6c800c1f0 buf={"return": "OK\r\n", "id": "libvirt-16"}
 len=42
2016-07-29 12:53:51.671+0000: 6955: debug : qemuMonitorJSONIOProcessLine:157 : Line [{"return": "OK\r\n", "id": "libvirt-16"}]

-------------end of excerpt--------------------

This issue came up in the context of the OpenStack project but can be reproduced in a plain libvirt setup.


Version-Release number of selected component (if applicable):
These issues came up with an ubuntu 16.04 (apparmor=2.10.95-0ubuntu2, libvirt-bin=1.3.1-1ubuntu10.1) as well as an ubuntu 14.04 setup (apparmor 2.8.95~2430-0ubuntu5.3, libvirt 1.2.2-0ubuntu13.1.17).


How reproducible:
always


Steps to Reproduce:
1) create a volume -> works
2) attach volume to guest -> works
3) detach volume from guest -> works
4) create snapshot from volume -> works
5) attach volume(!) to guest -> FAIL
6) delete snapshot -> works
7) attach volume to guest -> works(!)
As described earlier, when running this procedure with a stopped guest things work fine, e.g.:
[..]
4) create snapshot from volume -> works
5) attach volume to (shutdown) guest -> works (checked by powering up and looking for the volumes device)

Actual results:
image cannot be attached as a device.

Expected results:
image should be attached as requested

Additional info:
- Most of the testing was done in an Openstack dev setup (DevStack) but i could verify the same results by running the underlying libvirt commands manually via virsh. Results stayed the same.
- The /etc/libvirt/qemu.conf file has some specific settings:
    user=<non root nova user>
    group=<non root cinder group>
    dynamic_ownership = 0
- security_driver is set to default in this setup

Comment 1 silvan 2016-07-29 13:36:07 UTC
The second log excerpt is cut a bit short. The following lines should be added at the top of the second log excerpt (they contain the actual virt-aa-helper call):

2016-07-29 12:53:51.297+0000: 6958: debug : qemuSetupDiskPathAllow:60 : Process path /mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-48535ec1-31be-422a-8285-da24fbf5cc44 for disk
2016-07-29 12:53:51.297+0000: 6958: debug : virCommandRunAsync:2282 : About to run /usr/lib/libvirt/virt-aa-helper -p 0 -r -u libvirt-692a3a05-4a42-4167-bec6-8876e99e3b8b -f /mnt/quobyte-volume/abfa1002557ab2b21
ec218a86487dd92/volume-48535ec1-31be-422a-8285-da24fbf5cc44
2016-07-29 12:53:51.298+0000: 6958: debug : virCommandRunAsync:2285 : Command result 0, with PID 17651
2016-07-29 12:53:51.645+0000: 6958: debug : virCommandRun:2142 : Result status 0, stdout: '' stderr: ''
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:01 already in use
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:02 already in use
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:03 already in use
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainPCIAddressGetNextSlot:2270 : PCI slot 0000:00:04 already in use
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainPCIAddressGetNextSlot:2313 : Found free PCI slot 0000:00:05
2016-07-29 12:53:51.645+0000: 6958: debug : qemuDomainPCIAddressReserveAddr:2114 : Reserving PCI slot 0000:00:05.0 (multifunction='off')

Comment 2 Erlon Cruz 2016-08-24 20:01:09 UTC
More info in this libvirt-users' thread.

https://www.redhat.com/archives/libvirt-users/2016-August/msg00048.html

Comment 3 Povilas Kanapickas 2018-08-15 16:09:05 UTC
I believe this bug is partially fixed as of libvirt v4.0.0. I see the path to the backing image in the apparmor profile.

Unfortunately, this only works for single level of backing images. If the backing image itself has a backing image then the latter is not included into the apparmor profile.

I did some debugging of the virt-aa-helper, will post the analysis shortly.

Comment 4 Povilas Kanapickas 2018-08-15 17:58:56 UTC
Details here: https://www.redhat.com/archives/libvir-list/2018-August/msg00959.html


Note You need to log in before you can comment on or make changes to this bug.