Red Hat Bugzilla – Bug 1297765
qemu-kvm permission denied to access image on iscsi domain (unable to start the vm)
Last modified: 2016-02-10 12:23:25 EST
Description of problem:
One of the host in the cluster seems not able to start vms or to hotplug with ISCSI disks. There's another host in the same cluster that doesn't have this issue, I've checked the packages and there's doesn't seem to be an issue with it. Also the host works as SPM for typical operations, migration of disks, adding domains, ...
Packages are the proper ones for the release. Wonder if you guys can take a look.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Create a vm with a boot disk on the iscsi domain (or use an already created one)
2. Try to start the vm
Thread-23957::ERROR::2016-01-12 05:34:03,418::vm::758::virt.vm::(_startUnderlyingVm) vmId=`ccfc6e2b-60dc-4b29-a10f-ddc6d00b1c99`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 702, in _startUnderlyingVm
File "/usr/share/vdsm/virt/vm.py", line 1889, in _run
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2016-01-12T10:34:03.210899Z qemu-kvm: -drive file=/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/bc7ac735-26d4-4bbd-a45b-0ac909896d00/images/9b158596-e5fe-40d5-95ce-da802a07756a/1ec33a65-7728-4259-a36b-9c1508907e35,if=none,id=drive-virtio-disk1,format=qcow2,serial=9b158596-e5fe-40d5-95ce-da802a07756a,cache=none,werror=stop,rerror=stop,aio=native: Could not open '/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/bc7ac735-26d4-4bbd-a45b-0ac909896d00/images/9b158596-e5fe-40d5-95ce-da802a07756a/1ec33a65-7728-4259-a36b-9c1508907e35': Permission denied
Also regarding the hotplug:
Steps to Reproduce:
1. Use a vm with a boot disk on an nfs domain and start it
2. Hotplug a iscsi disk (in VMs -> Disks -> New)
Disk is added but fails to hotplug with:
[org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (ajp-/127.0.0.1:8702-2) [5ab6ea5f] Failed in 'HotPlugDiskVDS' method
2016-01-10 02:37:01,532 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [5ab6ea5f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host_mixed_1 command failed: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk0' could not be initialized
- The host doesn't seem to have any issue with nfs or to handle iscsi domains (add them/remove, create disks/migrate, ...)
- The other host in the cluster doesn't have any issue like this one (also doesn't change anything if any of the host is the SPM)
kernel version on host: kernel-3.10.0-327.2.1.el7.ppc64le
guest kernel version: kernel-3.10.0-327.2.1.el7.ppc64le
Created attachment 1113922 [details]
vdsm log, multiple tries to start vms/hotplug. Look for the same string as in the description of the bug.
Just point out that I've also tried to remove and add again the host and the issue still persists.
Created attachment 1113923 [details]
qemu log for the failed start of the vm
selinux packages (same in both hosts):
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
Changed mode to permisse instead of enforcing and works. Checking the selinux settings now
Following the fact that problem is related to selinux,
found that there is a differences between hosts selinux labeling, for the images directory.
After getting this permission denied for this path on the failing host (on start VM/disk hotplug):
ls -Lz on the images directory showed this label for the images:
vdsm kvm system_u:object_r:unlabeled_t:s0
While on the second host, that has NO issues, the label for the images is:
vdsm kvm system_u:object_r:mnt_t:s0
At this stage it is not clear if when storage was initially mounted selinux was disabled or not.
If indeed it was disabled, this bug might be a duplicate of BZ 1271573
This is a test of the positive case of having a host with selinux Enforced,
mount the storage,
and see that the labeling is correct:
For the failing host (one that missing labels):
Removed it from the setup,
verified it's selinux is Enforcing,
Installed it again in the setup (to the same dataCenter, that has the iscsi connections).
Now that the storage is mounted from new,
operations like start VM work OK.
Also the labeling is correct:
[root@ibm-p8-rhevm-03 qemu]# ls -lZ /rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/aa1d1568-448c-48fe-aad8-2c5b128b7d05/images/
drwxr-xr-x. vdsm kvm system_u:object_r:nfs_t:s0 6e4b57a6-7ed8-42a9-a07c-9d59b7a46e8e
With this result,
we can close this bug on duplicate of BZ 1271573.
Also removing the dependency of Power, as it is not PPC specific.
*** This bug has been marked as a duplicate of bug 1271573 ***
*** Bug 1297760 has been marked as a duplicate of this bug. ***