Bug 1297765 - qemu-kvm permission denied to access image on iscsi domain (unable to start the vm)
qemu-kvm permission denied to access image on iscsi domain (unable to start t...
Status: CLOSED DUPLICATE of bug 1271573
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage (Show other bugs)
3.6.1
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-4.0.0-alpha
: 4.0.0
Assigned To: Allon Mureinik
Aharon Canan
storage
:
: 1297760 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-12 07:24 EST by Carlos Mestre González
Modified: 2016-02-10 12:23 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-14 07:12:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
amureini: ovirt‑4.0.0?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
vdsm.log (15.62 MB, text/plain)
2016-01-12 07:33 EST, Carlos Mestre González
no flags Details
qemu log for the failed start of the vm (12.11 KB, text/plain)
2016-01-12 07:35 EST, Carlos Mestre González
no flags Details

  None (edit)
Description Carlos Mestre González 2016-01-12 07:24:10 EST
Description of problem:
One of the host in the cluster seems not able to start vms or to hotplug with ISCSI disks. There's another host in the same cluster that doesn't have this issue, I've checked the packages and there's doesn't seem to be an issue with it. Also the host works as SPM for typical operations, migration of disks, adding domains, ...

Packages are the proper ones for the release. Wonder if you guys can take a look.


Version-Release number of selected component (if applicable):
rhevm-3.6.1.3-0.1.el6.noarch
libvirt-daemon-kvm-1.2.17-13.el7_2.2.ppc64le
libvirt-docs-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.2.ppc64le
libvirt-lock-sanlock-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.2.ppc64le
libvirt-python-1.2.17-2.el7.ppc64le
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-interface-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-config-network-1.2.17-13.el7_2.2.ppc64le
libvirt-debuginfo-1.2.17-13.el7_2.2.ppc64le
libvirt-client-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-storage-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-secret-1.2.17-13.el7_2.2.ppc64le
libvirt-devel-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-network-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.2.ppc64le
libvirt-login-shell-1.2.17-13.el7_2.2.ppc64le
qemu-img-rhev-2.3.0-31.el7_2.4.ppc64le
ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.2.ppc64le
qemu-kvm-tools-rhev-2.3.0-31.el7_2.4.ppc64le
qemu-kvm-rhev-2.3.0-31.el7_2.4.ppc64le
qemu-kvm-common-rhev-2.3.0-31.el7_2.4.ppc64le
vdsm-jsonrpc-4.17.13-1.el7ev.noarch
vdsm-xmlrpc-4.17.13-1.el7ev.noarch
vdsm-python-4.17.13-1.el7ev.noarch
vdsm-4.17.13-1.el7ev.noarch
vdsm-infra-4.17.13-1.el7ev.noarch
vdsm-yajsonrpc-4.17.13-1.el7ev.noarch
vdsm-cli-4.17.13-1.el7ev.noarch


How reproducible:
100%

Steps to Reproduce:
1. Create a vm with a boot disk on the iscsi domain (or use an already created one)
2. Try to start the vm

Actual results:
Thread-23957::ERROR::2016-01-12 05:34:03,418::vm::758::virt.vm::(_startUnderlyingVm) vmId=`ccfc6e2b-60dc-4b29-a10f-ddc6d00b1c99`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 702, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/virt/vm.py", line 1889, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2016-01-12T10:34:03.210899Z qemu-kvm: -drive file=/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/bc7ac735-26d4-4bbd-a45b-0ac909896d00/images/9b158596-e5fe-40d5-95ce-da802a07756a/1ec33a65-7728-4259-a36b-9c1508907e35,if=none,id=drive-virtio-disk1,format=qcow2,serial=9b158596-e5fe-40d5-95ce-da802a07756a,cache=none,werror=stop,rerror=stop,aio=native: Could not open '/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/bc7ac735-26d4-4bbd-a45b-0ac909896d00/images/9b158596-e5fe-40d5-95ce-da802a07756a/1ec33a65-7728-4259-a36b-9c1508907e35': Permission denied

Also regarding the hotplug:

Steps to Reproduce:
1. Use a vm with a boot disk on an nfs domain and start it
2. Hotplug a iscsi disk (in VMs -> Disks -> New)

Actual results:
Disk is added but fails to hotplug with:
[org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (ajp-/127.0.0.1:8702-2) [5ab6ea5f] Failed in 'HotPlugDiskVDS' method
2016-01-10 02:37:01,532 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [5ab6ea5f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host_mixed_1 command failed: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk0' could not be initialized


Additional info:
- The host doesn't seem to have any issue with nfs or to handle iscsi domains (add them/remove, create disks/migrate, ...)
- The other host in the cluster doesn't have any issue like this one (also doesn't change anything if any of the host is the SPM)
Comment 2 Carlos Mestre González 2016-01-12 07:32:19 EST
kernel version on host: kernel-3.10.0-327.2.1.el7.ppc64le

guest kernel version:   kernel-3.10.0-327.2.1.el7.ppc64le
Comment 3 Carlos Mestre González 2016-01-12 07:33 EST
Created attachment 1113922 [details]
vdsm.log

vdsm log, multiple tries to start vms/hotplug. Look for the same string as in the description of the bug.

Just point out that I've also tried to remove and add again the host and the issue still persists.
Comment 4 Carlos Mestre González 2016-01-12 07:35 EST
Created attachment 1113923 [details]
qemu log for the failed start of the vm
Comment 5 Carlos Mestre González 2016-01-12 07:37:14 EST
selinux packages (same in both hosts):

libselinux-utils-2.2.2-6.el7.ppc64le
selinux-policy-targeted-3.13.1-60.el7.noarch
selinux-policy-3.13.1-60.el7.noarch
libselinux-python-2.2.2-6.el7.ppc64le
libselinux-2.2.2-6.el7.ppc64le

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28
Comment 6 Carlos Mestre González 2016-01-12 08:47:18 EST
Changed mode to permisse instead of enforcing and works. Checking the selinux settings now
Comment 7 Ilanit Stein 2016-01-12 11:02:34 EST
Following the fact that problem is related to selinux,
found that there is a differences between hosts selinux labeling, for the images directory. 

After getting this permission denied for this path on the failing host (on start VM/disk hotplug):
/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/0526738b-1aa3-4389-a542-c3916a47813b/images/b6c6db65-d023-4326-a15b-5b84e85376d7/c214b4bd-ad23-47c8-bf5d-872065ec6bb7
 
ls -Lz on the images directory showed this label for the images:
vdsm kvm system_u:object_r:unlabeled_t:s0

While on the second host, that has NO issues, the label for the images is:
vdsm kvm system_u:object_r:mnt_t:s0 

At this stage it is not clear if when storage was initially mounted selinux was disabled or not.

If indeed it was disabled, this bug might be a duplicate of BZ 1271573
Comment 8 Ilanit Stein 2016-01-14 07:12:11 EST
This is a test of the positive case of having a host with selinux Enforced,
mount the storage,
and see that the labeling is correct:


For the failing host (one that missing labels):
Removed it from the setup,
rebooted it,
verified it's selinux is Enforcing,
Installed it again in the setup (to the same dataCenter, that has the iscsi connections).
Now that the storage is mounted from new,
operations like start VM work OK.

Also the labeling is correct:

[root@ibm-p8-rhevm-03 qemu]# ls -lZ /rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/aa1d1568-448c-48fe-aad8-2c5b128b7d05/images/
drwxr-xr-x. vdsm kvm system_u:object_r:nfs_t:s0       6e4b57a6-7ed8-42a9-a07c-9d59b7a46e8e


With this result,
we can close this bug on duplicate of BZ 1271573. 

Also removing the dependency of Power, as it is not PPC specific.

*** This bug has been marked as a duplicate of bug 1271573 ***
Comment 9 Allon Mureinik 2016-01-14 07:56:58 EST
*** Bug 1297760 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.