Description of problem: One of the host in the cluster seems not able to start vms or to hotplug with ISCSI disks. There's another host in the same cluster that doesn't have this issue, I've checked the packages and there's doesn't seem to be an issue with it. Also the host works as SPM for typical operations, migration of disks, adding domains, ... Packages are the proper ones for the release. Wonder if you guys can take a look. Version-Release number of selected component (if applicable): rhevm-3.6.1.3-0.1.el6.noarch libvirt-daemon-kvm-1.2.17-13.el7_2.2.ppc64le libvirt-docs-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-lxc-1.2.17-13.el7_2.2.ppc64le libvirt-lock-sanlock-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.2.ppc64le libvirt-python-1.2.17-2.el7.ppc64le libvirt-daemon-driver-qemu-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-interface-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-config-network-1.2.17-13.el7_2.2.ppc64le libvirt-debuginfo-1.2.17-13.el7_2.2.ppc64le libvirt-client-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-storage-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-secret-1.2.17-13.el7_2.2.ppc64le libvirt-devel-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-driver-network-1.2.17-13.el7_2.2.ppc64le libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.2.ppc64le libvirt-login-shell-1.2.17-13.el7_2.2.ppc64le qemu-img-rhev-2.3.0-31.el7_2.4.ppc64le ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch libvirt-daemon-driver-qemu-1.2.17-13.el7_2.2.ppc64le qemu-kvm-tools-rhev-2.3.0-31.el7_2.4.ppc64le qemu-kvm-rhev-2.3.0-31.el7_2.4.ppc64le qemu-kvm-common-rhev-2.3.0-31.el7_2.4.ppc64le vdsm-jsonrpc-4.17.13-1.el7ev.noarch vdsm-xmlrpc-4.17.13-1.el7ev.noarch vdsm-python-4.17.13-1.el7ev.noarch vdsm-4.17.13-1.el7ev.noarch vdsm-infra-4.17.13-1.el7ev.noarch vdsm-yajsonrpc-4.17.13-1.el7ev.noarch vdsm-cli-4.17.13-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. Create a vm with a boot disk on the iscsi domain (or use an already created one) 2. Try to start the vm Actual results: Thread-23957::ERROR::2016-01-12 05:34:03,418::vm::758::virt.vm::(_startUnderlyingVm) vmId=`ccfc6e2b-60dc-4b29-a10f-ddc6d00b1c99`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 702, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1889, in _run self._connection.createXML(domxml, flags), File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: internal error: process exited while connecting to monitor: 2016-01-12T10:34:03.210899Z qemu-kvm: -drive file=/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/bc7ac735-26d4-4bbd-a45b-0ac909896d00/images/9b158596-e5fe-40d5-95ce-da802a07756a/1ec33a65-7728-4259-a36b-9c1508907e35,if=none,id=drive-virtio-disk1,format=qcow2,serial=9b158596-e5fe-40d5-95ce-da802a07756a,cache=none,werror=stop,rerror=stop,aio=native: Could not open '/rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/bc7ac735-26d4-4bbd-a45b-0ac909896d00/images/9b158596-e5fe-40d5-95ce-da802a07756a/1ec33a65-7728-4259-a36b-9c1508907e35': Permission denied Also regarding the hotplug: Steps to Reproduce: 1. Use a vm with a boot disk on an nfs domain and start it 2. Hotplug a iscsi disk (in VMs -> Disks -> New) Actual results: Disk is added but fails to hotplug with: [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (ajp-/127.0.0.1:8702-2) [5ab6ea5f] Failed in 'HotPlugDiskVDS' method 2016-01-10 02:37:01,532 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [5ab6ea5f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host_mixed_1 command failed: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk0' could not be initialized Additional info: - The host doesn't seem to have any issue with nfs or to handle iscsi domains (add them/remove, create disks/migrate, ...) - The other host in the cluster doesn't have any issue like this one (also doesn't change anything if any of the host is the SPM)
kernel version on host: kernel-3.10.0-327.2.1.el7.ppc64le guest kernel version: kernel-3.10.0-327.2.1.el7.ppc64le
Created attachment 1113922 [details] vdsm.log vdsm log, multiple tries to start vms/hotplug. Look for the same string as in the description of the bug. Just point out that I've also tried to remove and add again the host and the issue still persists.
Created attachment 1113923 [details] qemu log for the failed start of the vm
selinux packages (same in both hosts): libselinux-utils-2.2.2-6.el7.ppc64le selinux-policy-targeted-3.13.1-60.el7.noarch selinux-policy-3.13.1-60.el7.noarch libselinux-python-2.2.2-6.el7.ppc64le libselinux-2.2.2-6.el7.ppc64le SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28
Changed mode to permisse instead of enforcing and works. Checking the selinux settings now
Following the fact that problem is related to selinux, found that there is a differences between hosts selinux labeling, for the images directory. After getting this permission denied for this path on the failing host (on start VM/disk hotplug): /rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/0526738b-1aa3-4389-a542-c3916a47813b/images/b6c6db65-d023-4326-a15b-5b84e85376d7/c214b4bd-ad23-47c8-bf5d-872065ec6bb7 ls -Lz on the images directory showed this label for the images: vdsm kvm system_u:object_r:unlabeled_t:s0 While on the second host, that has NO issues, the label for the images is: vdsm kvm system_u:object_r:mnt_t:s0 At this stage it is not clear if when storage was initially mounted selinux was disabled or not. If indeed it was disabled, this bug might be a duplicate of BZ 1271573
This is a test of the positive case of having a host with selinux Enforced, mount the storage, and see that the labeling is correct: For the failing host (one that missing labels): Removed it from the setup, rebooted it, verified it's selinux is Enforcing, Installed it again in the setup (to the same dataCenter, that has the iscsi connections). Now that the storage is mounted from new, operations like start VM work OK. Also the labeling is correct: [root@ibm-p8-rhevm-03 qemu]# ls -lZ /rhev/data-center/b3115183-d522-428b-9dce-2809fe39a79d/aa1d1568-448c-48fe-aad8-2c5b128b7d05/images/ drwxr-xr-x. vdsm kvm system_u:object_r:nfs_t:s0 6e4b57a6-7ed8-42a9-a07c-9d59b7a46e8e With this result, we can close this bug on duplicate of BZ 1271573. Also removing the dependency of Power, as it is not PPC specific. *** This bug has been marked as a duplicate of bug 1271573 ***
*** Bug 1297760 has been marked as a duplicate of this bug. ***