Description of problem: Cannot start virtual machine. # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor 6.6 (20150319.42.el6ev) # rpm -qa | grep -i libvirt libvirt-client-0.10.2-46.el6_6.3.x86_64 libvirt-python-0.10.2-46.el6_6.3.x86_64 libvirt-lock-sanlock-0.10.2-46.el6_6.3.x86_64 libvirt-0.10.2-46.el6_6.3.x86_64 libvirt-cim-0.6.1-12.el6.x86_64 # rpm -qa | grep -i vdsm vdsm-reg-4.16.12.1-3.el6ev.noarch ovirt-node-plugin-vdsm-0.2.0-20.el6ev.noarch vdsm-python-zombiereaper-4.16.12.1-3.el6ev.noarch vdsm-jsonrpc-4.16.12.1-3.el6ev.noarch vdsm-python-4.16.12.1-3.el6ev.noarch vdsm-cli-4.16.12.1-3.el6ev.noarch vdsm-hook-vhostmd-4.16.12.1-3.el6ev.noarch vdsm-4.16.12.1-3.el6ev.x86_64 vdsm-yajsonrpc-4.16.12.1-3.el6ev.noarch vdsm-hook-ethtool-options-4.16.12.1-3.el6ev.noarch vdsm-xmlrpc-4.16.12.1-3.el6ev.noarch Red Hat Enterprise Virtualization Manager Version: 3.5.1-0.2.el6ev # vdsClient -s 0 getVdsCaps HBAInventory = {'FC': [], 'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:b37466b21b8e'}]} ISCSIInitiatorName = 'iqn.1994-05.com.redhat:b37466b21b8e' autoNumaBalancing = 2 bondings = {'bond0': {'addr': '', 'cfg': {}, 'hwaddr': '00:00:00:00:00:00', 'mtu': '1500', 'netmask': '', 'slaves': []}, 'bond1': {'addr': '', 'cfg': {}, 'hwaddr': '00:00:00:00:00:00', 'mtu': '1500', 'netmask': '', 'slaves': []}, 'bond2': {'addr': '', 'cfg': {}, 'hwaddr': '00:00:00:00:00:00', 'mtu': '1500', 'netmask': '', 'slaves': []}, 'bond3': {'addr': '', 'cfg': {}, 'hwaddr': '00:00:00:00:00:00', 'mtu': '1500', 'netmask': '', 'slaves': []}, 'bond4': {'addr': '', <snip> # service libvirtd status libvirtd (pid 16380) is running... # virsh Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # list Please enter your authentication name: vdsm@rhevh Please enter your password: Id Name State ---------------------------------------------------- virsh # How reproducible: #1) Execute auto-install on RHEV-H 6.6 (20150319.42.el6ev) Use the params: firstboot storage_init=/dev/sda adminpw=RHhwCLrQXB8zE management_server=192.168.122.70 BOOTIF=link #2) Add the data storage via RHEV-M admin page (In my case,ISCSI) #3) Add the ISO storage via RHEV-M admin page #4) Copy a .iso file to ISO storage #5) Create a virtual machine and try to run it. Actual results: VM vm is down with error. Exit message: Child quit during startup handshake: Input/output error. Expected results: Virtual machine run without issue. Additional info: This error is only noticed when using autoinstall on RHEV-H.
Created attachment 1005094 [details] logs
This initial error seems related to selinux, setting setenfor to 0 we go to a different error. For now, I am moving this bug to rhev-hypervisor. audit.log ============= <snip> type=AVC msg=audit(1427068608.315:2153): avc: denied { connectto } for pid=32549 comm="libvirtd" path="/var/run/sanlock/sanlock.sock" scontext=unconfined_u:system_r:svirt_t:s0:c326,c682 tcontext=system_u:system_r:ovirt_t:s0 tclass=unix_stream_socket type=SYSCALL msg=audit(1427068608.315:2153): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=7f4c3d3e9890 a2=6e a3=0 items=0 ppid=1 pid=32549 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=2 comm="libvirtd" exe="/usr/sbin/libvirtd" subj=unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 key=(null) type=ANOM_ABEND msg=audit(1427068608.315:2154): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 pid=32549 comm="libvirtd" sig=11 #audit2allow -a #============= svirt_t ============== allow svirt_t ovirt_t:unix_stream_socket connectto; However, now I am getting the below error trying to start the virtual machine: 2015-03-22 23:59:50.069+0000: 16382: error : qemuProcessWaitForMonitor:1858 : internal error process exited while connecting to monitor: ((null):728): Spice-Warning **: reds.c:3269:reds_init_ssl: Could not use private key file failed to initialize spice server 2015-03-23 00:00:21.244+0000: 16383: error : qemuMonitorOpenUnix:294 : failed to connect to monitor socket: No such process 2015-03-23 00:00:21.244+0000: 16383: error : qemuProcessWaitForMonitor:1858 : internal error process exited while connecting to monitor: ((null):1122): Spice-Warning **: reds.c:3269:reds_init_ssl: Could not use private key file failed to initialize spice server # pwd /etc/pki/libvirt [root@localhost libvirt]# ls -la -R .: total 3 drwxr-xr-x. 3 vdsm kvm 80 2015-03-22 22:23 . drwxr-xr-x. 12 root root 240 2015-03-22 22:23 .. -rw-r--r--. 1 root root 1554 2015-03-22 22:23 clientcert.pem drwxr-xr-x. 2 vdsm kvm 60 2015-03-22 22:23 private ./private: total 3 drwxr-xr-x. 2 vdsm kvm 60 2015-03-22 22:23 . drwxr-xr-x. 3 vdsm kvm 80 2015-03-22 22:23 .. -r--r-----. 1 root root 1679 2015-03-22 22:23 clientkey.pem @Jiri, Could you please review the above error from libvirt/spice? Ideas? Should we open a different bug? Thanks!
Well, unless qemu is a member of root group, it can't read clientkey.pem. I don't see any bug in libvirt here.
This looks similar to bug 1188255
Running chown -R vdsm:kvm /etc/pki/vdsm fixes the issue for me. Dan, can you tell what the correct owner of the files in /etc/pki/vdsm should be? And who is taking care of setting the correct permissions?
/etc/pki/vdsm is installed vdsm:kvm by vdsm.rpm. Nothing should have changed that, if I correctly recall.
(In reply to Fabian Deutsch from comment #8) > Running chown -R vdsm:kvm /etc/pki/vdsm fixes the issue for me. Before that call, the permissions were as follows: /etc/pki/vdms/keys root:root vdsmkey.pem /etc/pki/vdsm/libvirt-spice root:root ca-cert.pem root:root server-cert.pem root:root server-key.pem And for other pki related files it was the same (root:root)
Created attachment 1005374 [details] pki file permissions after vdsm-reg but before approval
Created attachment 1005376 [details] pki file permissions after approval and node up
Created attachment 1005377 [details] /etc and /var/logs from a run with incorrect permissions
Created attachment 1005388 [details] host-deploy logs from the run with the wrong permissions The last few attachments include node and engine side logs from a run where the final file permissions of files in /etc/pki/vdsm (maybe /etc/pki/libvirt as well), where incorrect. The steps to reproduce were: 1. Install latest vt engine 2. Install Latest 6.6 based RHEV-H in TUI mode 3. Register RHEV-H to Engine using the TUI (vdsm-reg) 4. Approve node in Engine 5. Configure node with local storage domain 6. Create a PXE booting VM and launch it
Created attachment 1005437 [details] Logs from a failed run, including permissions before and after
Created attachment 1005438 [details] host-deploy logs from a failed run
I could reproduce this bug in the follow version: rhev-hypervisor6-6.6-20150319.42.iso ovirt-node-3.2.1-11.el6.noarch vdsm-reg-4.16.12.1-3.el6ev.noarch ovirt-node-plugin-vdsm-0.2.0-20.el6ev.noarch vdsm-4.16.12.1-3.el6ev.x86_64 Red Hat Enterprise Virtualization Manager Version: 3.5.1-0.2.el6ev Test steps: 1. Clean install latest vt14.1 engine 2. Install rhev-hypervisor6-6.6-20150319.42.iso in TUI mode 3. Register RHEV-H to Engine using the TUI (vdsm-reg) 4. Approve node in Engine 5. Configure node with local storage domain 6. Create a VM and launch it. Test result: After step6, VM vm is down with the follow error info: 2015-Mar-24, 14:13 Failed to run VM vm1 (User: admin@internal). 2015-Mar-24, 14:13 Failed to run VM vm1 on Host dhcp-8-165.nay.redhat.com. 2015-Mar-24, 14:13 VM vm1 is down with error. Exit message: Child quit during startup handshake: Input/output error.
No this issue for RHEV-H 6.6 for RHEV 3.5 GA build version: rhev-hypervisor6-6.6-20150128.0.el6ev.noarch.rpm Red Hat Enterprise Virtualization Manager Version: 3.5.0-0.34.el6ev so it should be a regression bug.
See comment 20, here already cloned this bug to 3.5.z bug #1206537, so remove this 3.5.1 tracker bug 1193058 from this bug.
According comment 17, verify this bug with the following steps on build rhev-hypervisor7-7.2-20151104.0 Version: rhev-hypervisor7-7.2-20151104.0.iso ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch Steps: 1. Install rhev-hypervisor7-7.2-20151104.0.iso in TUI mode and configure network successful. 2. Register rhevm to rhevm3.6.0.3 in RHEV-M page 3. Approve rhevh on RHEV-M 4. Configure rhevh with NFS storage. 5. Create a VM and launch it. Result: After step5, VM install and launch successfully. This bug has been fixed, so change the status to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0378.html