Red Hat Bugzilla – Bug 1482987
permissions on the file /etc/ssl/glusterfs.ca has to be changed so that qemu-kvm can read the file
Last modified: 2017-11-28 22:27:19 EST
Description of problem: Cannot start a vm on RHHI setup with libgfapi and SSL enabled. Version-Release number of selected component (if applicable): vdsm-4.19.28-1.el7ev.x86_64 How reproducible: Always Steps to Reproduce: 1. Install RHHI by Configure gluster and HostedEngine using cockpit UI and enable SSL on the setup. 2. Now create a vm and start it the vm. 3. Actual results: Following error is seen while starting the vm in te events tab in UI and qemu logs. VM applinuxvm1 is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: [2017-08-17 06:52:55.961192] E [socket.c:4320:socket_init] 0-gfapi: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2017-08-17 06:52:55.961371] E [socket.c:4406:socket_init] 0-gfapi: could not load CA list 2017-08-17T06:52:55.961686Z qemu-kvm: -drive file=gluster://10.70.36.79/vmstore/04396d72-6e34-4b00-b5e2-c440d33139be/images/4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a/78d8559f-b5c9-4450-ac9f-785401832472,file.debug=4,format=raw,if=none,id=drive-scsi0-0-0-0,serial=4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a,cache=none,werror=stop,rerror=stop,aio=threads: Gluster connection for volume vmstore, path 04396d72-6e34-4b00-b5e2-c440d33139be/images/4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a/78d8559f-b5c9-4450-ac9f-785401832472 failed to connect hint: failed on host 10.70.36.79 and port 24007 Please refer to gluster logs for more info. Expected results: vm should start successfully. Additional info: Traceback in vdsm logs: ============================ 2017-08-18 19:35:16,412+0530 ERROR (vm/9ee5655f) [virt.vm] (vmId='9ee5655f-9bd8-4d4b-ba61-da3a3c5a3ad6') The vm start process failed (vm:631) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 562, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 2022, in _run self._connection.createXML(domxml, flags), File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 944, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3567, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: internal error: qemu unexpectedly closed the monitor: [2017-08-18 14:05:16.191743] E [socket.c:4320:socket_init] 0-gfapi: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2017-08-18 14:05:16.191970] E [socket.c:4406:socket_init] 0-gfapi: could not load CA list 2017-08-18T14:05:16.192477Z qemu-kvm: -drive file=gluster://10.70.36.79/vmstore/04396d72-6e34-4b00-b5e2-c440d33139be/images/4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a/78d8559f-b5c 9-4450-ac9f-785401832472,file.debug=4,format=raw,if=none,id=drive-scsi0-0-0-0,serial=4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a,cache=none,werror=stop,rerror=stop,aio=threads: Glu ster connection for volume vmstore, path 04396d72-6e34-4b00-b5e2-c440d33139be/images/4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a/78d8559f-b5c9-4450-ac9f-785401832472 failed to conn ect hint: failed on host 10.70.36.79 and port 24007 Please refer to gluster logs for more info 2017-08-18 19:35:16,414+0530 INFO (vm/9ee5655f) [virt.vm] (vmId='9ee5655f-9bd8-4d4b-ba61-da3a3c5a3ad6') Changed state to Down: internal error: qemu unexpectedly closed the monitor: [2017-08-18 14:05:16.191743] E [socket.c:4320:socket_init] 0-gfapi: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2017-08-18 14:05:16.191970] E [socket.c:4406:socket_init] 0-gfapi: could not load CA list 2017-08-18T14:05:16.192477Z qemu-kvm: -drive file=gluster://10.70.36.79/vmstore/04396d72-6e34-4b00-b5e2-c440d33139be/images/4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a/78d8559f-b5c 9-4450-ac9f-785401832472,file.debug=4,format=raw,if=none,id=drive-scsi0-0-0-0,serial=4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a,cache=none,werror=stop,rerror=stop,aio=threads: Glu ster connection for volume vmstore, path 04396d72-6e34-4b00-b5e2-c440d33139be/images/4dd641a9-2a37-4d4d-9ef4-bb6daebbc38a/78d8559f-b5c9-4450-ac9f-785401832472 failed to conn ect hint: failed on host 10.70.36.79 and port 24007 Please refer to gluster logs for more info (code=1) (vm:1221) No errors are seen in glusterd.logs other than the below one: ============================================================= [2017-08-18 11:32:08.952117] E [socket.c:2631:socket_poller] 0-socket.management: socket_poller 10.70.36.79:993 failed (Input/output error)
sosreports are present in the link below: ================================================ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/1482987/
Could you please help extract the relevant /var/log/libvirt/qemu and gluster logs from the sosreport?
Hi Dan, I have extracted both and already pasted them in the bug description itself. Thanks kasturi
Sorry for having missed this quite prominent data.
Niels, is this issue at the gfapi layer?
Moving this to RHGS
permissions on the file /etc/ssl/glusterfs.ca file has to be changed so that qemu-kvm can read the file. currently the permissions on this file is set as below and due to that qemu-kvm cannot read these and vm fails to start. [root@dhcp37-55 ~]# ls -l /etc/ssl/glusterfs.ca -rw-------. 1 root root 1099 Aug 22 15:15 /etc/ssl/glusterfs.ca After changing the permissions as below, i was able to successfully start vm. chmod 644 /etc/ssl/glusterfs.ca
proposing this as a blocker since vm cannot be started using libgfapi when self signed certs are generated using gdeploy.
Verified and works fine with build gdeploy-2.0.2-15.el7rhgs.noarch. I see that permission on the file /etc/ssl/glusterfs.ca file has been changed as below. [root@rhsqa-grafton1 ~]# ls -l /etc/ssl/glusterfs.ca -rw-r--r--. 1 root root 3297 Sep 21 19:26 /etc/ssl/glusterfs.ca
Minor changes basis comments from Peer review.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3274