Created attachment 608047 [details] /var/log/vdsm/libvirt.log Description of problem: "initctl restart libvirtd" will cause libvirt client socket close in rhevm Version-Release number of selected component (if applicable): # rpm -qa libvirt kernel qemu-kvm-rhev vdsm spice-server spice-server-0.10.1-10.el6.x86_64 libvirt-0.10.0-1.el6.x86_64 vdsm-4.9.6-31.0.el6_3.x86_64 qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64 kernel-2.6.32-298.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1.Prepare a rhevm enviroment, with libvirt + vdsm packages. 2.#initctl restart libvirtd 3.Start a guest in rhevm,will get error: libvirtError: internal error client socket is closed and can not start guest normally. 4.#service vdsmd restart will take libvirtd to normal environment. 5.Start a guest in rhevm normally, then redo "initctl restart libvirtd", and check /var/log/vdsm/libvirt.log and /var/log/vdsm/vdsm.log also can get that error. Actual results: As steps. Expected results: Libvirtd runs normally. Additional info:
Created attachment 608048 [details] /var/log/vdsm/vdsm.log
BTW, pure libvirt environment, restart libvirtd using service will not cause that error,and working well.
It seems like vdsm is not properly catching closed connection to libvirtd. I'm moving this bug to vdsm for further investigation.
seems like the code is obsolete, in current libvirt you can find a check for connection health: + (((last_error->code == VIR_ERR_SYSTEM_ERROR) && + (last_error->domain == VIR_FROM_REMOTE)) || + (last_error->code == VIR_ERR_RPC) || + (last_error->code == VIR_ERR_NO_CONNECT) || + (last_error->code == VIR_ERR_INVALID_CONN))) so adding those last two would be needed in libvirtconnection.py wrapMethod() Libvirt team says they have a callback for this in RHEL 6.4 so we may want to replace this in future/
upstream review: http://gerrit.ovirt.org/#/c/8283/
merged u/s master: 0bfe41291e12fe583641b1844c753d51e15fded7
Created attachment 736326 [details] vdsm-lib-socket.log.tgz full vdsm log
*** Bug 956197 has been marked as a duplicate of this bug. ***
*** Bug 957335 has been marked as a duplicate of this bug. ***
Created attachment 741901 [details] vdsm.log and libvirtd.log I was manage to reproduce this bug. It reproduced during verification of this https://bugzilla.redhat.com/show_bug.cgi?id=953645 we saw that prepareForShutdown is not called after vdsm looses connection to libvirt. I did "kill -6 (libvirtd_pid)" This is happening only when there are no running vms. otherwise, it does not reproduce. logs attached. moving back to ASSIGN.
(In reply to comment #25) > > This is happening only when there are no running vms. otherwise, it does not > reproduce. This awkward behavior is not new. It's here since getCaps started polling libvirt for netinfo. It should be be fixed, but I'm not sure at all that the suggested http://gerrit.ovirt.org/14345/ should make it into rhev-3.2, as it is not very probable that libvirt would spontaneously die when there are no VMs running. Please open a new bug to track this behavior and let's solve it on rhev-3.3.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0886.html