Red Hat Bugzilla – Bug 852956
3.2 - prepareForShutdown is not called when connection to libvirt is broken with event: libvirtError: internal error client socket is closed
Last modified: 2015-09-22 09:09 EDT
Created attachment 608047 [details]
Description of problem:
"initctl restart libvirtd" will cause libvirt client socket close in rhevm
Version-Release number of selected component (if applicable):
# rpm -qa libvirt kernel qemu-kvm-rhev vdsm spice-server
Steps to Reproduce:
1.Prepare a rhevm enviroment, with libvirt + vdsm packages.
2.#initctl restart libvirtd
3.Start a guest in rhevm,will get error:
libvirtError: internal error client socket is closed
and can not start guest normally.
4.#service vdsmd restart
will take libvirtd to normal environment.
5.Start a guest in rhevm normally, then redo "initctl restart libvirtd", and check /var/log/vdsm/libvirt.log and /var/log/vdsm/vdsm.log also can get that error.
Libvirtd runs normally.
Created attachment 608048 [details]
BTW, pure libvirt environment, restart libvirtd using service will not cause that error,and working well.
It seems like vdsm is not properly catching closed connection to libvirtd. I'm moving this bug to vdsm for further investigation.
seems like the code is obsolete, in current libvirt you can find a check for connection health:
+ (((last_error->code == VIR_ERR_SYSTEM_ERROR) &&
+ (last_error->domain == VIR_FROM_REMOTE)) ||
+ (last_error->code == VIR_ERR_RPC) ||
+ (last_error->code == VIR_ERR_NO_CONNECT) ||
+ (last_error->code == VIR_ERR_INVALID_CONN)))
so adding those last two would be needed in libvirtconnection.py wrapMethod()
Libvirt team says they have a callback for this in RHEL 6.4 so we may want to replace this in future/
upstream review: http://gerrit.ovirt.org/#/c/8283/
merged u/s master: 0bfe41291e12fe583641b1844c753d51e15fded7
Created attachment 736326 [details]
full vdsm log
*** Bug 956197 has been marked as a duplicate of this bug. ***
*** Bug 957335 has been marked as a duplicate of this bug. ***
Created attachment 741901 [details]
vdsm.log and libvirtd.log
I was manage to reproduce this bug.
It reproduced during verification of this https://bugzilla.redhat.com/show_bug.cgi?id=953645
we saw that prepareForShutdown is not called after vdsm looses connection to libvirt.
I did "kill -6 (libvirtd_pid)"
This is happening only when there are no running vms. otherwise, it does not reproduce.
moving back to ASSIGN.
(In reply to comment #25)
> This is happening only when there are no running vms. otherwise, it does not
This awkward behavior is not new. It's here since getCaps started polling libvirt for netinfo. It should be be fixed, but I'm not sure at all that the suggested http://gerrit.ovirt.org/14345/ should make it into rhev-3.2, as it is not very probable that libvirt would spontaneously die when there are no VMs running.
Please open a new bug to track this behavior and let's solve it on rhev-3.3.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.