Bug 852956 - 3.2 - prepareForShutdown is not called when connection to libvirt is broken with event: libvirtError: internal error client socket is closed
3.2 - prepareForShutdown is not called when connection to libvirt is broken w...
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
x86_64 Linux
medium Severity medium
: ---
: 3.2.0
Assigned To: Yaniv Bronhaim
: ZStream
: 956197 957335 (view as bug list)
Depends On:
Blocks: 915537 922117 928309 951576
  Show dependency treegraph
Reported: 2012-08-30 01:29 EDT by EricLee
Modified: 2015-09-22 09 EDT (History)
20 users (show)

See Also:
Fixed In Version: vdsm-4.10.2-16.0.el6ev
Doc Type: Bug Fix
Doc Text:
Domain codes and libvirt error codes were mixed by mistake, so restarting the libvirt daemon caused the libvirt client socket to close on Red Hat Enterprise Virtualization Manager. In addition, libvirt reported internal errors if libvirtd is restarted or stopped, for example after a crash. This update resolves the mixed codes and adds missing error codes. Restarting libvirtd now correctly restarts VDSM connections.
Story Points: ---
Clone Of:
: 951576 (view as bug list)
Last Closed: 2013-06-10 16:31:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
/var/log/vdsm/libvirt.log (409.19 KB, text/plain)
2012-08-30 01:29 EDT, EricLee
no flags Details
/var/log/vdsm/vdsm.log (2.48 MB, text/plain)
2012-08-30 01:30 EDT, EricLee
no flags Details
vdsm-lib-socket.log.tgz (44.61 KB, application/x-gzip)
2013-04-16 10:38 EDT, Pavel Stehlik
no flags Details
vdsm.log and libvirtd.log (822.33 KB, application/x-gzip)
2013-04-30 11:11 EDT, Elad
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 13990 None None None Never

  None (edit)
Description EricLee 2012-08-30 01:29:39 EDT
Created attachment 608047 [details]

Description of problem:
"initctl restart libvirtd" will cause libvirt client socket close in rhevm

Version-Release number of selected component (if applicable):
# rpm -qa libvirt kernel qemu-kvm-rhev vdsm spice-server

How reproducible:

Steps to Reproduce:
1.Prepare a rhevm enviroment, with libvirt + vdsm packages. 
2.#initctl restart libvirtd
3.Start a guest in rhevm,will get error:
libvirtError: internal error client socket is closed
and can not start guest normally.
4.#service vdsmd restart
will take libvirtd to normal environment.
5.Start a guest in rhevm normally, then redo "initctl restart libvirtd", and check /var/log/vdsm/libvirt.log and /var/log/vdsm/vdsm.log also can get that error.

Actual results:
As steps.

Expected results:
Libvirtd runs normally.

Additional info:
Comment 1 EricLee 2012-08-30 01:30:27 EDT
Created attachment 608048 [details]
Comment 3 EricLee 2012-08-30 01:33:39 EDT
BTW, pure libvirt environment, restart libvirtd using service will not cause that error,and working well.
Comment 4 Jiri Denemark 2012-08-30 04:52:53 EDT
It seems like vdsm is not properly catching closed connection to libvirtd. I'm moving this bug to vdsm for further investigation.
Comment 9 Michal Skrivanek 2012-10-01 04:49:16 EDT
seems like the code is obsolete, in current libvirt you can find a check for connection health:
+             (((last_error->code == VIR_ERR_SYSTEM_ERROR) &&
+               (last_error->domain == VIR_FROM_REMOTE)) ||
+              (last_error->code == VIR_ERR_RPC) ||
+              (last_error->code == VIR_ERR_NO_CONNECT) ||
+              (last_error->code == VIR_ERR_INVALID_CONN)))

so adding those last two would be needed in libvirtconnection.py wrapMethod()

Libvirt team says they have a callback for this in RHEL 6.4 so we may want to replace this in future/
Comment 10 Michal Skrivanek 2012-10-01 07:51:52 EDT
upstream review: http://gerrit.ovirt.org/#/c/8283/
Comment 12 Michal Skrivanek 2013-01-03 08:30:14 EST
merged u/s master: 0bfe41291e12fe583641b1844c753d51e15fded7
Comment 18 Pavel Stehlik 2013-04-16 10:38:17 EDT
Created attachment 736326 [details]

full vdsm log
Comment 22 Michal Skrivanek 2013-04-25 03:20:09 EDT
*** Bug 956197 has been marked as a duplicate of this bug. ***
Comment 24 Iulian 2013-04-29 13:51:03 EDT
*** Bug 957335 has been marked as a duplicate of this bug. ***
Comment 25 Elad 2013-04-30 11:11:40 EDT
Created attachment 741901 [details]
vdsm.log and libvirtd.log

I was manage to reproduce this bug.

It reproduced during verification of this https://bugzilla.redhat.com/show_bug.cgi?id=953645
we saw that prepareForShutdown is not called after vdsm looses connection to libvirt.

I did "kill -6 (libvirtd_pid)"

This is happening only when there are no running vms. otherwise, it does not reproduce. 

logs attached.

moving back to ASSIGN.
Comment 26 Dan Kenigsberg 2013-05-01 03:22:47 EDT
(In reply to comment #25)
> This is happening only when there are no running vms. otherwise, it does not
> reproduce. 

This awkward behavior is not new. It's here since getCaps started polling libvirt for netinfo. It should be be fixed, but I'm not sure at all that the suggested http://gerrit.ovirt.org/14345/ should make it into rhev-3.2, as it is not very probable that libvirt would spontaneously die when there are no VMs running.

Please open a new bug to track this behavior and let's solve it on rhev-3.3.
Comment 28 errata-xmlrpc 2013-06-10 16:31:40 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.