Bug 949192
| Summary: | [vdsm] [scale] After libvirt failure vdsm restarts and starts responding to XML-RPC after a big delay | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | vvyazmin <vvyazmin> | ||||
| Component: | vdsm | Assignee: | Yaniv Bronhaim <ybronhei> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Elad <ebenahar> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.2.0 | CC: | abaron, alonbl, bazulay, dallan, danken, hateya, iheim, italkohe, jentrena, lpeer, sgrinber, ykaul, zdover | ||||
| Target Milestone: | --- | Keywords: | ZStream | ||||
| Target Release: | 3.2.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | infra | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Previously, when a connectivity failure was raised by libvirt, VDSM began self-fencing. When VDSM restarted, it restarted the libvirt service. In large environments with high host loads, establishing the connection between VDSM and libvirt took quite a long time. Because the host doesn't respond until the connection to libvirt is back, this meant that host fencing would begin before the connection between VDSM and libvirt was established.
An upgrade allows VDSM to respond to API calls and report its status, which prevents the condition that previously caused premature and unwanted host fencing.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 953645 (view as bug list) | Environment: | |||||
| Last Closed: | 2013-06-10 20:48:06 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Infra | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 948216 | ||||||
| Bug Blocks: | 953645 | ||||||
| Attachments: |
|
||||||
|
Description
vvyazmin@redhat.com
2013-04-06 23:42:23 UTC
1. Currently we restart vdsm only if libvirt throws VIR_ERR_SYSTEM_ERROR, This might changed in libvirt implementation. When we try to reproduce it now vdsm receives from libvirt VIR_ERR_INTERNAL_ERROR. I consider using virConnectRegisterCloseCallback or catching also INTERNAL_ERROR. 2. When the connection is broken we perform prepareForShutdown that takes time when large amount of vms are running on host (Bug 924801). We consider better flow to make it quicker. The steps to reproduce can be much easier.. just "kill -s SIGABRT [libvirt pid]" or "rm /var/run/libvirt/libvirt-sock" In both cases vdsm should perform self fencing. libvirt has registration method for callback that signaled when the connection with libvirt is closed (registerCloseCallback - http://www.libvirt.org/html/libvirt-libvirt.html#virConnectRegisterCloseCallback) We should use this callback to distinguish connectivity errors. This callback is not available in libvirt < 1.0.1 libvirt for rhel6.4 doesn't contain it. First it should backported to rhel6.4.z and then merging vdsm patch that uses it. Dave, Can registerCloseCallback be back-ported to 6.4.z ? The above will also solve the SIGABRT not cought be VDSM. (In reply to comment #3) > Dave, > > Can registerCloseCallback be back-ported to 6.4.z ? > The above will also solve the SIGABRT not cought be VDSM. No, unfortunately that's a new API call and cannot be backported. (In reply to comment #4) > (In reply to comment #3) > > Dave, > > > > Can registerCloseCallback be back-ported to 6.4.z ? > > The above will also solve the SIGABRT not cought be VDSM. > > No, unfortunately that's a new API call and cannot be backported. Because of compatibility? because the changes are just too bug? or just because we do not do features in a Z release? *** Bug 948216 has been marked as a duplicate of this bug. *** When connectivity failure is raised by libvirt, vdsm starts self fencing. When vdsm restarts, it also restarts libvirt service. On large scale and high load on host, the connection to libvirt takes more than a minute. Unlike described in the bug description, the issue is that the host doesn't response until the connection to libvirt is back. This leads to host fencing (Bug 924801). To avoid that we move the first connectivity to libvirt to be in external thread. this way vdsm will be able to response to API calls and report its status. Verified on RHEVM - 3.2 - SF14 vdsm-4.10.2-16.0.el6ev.x86_64 libvirt-0.10.2-18.el6_4.4.x86_64 rhevm-3.2.0-10.20.master.el6ev.noarch vdsm initiate prepareForShutdown immediately after connection to libvirt breaks. 15:11:11,036:: libvirtconnection::123::vds::(wrapper) connection to libvirt broken. taking vdsm down. 15:11:12,036:: logUtils::40::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0886.html |