Bug 956197
| Summary: | VDSM produces huge amount of error messages after libvirt upgrade, but VMs continue to run | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Petr Spacek <pspacek> |
| Component: | vdsm | Assignee: | Barak <bazulay> |
| Status: | CLOSED DUPLICATE | QA Contact: | Elad <ebenahar> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | unspecified | CC: | bazulay, hateya, iheim, lpeer, michal.skrivanek, ykaul |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | infra | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-04-25 07:20:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Infra | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Yaniv, I think we already fixed it, the issue is after libvirt gets updated, service is restarted, and vdsm can't connect to socket and doesn't take itself down, please confirm its indeed the issue and set as duplicate. indeed *** This bug has been marked as a duplicate of bug 852956 *** |
Description of problem: I upgraded two RHEL 6.4 machines to latest libvirt packages. After some time I found that VDSM is spamming /var/log/messages with error messages (see below). The error message appeared approximatelly 10 times per seconds, i.e. it consumed log space at speed 10 kiB/s. I noticed this problem after 4 hours and it consumed significant amount of space in meanwhile. The problem appeared few seconds after upgrade: Apr 24 08:56:15 laika yum[4248]: Updated: libvirt-client-0.10.2-18.el6_4.4.x86_64 Apr 24 08:56:15 laika yum[4248]: Updated: 2:qemu-img-rhev-0.12.1.2-2.355.el6_4.3.x86_64 Apr 24 08:56:15 laika yum[4248]: Updated: libuuid-2.17.2-12.9.el6_4.2.x86_64 Apr 24 08:56:15 laika yum[4248]: Updated: libblkid-2.17.2-12.9.el6_4.2.x86_64 Apr 24 08:56:17 laika yum[4248]: Updated: util-linux-ng-2.17.2-12.9.el6_4.2.x86_64 Apr 24 08:56:19 laika vdsm vm.Vm ERROR vmId=`23f276b1-d9ac-4291-b49e-d89ff11c5e64`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x2c60ab0>#012Traceback (most recent call last):#012 File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect#012 statsFunction()#012 File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet#012 netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)#012 File "/usr/share/vdsm/libvirtvm.py", line 515, in f#012 ret = attr(*args, **kwargs)#012 File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper#012 ret = f(*args, **kwargs)#012 File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats#012 if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)#012libvirtError: internal error client socket is closed "service vdsmd restart" and "service libvirtd restart" solved the problem. AFAIK all VMs on the host were running fine, the problem manifested itself only in the log. Version-Release number of selected component (if applicable): 4.10.2-1.9.el6ev.x86_64 How reproducible: 100 % (i.e. it happened on both hosts) Steps to Reproduce: 1. Do yum upgdade between following package versions on a host where some VMs are running under VDSM: libvirt-0.10.2-18.el6_4.3.x86_64 -> libvirt-0.10.2-18.el6_4.4.x86_64 libvirt-client-0.10.2-18.el6_4.3.x86_64 -> libvirt-client-0.10.2-18.el6_4.4.x86_64 libvirt-lock-sanlock-0.10.2-18.el6_4.3.x86_64 -> libvirt-lock-sanlock-0.10.2-18.el6_4.4.x86_64 libvirt-python-0.10.2-18.el6_4.3.x86_64 -> libvirt-python-0.10.2-18.el6_4.4.x86_64 qemu-kvm-rhev-2:0.12.1.2-2.355.el6_4.2.x86_64 -> qemu-kvm-rhev-2:0.12.1.2-2.355.el6_4.3.x86_64 Actual results: Huge amount of errors from VDSM is logged into /var/log/messages. Expected results: Upgrade is handled in some nice way. Additional info: The complete YUM transaction was: Packages Altered: Erase kernel-2.6.32-279.19.1.el6.x86_64 @rhel-x86_64-server-6 Install kernel-2.6.32-358.6.1.el6.x86_64 @rhel-x86_64-server-6 Updated kernel-firmware-2.6.32-358.2.1.el6.noarch @rhel-x86_64-server-6 Update 2.6.32-358.6.1.el6.noarch @rhel-x86_64-server-6 Updated krb5-libs-1.10.3-10.el6_4.1.x86_64 @rhel-x86_64-server-6 Update 1.10.3-10.el6_4.2.x86_64 @rhel-x86_64-server-6 Updated krb5-workstation-1.10.3-10.el6_4.1.x86_64 @rhel-x86_64-server-6 Update 1.10.3-10.el6_4.2.x86_64 @rhel-x86_64-server-6 Updated libblkid-2.17.2-12.9.el6.x86_64 @rhel-x86_64-server-6 Update 2.17.2-12.9.el6_4.2.x86_64 @rhel-x86_64-server-6 Updated libuuid-2.17.2-12.9.el6.x86_64 @rhel-x86_64-server-6 Update 2.17.2-12.9.el6_4.2.x86_64 @rhel-x86_64-server-6 Updated libvirt-0.10.2-18.el6_4.3.x86_64 @rhel-x86_64-server-6 Update 0.10.2-18.el6_4.4.x86_64 @rhel-x86_64-server-6 Updated libvirt-client-0.10.2-18.el6_4.3.x86_64 @rhel-x86_64-server-6 Update 0.10.2-18.el6_4.4.x86_64 @rhel-x86_64-server-6 Updated libvirt-lock-sanlock-0.10.2-18.el6_4.3.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Update 0.10.2-18.el6_4.4.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Updated libvirt-python-0.10.2-18.el6_4.3.x86_64 @rhel-x86_64-server-6 Update 0.10.2-18.el6_4.4.x86_64 @rhel-x86_64-server-6 Updated net-snmp-libs-1:5.5-44.el6.x86_64 @rhel-x86_64-server-6 Update 1:5.5-44.el6_4.1.x86_64 @rhel-x86_64-server-6 Updated net-snmp-utils-1:5.5-44.el6.x86_64 @rhel-x86_64-server-6 Update 1:5.5-44.el6_4.1.x86_64 @rhel-x86_64-server-6 Updated perf-2.6.32-358.2.1.el6.x86_64 @rhel-x86_64-server-6 Update 2.6.32-358.6.1.el6.x86_64 @rhel-x86_64-server-6 Updated qemu-img-rhev-2:0.12.1.2-2.355.el6_4.2.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Update 2:0.12.1.2-2.355.el6_4.3.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Updated qemu-kvm-rhev-2:0.12.1.2-2.355.el6_4.2.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Update 2:0.12.1.2-2.355.el6_4.3.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Updated qemu-kvm-rhev-tools-2:0.12.1.2-2.355.el6_4.2.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Update 2:0.12.1.2-2.355.el6_4.3.x86_64 @rhel-x86_64-rhev-mgmt-agent-6 Updated util-linux-ng-2.17.2-12.9.el6.x86_64 @rhel-x86_64-server-6 Update 2.17.2-12.9.el6_4.2.x86_64 @rhel-x86_64-server-6