+++ This bug was initially created as a clone of Bug #1632759 +++ Description of problem: Running a VM with guest agent (and even guest tools) on a host with latest vdsm doesn't show any info reported by guest agent (network info might be reported but I think it's just cached data). This is also reproducible when migrating a VM from a host with older vdsm (all info reported correctly) to a host with latest vdsm (suddenly all info is gone). Version-Release number of selected component (if applicable): vdsm-4.20.40-1.el7ev.x86_64 How reproducible: always Steps to Reproduce: 1. run a VM with guest agent 2. 3. Actual results: no guest agent info is reported Expected results: guest agent info reported correctly Additional info: I have the env ready for investigation purposes or to provide additional logs.
Hi, Does it affect VM shutdown as well? For example on VM shutdown i get: 2018-10-04 12:42:50,972+0300 ERROR (Thread-3) [root] Shutdown by QEMU Guest Agent failed (vm:5413) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5404, in qemuGuestAgentShutdown self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2515, in shutdownFlags if ret == -1: raise libvirtError ('virDomainShutdownFlags() failed', dom=self) libvirtError: Guest agent is not responding: Guest agent not available for now Asking if it's the same bug or a different one..Thanks,
(In reply to Michael Burman from comment #3) > Does it affect VM shutdown as well? It does, but in a slightly different way on my machine: libvirtError: Guest agent is not responding: QEMU guest agent is not connected This message says that guest agent is not connected at all, which makes sense with the permission problem. While your message > libvirtError: Guest agent is not responding: Guest agent not available for > now indicates that the agent is generally available, but not responding at the given moment. So I'd say it's a different bug.
(In reply to Milan Zamazal from comment #4) > (In reply to Michael Burman from comment #3) > > > Does it affect VM shutdown as well? > > It does, but in a slightly different way on my machine: > > libvirtError: Guest agent is not responding: QEMU guest agent is not > connected > > This message says that guest agent is not connected at all, which makes > sense with the permission problem. > > While your message > > > libvirtError: Guest agent is not responding: Guest agent not available for > > now > > indicates that the agent is generally available, but not responding at the > given moment. > > So I'd say it's a different bug. Ok thanks Milan, i will try to reproduce and report fresh bug
Hi Milan, Reported fresh bug for the issue - BZ 1636723 Tell me please, does this error - 2018-10-07 11:19:54,278+0300 ERROR (Thread-1) [virt.vm] (vmId='7a161d40-a7cc-4832-8dc1-9f899b0f730f') desktopLock failed with unexpected exception (guestagent:496) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 491, in desktopLock self._forward("lock-screen") File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 333, in _forward self._sock.sendall(message) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 107] Transport endpoint is not connected Related to one of this bugs? or it is another different issue?
(In reply to Milan Zamazal from comment #4) > So I'd say it's a different bug. Wwhy do you think it’s a different bug?
(In reply to Michael Burman from comment #6) > Hi Milan, > > Reported fresh bug for the issue - BZ 1636723 > > Tell me please, does this error - > > > 2018-10-07 11:19:54,278+0300 ERROR (Thread-1) [virt.vm] > (vmId='7a161d40-a7cc-4832-8dc1-9f899b0f730f') desktopLock failed with > unexpected exception (guestagent:496) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 491, > in desktopLock > self._forward("lock-screen") > File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 333, > in _forward > self._sock.sendall(message) > File "/usr/lib64/python2.7/socket.py", line 224, in meth > return getattr(self._sock,name)(*args) > error: [Errno 107] Transport endpoint is not connected > > Related to one of this bugs? or it is another different issue? Hi Michael, it looks like related to this bug.
(In reply to Michal Skrivanek from comment #7) > (In reply to Milan Zamazal from comment #4) > > So I'd say it's a different bug. > > Wwhy do you think it’s a different bug? Because the error I receive on shutdown is generated in qemuDomainAgentAvailable libvirt function, while Michael's error originates in qemuAgentSend. A logical explanation would be that in the latter case libvirt was able to open agent connection before, which couldn't happen in case of this bug. Indeed, qemuDomainAgentAvailable is called before qemuAgentShutdown in qemuDomainShutdownFlags. But I still can't completely exclude it's the same problem. If the libvirt + QEMU versions are the "wrong" ones then it's likely to be the same bug, although I'd wonder about the Michael's error message in such a case.
I don't think it's worth looking into it. just try to reproduce with the fixed libvirt and let's see
*** Bug 1636723 has been marked as a duplicate of this bug. ***
Hi Milan, I managed to reproduce this error with libvirt-4.5.0-10.el7_6.2.x86_64 and vdsm-4.20.43-4.gitb76af64.el7.x86_64 2018-11-12 11:14:00,188+0200 ERROR (Thread-7) [root] Shutdown by QEMU Guest Agent failed (vm:5409) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5400, in qemuGuestAgentShutdown self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2515, in shutdownFlags if ret == -1: raise libvirtError ('virDomainShutdownFlags() failed', dom=self) libvirtError: Guest agent is not responding: QEMU guest agent is not connected 2018-11-12 11:14:00,369+0200 INFO (jsonrpc/0) [api.virt] START shutdown(delay=u'30', message=u'System Administrator has initiated shutdown of this Virtual Machine. Virtua l Machine is shutting down.' Bottom line, I'm not sure that libvirt-4.5.0-10.el7_6.2.x86_64 is good enough as i still see this error on graceful VM shutdown with this libvirt version and VM failed to shutdown...)
Created attachment 1504641 [details] bug reproduced with libvirt-4.5.0-10.el7_6.2.x86_64 bug reproduced with libvirt-4.5.0-10.el7_6.2.x86_64 and vdsm-4.20.43-4.gitb76af64.el7.x86_64
Hi Michael, I can't reproduce it. libvirt-4.5.0-10.el7_6.2.x86_64 is enough, but this looks like a different problem again. Could you please provide vdsm.log with DEBUG enabled not only for storage, but for all components, especially logger_virt? And could you please also double check that QEMU guest agent runs happily in the guest OS?
Created attachment 1504668 [details] vdsm log in debug
After looking at it more closely, I can see this is a different bug, unrelated to the original problem. Unlike the original problem, where Vdsm was unable to connect to QEMU guest agent all, this one happens only from time to time (I couldn't reproduce it myself in the provided environment) and fails in a libvirt call. So it's an unrelated, most likely libvirt, bug.
Verified on vdsm-4.30.4-1.el7ev.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1077