Bug 1634765

Hi,

Does it affect VM shutdown as well?

For example on VM shutdown i get:

2018-10-04 12:42:50,972+0300 ERROR (Thread-3) [root] Shutdown by QEMU Guest Agent failed (vm:5413)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5404, in qemuGuestAgentShutdown
    self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2515, in shutdownFlags
    if ret == -1: raise libvirtError ('virDomainShutdownFlags() failed', dom=self)
libvirtError: Guest agent is not responding: Guest agent not available for now

Asking if it's the same bug or a different one..Thanks,

(In reply to Michael Burman from comment #3)

> Does it affect VM shutdown as well?

It does, but in a slightly different way on my machine:

  libvirtError: Guest agent is not responding: QEMU guest agent is not connected

This message says that guest agent is not connected at all, which makes sense with the permission problem.

While your message

> libvirtError: Guest agent is not responding: Guest agent not available for
> now

indicates that the agent is generally available, but not responding at the given moment.

So I'd say it's a different bug.

(In reply to Milan Zamazal from comment #4)
> (In reply to Michael Burman from comment #3)
> 
> > Does it affect VM shutdown as well?
> 
> It does, but in a slightly different way on my machine:
> 
>   libvirtError: Guest agent is not responding: QEMU guest agent is not
> connected
> 
> This message says that guest agent is not connected at all, which makes
> sense with the permission problem.
> 
> While your message
> 
> > libvirtError: Guest agent is not responding: Guest agent not available for
> > now
> 
> indicates that the agent is generally available, but not responding at the
> given moment.
> 
> So I'd say it's a different bug.

Ok thanks Milan, i will try to reproduce and report fresh bug

Hi Milan,

Reported fresh bug for the issue  - BZ 1636723

Tell me please, does this error -


2018-10-07 11:19:54,278+0300 ERROR (Thread-1) [virt.vm] (vmId='7a161d40-a7cc-4832-8dc1-9f899b0f730f') desktopLock failed with unexpected exception (guestagent:496)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 491, in desktopLock
    self._forward("lock-screen")
  File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 333, in _forward
    self._sock.sendall(message)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 107] Transport endpoint is not connected

Related to one of this bugs? or it is another different issue?

(In reply to Milan Zamazal from comment #4)
> So I'd say it's a different bug.

Wwhy do you think it’s a different bug?

(In reply to Michael Burman from comment #6)
> Hi Milan,
> 
> Reported fresh bug for the issue  - BZ 1636723
> 
> Tell me please, does this error -
> 
> 
> 2018-10-07 11:19:54,278+0300 ERROR (Thread-1) [virt.vm]
> (vmId='7a161d40-a7cc-4832-8dc1-9f899b0f730f') desktopLock failed with
> unexpected exception (guestagent:496)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 491,
> in desktopLock
>     self._forward("lock-screen")
>   File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 333,
> in _forward
>     self._sock.sendall(message)
>   File "/usr/lib64/python2.7/socket.py", line 224, in meth
>     return getattr(self._sock,name)(*args)
> error: [Errno 107] Transport endpoint is not connected
> 
> Related to one of this bugs? or it is another different issue?

Hi Michael, it looks like related to this bug.

(In reply to Michal Skrivanek from comment #7)
> (In reply to Milan Zamazal from comment #4)
> > So I'd say it's a different bug.
> 
> Wwhy do you think it’s a different bug?

Because the error I receive on shutdown is generated in qemuDomainAgentAvailable libvirt function, while Michael's error originates in qemuAgentSend. A logical explanation would be that in the latter case libvirt was able to open agent connection before, which couldn't happen in case of this bug. Indeed, qemuDomainAgentAvailable is called before qemuAgentShutdown in qemuDomainShutdownFlags.

But I still can't completely exclude it's the same problem. If the libvirt + QEMU versions are the "wrong" ones then it's likely to be the same bug, although I'd wonder about the Michael's error message in such a case.

I don't think it's worth looking into it. just try to reproduce with the fixed libvirt and let's see

*** Bug 1636723 has been marked as a duplicate of this bug. ***

Hi Milan,

I managed to reproduce this error with libvirt-4.5.0-10.el7_6.2.x86_64 and
vdsm-4.20.43-4.gitb76af64.el7.x86_64

2018-11-12 11:14:00,188+0200 ERROR (Thread-7) [root] Shutdown by QEMU Guest Agent failed (vm:5409)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5400, in qemuGuestAgentShutdown
    self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2515, in shutdownFlags
    if ret == -1: raise libvirtError ('virDomainShutdownFlags() failed', dom=self)
libvirtError: Guest agent is not responding: QEMU guest agent is not connected
2018-11-12 11:14:00,369+0200 INFO  (jsonrpc/0) [api.virt] START shutdown(delay=u'30', message=u'System Administrator has initiated shutdown of this Virtual Machine. Virtua
l Machine is shutting down.'

Bottom line, I'm not sure that libvirt-4.5.0-10.el7_6.2.x86_64 is good enough as i still see this error on graceful VM shutdown with this libvirt version and VM failed to shutdown...)

Created attachment 1504641 [details]
bug reproduced with libvirt-4.5.0-10.el7_6.2.x86_64

bug reproduced with libvirt-4.5.0-10.el7_6.2.x86_64 and vdsm-4.20.43-4.gitb76af64.el7.x86_64

Hi Michael, I can't reproduce it. libvirt-4.5.0-10.el7_6.2.x86_64 is enough, but this looks like a different problem again. Could you please provide vdsm.log with DEBUG enabled not only for storage, but for all components, especially logger_virt? And could you please also double check that QEMU guest agent runs happily in the guest OS?

Created attachment 1504668 [details]
vdsm log in debug

After looking at it more closely, I can see this is a different bug, unrelated to the original problem. Unlike the original problem, where Vdsm was unable to connect to QEMU guest agent all, this one happens only from time to time (I couldn't reproduce it myself in the provided environment) and fails in a libvirt call. So it's an unrelated, most likely libvirt, bug.

Verified on vdsm-4.30.4-1.el7ev.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1077