Bug 1124372

Whenever

   File "/usr/share/vdsm/supervdsm.py", line 50, in __call__

is seen in vdsm.log, it means that the juicy stuff is in supervdsm.log. Would you attach it, too?

Created attachment 922127 [details]
supervdsm log

I see that libvirtd has been stopped since supervdsmd was started:

    allNets = ((net, net.name()) for net in conn.listAllNetworks(0))
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 129, in wrapper
    __connections.get(id(target)).pingLibvirt()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3389, in getLibVersion
    if ret == -1: raise libvirtError ('virConnectGetLibVersion() failed', conn=self)
libvirtError: internal error client socket is closed

Could you attach the log of ovirt-host-deploy, to understand why?

since supervdsmd was started ? no.. supervdsmd restarted because libvirt connection got broken somehow, maybe because libvirtd was restarted.. can we share also libvirt log?

(MainProcess|Thread-16::WARNING::2014-07-29 12:36:13,005::libvirtconnection::135::root::(wrapper) connection to libvirt broken. ecode: 1 edom: 7
MainProcess|Thread-16::CRITICAL::2014-07-29 12:36:13,005::libvirtconnection::137::root::(wrapper) taking calling process down.
MainProcess|Thread-16::DEBUG::2014-07-29 12:36:13,007::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 1 edom: 7 level: 2 message: internal error client socket is closed
MainThread::DEBUG::2014-07-29 12:36:13,007::supervdsmServer::445::SuperVdsm.Server::(main) Terminated normally

^^ this leads to SIGTERM to the called, which is supervdsmd)

can it be related to the recent change http://gerrit.ovirt.org/30659 ? I think not, as in the description you mention only el6 packages..

Created attachment 922187 [details]
libvirt log

please add also the deploy log as dan requested and /var/log/messages for me 

thanks.

Created attachment 922863 [details]
messages

(In reply to Yaniv Bronhaim from comment #6)
> please add also the deploy log as dan requested and /var/log/messages for me 
> 
> thanks.

- I'm sorry, but there is no deploy log, at that point the host-deploy wasn't in the game yet...

libvirtd is crashing:

Jul 29 15:49:21 dev-27 init: libvirtd main process (2151) terminated with status 1
Jul 29 15:49:21 dev-27 init: libvirtd main process ended, respawning
Jul 29 15:49:21 dev-27 init: libvirtd main process (2159) terminated with status 1
Jul 29 15:49:21 dev-27 init: libvirtd main process ended, respawning
Jul 29 15:49:21 dev-27 init: libvirtd main process (2167) terminated with status 1
Jul 29 15:49:21 dev-27 init: libvirtd respawning too fast, stopped


2014-07-29 13:49:21.853+0000: 2159: error : virPidFileAcquirePath:410 : Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable


Can you start libvirt from the command line (as root)?

# libvirtd --listen

(In reply to Dan Kenigsberg from comment #9)
> libvirtd is crashing:
> 
> Jul 29 15:49:21 dev-27 init: libvirtd main process (2151) terminated with
> status 1
> Jul 29 15:49:21 dev-27 init: libvirtd main process ended, respawning
> Jul 29 15:49:21 dev-27 init: libvirtd main process (2159) terminated with
> status 1
> Jul 29 15:49:21 dev-27 init: libvirtd main process ended, respawning
> Jul 29 15:49:21 dev-27 init: libvirtd main process (2167) terminated with
> status 1
> Jul 29 15:49:21 dev-27 init: libvirtd respawning too fast, stopped
> 
> 
> 2014-07-29 13:49:21.853+0000: 2159: error : virPidFileAcquirePath:410 :
> Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily
> unavailable
> 
> 
> Can you start libvirt from the command line (as root)?
> 
> # libvirtd --listen

I've restarted the machine and it doesn't happen anymore. libvirtd and vdsm seems up'n'running.

Michal Privoznik explains here https://www.redhat.com/archives/libvirt-users/2012-October/msg00146.html that this condition can happen when there is a stale libvirtd process.

When this reproduces, please run `lsof /var/run/libvirtd.pid` so we can trace which is the other process that locks the pid file. Make sure to extract as much info about that process, in particular `ps -o stime,ppid` so we can guess when it was started and by whom.

Dan - anything we can do with that?

I follow that up, with dan observation and comments :)
Jiri, let me know if you successfully reproduced it and ran lsof on the pid file?

Jiri, can you reply ? with no reproducer we can't proceed

I can't reproduce it anymore with vdsm-4.16.2-1.gite8cba75.el6.x86_64. Let's close this bug to unblock the depending bz and I'll reopen it if I run into that problem again.