Red Hat Bugzilla – Bug 866510
vdsm: vdsm is stopped when libvirt is down and it takes 15 for the watchdog to start it again once libvirt is started
Last modified: 2014-01-12 19:54:45 EST
Created attachment 627459 [details] logs Description of problem: I stopped libvirt and waited for vdsm to restart. once rhevm turned the host to non-responsive I started libvirt. although the vdsm watchdog is up vdsm is not started again for 15 minutes. Version-Release number of selected component (if applicable): vdsm-cli-4.9.6-37.0.el6_3.noarch libvirt-0.9.10-21.el6_3.5.x86_64 How reproducible: 100% Steps to Reproduce: 1. stop libvirt (initctl stop libvirtd) 2. wait for the host to become non-responsive 3. start libvirt Actual results: although vdsm watchdog is up vdsm is not started Expected results: watchdog should start vdsm once libvirt is started. Additional info: full logs Thread-16::ERROR::2012-10-15 15:50:14,351::libvirtconnection::92::vds::(wrapper) connection to libvirt broken. taking vdsm down. MainThread::ERROR::2012-10-15 15:50:19,271::vdsm::73::vds::(run) Exception raised Traceback (most recent call last): File "/usr/share/vdsm/vdsm", line 71, in run serve_clients(log) File "/usr/share/vdsm/vdsm", line 39, in serve_clients cif = clientIF.clientIF(log) File "/usr/share/vdsm/clientIF.py", line 70, in __init__ self._libvirt = libvirtconnection.get() File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 114, in get conn = libvirt.openAuth('qemu:///system', auth, 0) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth if ret is None:raise libvirtError('virConnectOpenAuth() failed') libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory MainThread::INFO::2012-10-15 15:50:19,273::vdsm::75::vds::(run) VDSM main thread ended. Waiting for 1 other threads... MainThread::INFO::2012-10-15 15:50:19,273::vdsm::78::vds::(run) <_MainThread(MainThread, started 140659584595712)> MainThread::INFO::2012-10-15 15:50:19,274::vdsm::78::vds::(run) <Thread(libvirtEventLoop, started daemon 140659512071936)> MainThread::INFO::2012-10-15 15:50:19,347::vdsm::70::vds::(run) I am the actual vdsm 4.9-37.0 vdsm is tarted 15 minutes later: MainThread::INFO::2012-10-15 16:05:50,068::vdsm::70::vds::(run) I am the actual vdsm 4.9-37.0 MainThread::DEBUG::2012-10-15 16:05:50,400::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'Storage' MainThread::DEBUG::2012-10-15 16:05:50,401::threadPool::45::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 MainThread::DEBUG::2012-10-15 16:05:50,413::sp::362::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/402471a6-154f-43a3-b9fe-5308024c33a6/master` is not mounted, skipping MainThread::DEBUG::2012-10-15 16:05:50,415::sp::362::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/76f6a2e6-e2a3-431c-a8ef-f5d58aa00629/master` is not mounted, skipping MainThread::DEBUG::2012-10-15 16:05:50,418::sp::362::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/dbed4cf3-a177-49a8-b41c-f12b85db4286/master` is not mounted, skipping MainThread::DEBUG::2012-10-15 16:05:50,420::sp::362::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/ac6943fb-d267-45c3-a5af-128a6e761a2e/master` is not mounted, skipping MainThread::DEBUG::2012-10-15 16:05:50,562::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/cat /etc/multipath.conf' (cwd None) MainThread::DEBUG::2012-10-15 16:05:50,597::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
The above is the intentional behaviour of VDSM. The code resides in the respawn script (which acts as the watchdog): there are 3 parameters defined in that script: MINLIFETIME=2 MAX_THRASH_INTERVAL=30 POST_FAIL_INTERVAL=900 This means that in order to prevent a vdsm restart storm, we identify a quick failure of vdsm (uptime < MINLIFETIME) and try to do a fast restart (immediate restart) for the next MAX_THRASH_INTERVAL (= 30 secs), if by than VDSM still fails to come up, we wait for POST_FAIL_INTERVAL (= 15 minutes) and than try to bring vdsm up again. This is exactly the behaviour described above. hence moving this BZ to CLOSED NOTABUG