Hide Forgot
Description of problem: Install libvirt with vdsm, restart libvirtd & vdsmd service over 2 times, it will cause libvirtd failed. Version-Release number of selected component (if applicable): libvirt-0.9.4-0rc2.el6.x86_64 vdsm-4.9-86.el6.x86_64 How reproducible: always Steps to Reproduce: [root@amd-1352-8-4 ~]# service libvirtd restart Stopping libvirtd daemon: [FAILED] Starting libvirtd daemon: [ OK ] [root@amd-1352-8-4 ~]# service vdsmd restart Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm: not running [FAILED] Stopping libvirtd daemon: [ OK ] Configuring libvirt for vdsm... Starting iscsid... Starting multipathd... Starting multipathd daemon: [ OK ] Starting iscsid: [ OK ] Starting up vdsm daemon: vdsm start [root@amd-1352-8-4 ~]# rpm -q libvirt libvirt-0.9.4-0rc2.el6.x86_64 [root@amd-1352-8-4 ~]# rpm -q vdsm vdsm-4.9-86.el6.x86_64 [root@amd-1352-8-4 ~]# service vdsmd status VDS daemon server is running [root@amd-1352-8-4 ~]# service libvirtd status libvirtd (pid 2757) is running... [root@amd-1352-8-4 ~]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: libvirtd: error: Unable to obtain pidfile. Check /var/log/messages or run without --daemon for more info. [FAILED] [root@amd-1352-8-4 ~]# service libvirtd status libvirtd (pid 3269) is running... [root@amd-1352-8-4 ~]# ls /var/run/libvirtd.pid /var/run/libvirtd.pid [root@amd-1352-8-4 ~]# cat /var/run/libvirtd.pid 3269 [root@amd-1352-8-4 ~]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info. [FAILED] [root@amd-1352-8-4 ~]# cat /var/run/libvirtd.pid cat: /var/run/libvirtd.pid: No such file or directory Actual results: libvirtd fail to restart. Expected results: libvirtd daemon restart sucessfully. Additional info:
Run with the following commands will increase the odds of reproducing. # service libvirtd restart && service vdsmd restart
Once encountered this problem, to do the below steps can resolve this issue. 1. Remove all the configurations which added by vdsm in the following 2 libvirtd config files. /etc/libvirt/libvirtd.conf -------------------------- listen_addr="0" # by vdsm unix_sock_group="kvm" # by vdsm unix_sock_rw_perms="0770" # by vdsm auth_unix_rw="sasl" # by vdsm save_image_format="lzop" # by vdsm log_outputs="1:file:/var/log/libvirtd.log" # by vdsm log_filters="1:libvirt 3:event 3:json 1:util 1:qemu" # by vdsm auth_tcp="none" # by vdsm listen_tcp=1 # by vdsm listen_tls=0 # by vdsm /etc/sysconfig/libvirtd -------------------------- LIBVIRTD_ARGS=--listen # by vdsm DAEMON_COREFILE_LIMIT=unlimited # by vdsm 2. Restart libvirtd service # service libvirtd restart 3. Restart vdsmd service # service vdsmd restart
Dan, is this addressed by the patch series you posted today (https://www.redhat.com/archives/libvir-list/2011-August/msg00349.html)?
That wasn't my intention with those patches, but it might be a happy accident.
I think the bug comes from libvirt daemon/libvirtd.upstart script which happens to do: post-stop script rm -f $PIDFILE rm -rf /var/cache/libvirt/* end script it removes the PIDFILE without checking if the daemon is still running. I think the vdsmd service is activating that upstart script and we end up with case where upstart think the daemon is dead (or just not 'responding' quickly enough) and it restarts it We then end up with a second daemon being started while another one is running, this leads to massive confusion, errors, etc ... IMHO the right thing is to fix that script to not remove the pidfile if the given process still exist and is a libvirt daemon. Daniel
This bug arises from using SysV Init script together with upstart script. VDSM starts libvirtd using upstart, as it requires automatic respawning of libvirtd in case of a crash.If the user thereafter uses the SysV init script, it removes the pid file and kills the daemon. This triggers upstart to respawn the daemon as upstart thinks it crashed. This results in a pretty much undefined state. Correct procedure for the user is to use initctl restart libvirtd to achieve the desired operation and not to trigger the bug. This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=728153 *** This bug has been marked as a duplicate of bug 728153 ***