Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 871192

Summary: All virsh commands hang and have to be terminated manually.
Product: Red Hat Enterprise Linux 5 Reporter: Travis <Travis.Ross>
Component: libvirtAssignee: Martin Kletzander <mkletzan>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5.7CC: bsarathy, dyuan, jdenemar, mzhan, rwu, Travis.Ross, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-06 15:55:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd debug log
none
GDB of libvirtd --daemon PSID none

Description Travis 2012-10-29 21:11:24 UTC 
Description of problem:
After site power down KVM seems to be having issues.  One of the VM's in autostart came online but 5 others did not.  When attempting to work with the VM's using "virsh" the shell hangs.  I have attempted many commands and they all have the same result.  This is not just for list I have tried start a new VM and stop the existing VM all result in shell hang.

I have tried dumping debug information to a log but no data is put into the log.



Version-Release number of selected component (if applicable):

[root@XXX autostart]# rpm -qa|egrep "virt|kvm|kernel"
kernel-2.6.18-238.el5
virt-viewer-0.0.2-3.el5
kernel-2.6.18-274.3.1.el5
virt-manager-0.6.1-14.el5
kernel-headers-2.6.18-274.3.1.el5
kvm-83-249.el5_8
kernel-devel-2.6.18-274.3.1.el5
kmod-kvm-83-249.el5_8
libvirt-python-0.8.2-22.el5
python-virtinst-0.400.3-12.el5
libvirt-0.8.2-22.el5
libvirt-0.8.2-22.el5
kernel-xen-2.6.18-274.3.1.el5
kvm-qemu-img-83-249.el5_8
etherboot-zroms-kvm-5.4.4-15.el5
kvm-tools-83-249.el5_8


How reproducible:

On this server any boot comes to this issue

Steps to Reproduce:
1. Boot server
2. Run virsh
3.
  
Actual results:

Console hangs

Expected results:

list of VM's

Additional info:
Comment 1 Travis 2012-10-29 21:40:32 UTC 
Created attachment 635225 [details]
libvirtd debug log

Adding log based on following settings.

/etc/libvirt/libvirtd.conf and setting:

log_level=1
log_filters=""
log_outputs="1:file:/var/log/libvirtd.log"
Comment 2 Jiri Denemark 2012-10-30 11:05:34 UTC 
Looks like qemu-kvm is not responding for some reason. And since the old
libvirtd from RHEL-5 starts all autostarted domains synchronously before even
accepting any client connections, nothing works. Could you also attach a
debuger to libvirtd (gdb /usr/sbin/libvirtd `pidof libvirtd`) and attach the
output of "thread apply all backtrace" gdb command? Getting qemu logs for all
autostarted domains would help as well. You can find them in
/var/log/libvirt/qemu/NAME.log. In case you killed libvirtd since you gathered
the debug logs and need to reboot the host again to get the backtrace and qemu
logs, please, attach libvirtd.log again so that it matches the backtrace and
qemu logs.
Comment 3 Travis 2012-10-31 14:23:56 UTC 
Created attachment 636160 [details]
GDB of libvirtd --daemon PSID

There have been no additional reboots.  I was able to remove the non-required VM's from the autostart to keep the business moving while I was traveling.

I have attached the output from the gdb trace against the libvirtd --daemon process not sure that's it's going to provide any usefull information it seems to have had issues.

However, the virsh commands area actually now functioning as expected, so I might have been trying to work with it too soon after it's boot.
Comment 4 Martin Kletzander 2012-10-31 17:13:50 UTC 
(In reply to comment #3)
Thanks for your patience.  If I understand correctly you're saying these problems go away if you remove the particular machine from the autostart, right?

If you are able to reproduce that again, could you install debuginfo package (at least for libvirt) and try to attach to GDB when it's stuck as described By Jiri in comment #2?

Thanks
Comment 5 Martin Kletzander 2013-03-06 15:55:17 UTC 
Closing as INSUFFICIENT_DATA, feel free to reopen the bug if you encounter the problem again and are able to post additional information.  Thanks, Martin.