Bug 807901 - fail to start the guest since cannot acquire state change lock
fail to start the guest since cannot acquire state change lock
Status: CLOSED DUPLICATE of bug 786648
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.2
x86_64 Linux
unspecified Severity low
: rc
: ---
Assigned To: Jiri Denemark
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-29 01:36 EDT by Xiaowei Li
Modified: 2015-01-26 19:10 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-04-10 02:52:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
xml description of the guest (2.54 KB, text/xml)
2012-04-05 23:14 EDT, Xiaowei Li
no flags Details
libvirtd backtrace log (5.68 KB, text/plain)
2012-04-08 23:39 EDT, Wayne Sun
no flags Details
domain qemu log (38.40 KB, text/plain)
2012-04-08 23:41 EDT, Wayne Sun
no flags Details

  None (edit)
Description Xiaowei Li 2012-03-29 01:36:05 EDT
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:
sometimes

Steps to Reproduce:
1. no sure the previous steps
2. start the guest
  
Actual results:

The following error message was encountered.

Error starting domain: Timed out during operation: cannot acquire state change lock

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 44, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1050, in startup
    self._backend.create()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 511, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Timed out during operation: cannot acquire state change lock



Expected results:


Additional info:
Comment 4 Dave Allan 2012-03-30 10:13:52 EDT
Is this the same system that was used for BZ 807996 ?
Comment 5 weizhang 2012-03-31 23:24:09 EDT
Hi Xiaowei,

Could you please attach logs in /var/log/libvirt/libvirtd.log and the history of your executed commands?
Comment 6 Xiaowei Li 2012-04-01 00:10:26 EDT
(In reply to comment #5)
> Hi Xiaowei,
> 
> Could you please attach logs in /var/log/libvirt/libvirtd.log and the history
> of your executed commands?

attaching the logs.
Not sure if something is changed since i encountered this issue on my laptop and done many operation for the libvirt.
for the history command, i cannot provide it since i used virt-manager usually.
Comment 8 Jiri Denemark 2012-04-03 08:04:17 EDT
Interesting, could you please provide the following data so that we can investigate the issue?

- domain XML of the guest you were trying to start (virsh dumpxml NAME)
- libvirtd debug logs; set the following in /etc/libvirt/libvirtd.conf and
  restart libvirtd service:

    log_filters="1:libvirt 1:conf 1:qemu 3:util/json 1:util"
    log_outputs="1:file:/var/log/libvirt/libvirtd.log"
Comment 9 Xiaowei Li 2012-04-04 22:44:48 EDT
(In reply to comment #4)
> Is this the same system that was used for BZ 807996 ?

not the same system. It's my laptop.
Comment 10 Xiaowei Li 2012-04-04 22:57:19 EDT
(In reply to comment #8)
> Interesting, could you please provide the following data so that we can
> investigate the issue?
> 
> - domain XML of the guest you were trying to start (virsh dumpxml NAME)
> - libvirtd debug logs; set the following in /etc/libvirt/libvirtd.conf and
>   restart libvirtd service:
> 
>     log_filters="1:libvirt 1:conf 1:qemu 3:util/json 1:util"
>     log_outputs="1:file:/var/log/libvirt/libvirtd.log"

delete the guests after server reboot so cannot get the useful information on my current environment.
anyway, i will try to reproduce this issue after set the libvirtd debug logs.
Comment 11 Xiaowei Li 2012-04-05 23:11:44 EDT
reproduced it on my laptop again.
# virsh start el6
error: Domain is already active
# virsh shutdown el6
error: Failed to shutdown domain el6
error: Timed out during operation: cannot acquire state change lock

attaching the libvirtd.log and guest xml file.
Comment 12 Xiaowei Li 2012-04-05 23:14:16 EDT
Created attachment 575600 [details]
xml description of the guest
Comment 13 Xiaowei Li 2012-04-05 23:40:29 EDT
for the libvirtd.log, cannot see anything after start & stop the guest.

[root@troy libvirt]# virsh start el6
error: Domain is already active

[root@troy libvirt]# ls -lh
total 730M
-rw-------. 1 root root    0 Apr  6 11:37 libvirtd.log
-rw-------. 1 root root 356M Apr  6 03:10 libvirtd.log-20120406
-rw-------. 1 root root 375M Apr  6 11:44 libvirtd.log-20120406-02
drwx------. 2 root root 4.0K Feb 22 10:24 lxc
drwx------. 2 root root 4.0K Apr  5 10:48 qemu
drwx------. 2 root root 4.0K Feb 22 10:24 uml
[root@troy libvirt]# virsh shutdown el6
error: Failed to shutdown domain el6
error: Timed out during operation: cannot acquire state change lock

[root@troy libvirt]# ls -lh
total 736M
-rw-------. 1 root root    0 Apr  6 11:37 libvirtd.log
-rw-------. 1 root root 356M Apr  6 03:10 libvirtd.log-20120406
-rw-------. 1 root root 381M Apr  6 11:45 libvirtd.log-20120406-02
drwx------. 2 root root 4.0K Feb 22 10:24 lxc
drwx------. 2 root root 4.0K Apr  5 10:48 qemu
drwx------. 2 root root 4.0K Feb 22 10:24 uml
Comment 14 Wayne Sun 2012-04-06 06:17:39 EDT
packages:
libvirt-0.9.10-10.el6.x86_64
libvirt-python-0.9.10-10.el6.x86_64
qemu-kvm-0.12.1.2-2.267.el6ev.x86_64
kernel-2.6.32-257.el6.x86_64

steps:
1. using the xml in comment 12 to define a domain(there is macvtap dircet type interfaece in domain xml)
2. start the domain
# virsh start el6

It will hung at here. No error spotted in libvirtd.log.

Cancel the operation with ^C.
3. check guest state
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     el6                            shut off


check qemu-kvm process
# ps aux|grep qemu-kvm
qemu     23193 97.5  0.0 1382668 21436 ?       Rl   18:03   0:58 /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name el6 -uuid d290f06e-e84f-e7bc-cbf9-c5fb3ed65f9b -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/el6.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/images/el6.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3f:22:21,bus=pci.0,addr=0x3,bootindex=1 -netdev tap,fd=23,id=hostnet1 -device rtl8139,netdev=hostnet1,id=net1,mac=52:54:00:33:78:95,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

4. try to restart or destroy domain
# virsh start el6
error: Domain is already active

# virsh shutdown el6
error: Failed to shutdown domain el6
error: Timed out during operation: cannot acquire state change lock

The symptoms are quite the same with Bug 786648, so I think this is a duplicate.
Comment 15 Jiri Denemark 2012-04-06 08:15:22 EDT
Yeah, it seems like it could be a duplicate of that bug. To confirm that, could you get libvirtd backtrace ("thread apply all backtrace" command in gdb) when after you attempted to start the domain? And also attach /var/log/libvirt/qemu/DOMAIN.log, please.
Comment 16 Wayne Sun 2012-04-08 23:39:29 EDT
Created attachment 576107 [details]
libvirtd backtrace log

libvirtd backtrace log after virsh start hung
Comment 17 Wayne Sun 2012-04-08 23:41:00 EDT
Created attachment 576108 [details]
domain qemu log
Comment 19 Jiri Denemark 2012-04-10 02:52:21 EDT
The backtrace is unusable (probably because of missing debuginfo package) but the qemu log makes me confident this is a duplicate of bug 786648.

*** This bug has been marked as a duplicate of bug 786648 ***

Note You need to log in before you can comment on or make changes to this bug.