Bug 739895 - "virsh shutdown fooguest" results in a 'paused' state
Summary: "virsh shutdown fooguest" results in a 'paused' state
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 747543
TreeView+ depends on / blocked
 
Reported: 2011-09-20 11:02 UTC by Kashyap Chamarthy
Modified: 2011-10-20 08:43 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 747543 (view as bug list)
Environment:
Last Closed: 2011-09-22 09:10:30 UTC


Attachments (Terms of Use)
virsh shutdown fooguest with LIBVIRT_DEBUG=1 (18.50 KB, text/plain)
2011-09-20 11:03 UTC, Kashyap Chamarthy
no flags Details

Description Kashyap Chamarthy 2011-09-20 11:02:32 UTC
Description of problem:
With the newest libvirt "virsh shutdown fooguest"  results in a 'paused' state

Version-Release number of selected component (if applicable):
For F15, I made a scratch build using the libvirt-0.9.5-1.fc14.src.rpm from ftp://libvirt.org/libvirt/   (http://koji.fedoraproject.org/koji/taskinfo?taskID=3363547)
######################
[root@moon ~]# cat /etc/redhat-release 
Fedora release 15 (Lovelock)
######################
[root@moon ~]# rpm -q libvirt libvirt-client
libvirt-0.9.5-1.fc15.x86_64
libvirt-client-0.9.5-1.fc15.x86_64
[root@moon ~]# 
######################

How reproducible:
All the time

Steps to Reproduce:
1. create a new fedora-15 minimal guest (only @core)
2. stop the guest using virsh

#####################
[root@moon ~]# virsh list 
 Id Name                 State
----------------------------------
  5 f15foo10             running

[root@moon ~]# 
#####################
[root@moon ~]# virsh shutdown f15foo10
#####################
[root@moon ~]# virsh list 
 Id Name                 State
----------------------------------
  5 f15foo10             paused

[root@moon ~]# 
#####################  


Actual results:
Guest shutdown result is 'paused' instead of shutdown

Expected results:
Guest should shutdown should happen successfully.  

Additional info:
Attached is the output of 'virsh shutdown f15foo10' with  LIBVIRT_DEBUG=1

Comment 1 Kashyap Chamarthy 2011-09-20 11:03:57 UTC
Created attachment 524000 [details]
virsh shutdown fooguest with  LIBVIRT_DEBUG=1

Comment 2 Jiri Denemark 2011-09-20 12:15:28 UTC
This is most likely a qemu bug. What does "virsh domstate --reason f15foo10" say when you see the guest is paused? What is the version of installed qemu package?

Comment 3 Kashyap Chamarthy 2011-09-20 12:25:32 UTC
Jiri,

Domain state and qemu-kvm version here:

###################
[root@moon ~]# virsh domstate --reason f15foo10
paused (shutting down)
###################
[root@moon ~]# rpm -q  qemu-kvm
qemu-kvm-0.15.0-4.fc15.x86_64
[root@moon ~]# 
###################

Comment 4 Jiri Denemark 2011-09-20 12:52:52 UTC
Thanks, the qemu-kvm package is most likely missing this patch: 

http://lists.nongnu.org/archive/html/qemu-devel/2011-09/msg01757.html

Comment 5 Daniel Berrangé 2011-09-20 13:01:41 UTC
Hmm, this is not very good, because it means virsh shutdown is now broken for every released version of QEMU in existence surely ?  I don't think we can go with such a widespread regression, even though the bug is technically in QEMU, we ought to avoid it in libvirt.

Comment 6 Jiri Denemark 2011-09-20 13:28:15 UTC
Not all QEMU versions. The bug was introduced sometime around 0.14.0-rc. Earlier versions should work fine I believe. We could avoid it by not using -no-shutdown for QEMU 0.14 and 0.15 (I don't see a better way of detecting if the bug is there or not).

Comment 7 Daniel Veillard 2011-09-20 13:48:14 UTC
Confirmed, on F14 with qemu-kvm-0.13.0-1.fc14.x86_64, virsh shutdown
seems to works normally:

[root@paphio ~]# virsh list 
 Id Name                 State
----------------------------------
  1 RHEL-5.4-64          running

[root@paphio ~]# virsh shutdown RHEL-5.4-64
Domain RHEL-5.4-64 is being shutdown

[root@paphio ~]# rpm -q qemu-kvm
qemu-kvm-0.13.0-1.fc14.x86_64
[root@paphio ~]# virsh list 
 Id Name                 State
----------------------------------

[root@paphio ~]# virsh list  --all
 Id Name                 State
----------------------------------
  - RHEL-5.4-64          shut off

Daniel

Comment 8 Jiri Denemark 2011-09-21 20:45:23 UTC
This issue should be fixed upstream by:

commit f84aedad090da1e05ccc5651815febba013eb3ad
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Sep 21 10:25:29 2011 +0200

    qemu: Fix shutdown regression with buggy qemu
    
    The commit that prevents disk corruption on domain shutdown
    (96fc4784177ecb70357518fa863442455e45ad0e) causes regression with QEMU
    0.14.* and 0.15.* because of a regression bug in QEMU that was fixed
    only recently in QEMU git. The affected versions of QEMU do not quit on
    SIGTERM if started with -no-shutdown, which we use to implement fake
    reboot. Since -no-shutdown tells QEMU not to quit automatically on guest
    shutdown, domains started using the affected QEMU cannot be shutdown
    properly and stay in a paused state.
    
    This patch disables fake reboot feature on such QEMU by not using
    -no-shutdown, which makes shutdown work as expected. However,
    virDomainReboot will not work in this case and it will report "Requested
    operation is not valid: Reboot is not supported with this QEMU binary".

Comment 9 Jiri Denemark 2011-09-22 09:10:30 UTC
This patch is now included in libvirt-0.9.6 release.

Comment 10 Kashyap Chamarthy 2011-09-22 11:34:47 UTC
Thanks. VERIFIED

I made a quick scratch build(for F15) using libvirt F14 SRPM  http://koji.fedoraproject.org/koji/taskinfo?taskID=3369923

And shutting down the guest is now graceful.

##########################################################################
[root@moon ~]# rpm -q libvirt
libvirt-0.9.6-1.fc15.x86_64
##########################################################################
[root@moon ~]# virsh list
 Id Name                 State
----------------------------------
  7 f15foo10             running
##########################################################################
[root@moon ~]# virsh shutdown f15foo10 ; virsh console f15foo10
Domain f15foo10 is being shutdown

Connected to domain f15foo10
Escape character is ^]
Killing mdmonitor: Stopping PC/SC smart card daemon (pcscd): Shutting down sm-client: [  143.322077] systemd[1]: var-lib-nfs-rpc_pipefs.mount mount process exited, code=exited status=1
[  OK  ]
Stopping sshd: [  143.475185] smartd[621]: smartd received signal 15: Terminated
[  143.475202] smartd[621]: smartd is exiting (exit status 0)
[  143.475213] acpid[644]: exiting
[  143.475233] modem-manager[672]: <info> Caught signal 15, shutting down...
[  143.475245] NetworkManager[600]: <warn> disconnected by the system bus.
[  143.475255] NetworkManager[600]: <info> caught signal 15, shutting down normally.
[  143.475273] NetworkManager[600]: <warn> quit request received, terminating...
[  143.475286] NetworkManager[600]: <info> exiting (success)
[  143.544614] sshd[797]: Received signal 15; terminating.
Stopping RPC idmapd: [  143.582970] systemd[1]: sshd.service: main process exited, code=exited, status=255
Stopping NFS statd: Shutting down sendmail: [  143.599799] rpc.statd[825]: Caught signal 15, un-registering and exiting
[  143.604281] systemd[1]: nfslock.service: main process exited, code=exited, status=1
[  OK  ]
[  OK  ]
[  143.665284] systemd[1]: Unit sshd.service entered failed state.
[  OK  ]


[  143.737185] systemd[1]: Unit nfslock.service entered failed state.
[  144.297288] systemd[1]: pcscd.service: main process exited, code=exited, status=1
[  OK  ]
[  144.461158] systemd[1]: Unit pcscd.service entered failed state.
Stopping rpcbind: [  144.720246] rpcbind[720]: rpcbind terminating on signal. Restart with "rpcbind -w"
[  144.724478] systemd[1]: rpcbind.service: main process exited, code=exited, status=2
[  OK  ]
[  144.818168] systemd[1]: Unit rpcbind.service entered failed state.
Stopping auditd: [  144.871352] type=1305 audit(1316690292.412:97): audit_pid=0 old=683 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[  144.873205] auditd[683]: The audit daemon is exiting.
[  OK  ]
[  144.980974] type=1305 audit(1316690292.521:98): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Not stopping monitoring, this is a dangerous operation. Please use force-stop to override.
[  145.095679] systemd[1]: lvm2-monitor.service: control process exited, code=exited status=1
[  145.112677] systemd[1]: Unit lvm2-monitor.service entered failed state.
iptables: Flushing firewall rules: ip6tables: Flushing firewall rules: [  OK  ]
[  OK  ]
iptables: Setting chains to policy ACCEPT: filter ip6tables: Setting chains to policy ACCEPT: filter [  OK  ]
iptables: Unloading modules: [  OK  ]
ip6tables: Unloading modules: [  OK  ]
[  OK  ]
[  146.994142] systemd[1]: Shutting down.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
[  147.337958] EXT4-fs (dm-1): re-mounted. Opts: (null)
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Not all DM devices detached, 1 left.
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and de[  147.361770] md: stopping all md devices.
vices, trying to kill remaining processes.
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and devices, giving up.
[  148.364613] ACPI: Preparing to enter system sleep state S5
[  148.366414] Disabling non-boot CPUs ...
[  148.369465] Unregister pv shared memory for cpu 1
[  148.371393] Broke affinity for irq 4
[  148.372091] Broke affinity for irq 43
[  148.428899] Power down.
##########################################################################
[root@moon ~]# virsh list
 Id Name                 State
----------------------------------

[root@moon ~]# 
##########################################################################


Note You need to log in before you can comment on or make changes to this bug.