1384007 – The lifecycle event for Guest OS Shutdown is not distinguishable from a qemu process that was quit with SIG_TERM

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1384007 - The lifecycle event for Guest OS Shutdown is not distinguishable from a qemu process that was quit with SIG_TERM

Summary: The lifecycle event for Guest OS Shutdown is not distinguishable from a qemu ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Martin Kletzander
QA Contact:	Yanqiu Zhang
Docs Contact:
URL:
Whiteboard:
Depends On:	1418927
Blocks:
TreeView+	depends on / blocked

Reported:	2016-10-12 10:50 UTC by Vinzenz Feenstra [evilissimo]
Modified:	2017-09-25 12:01 UTC (History)
CC List:	15 users (show)
Fixed In Version:	libvirt-3.2.0-7.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1418927 (view as bug list)
Environment:
Last Closed:	2017-08-01 17:16:43 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1463188	0	unspecified	CLOSED	event-test.py gets traceback when guest shutdown	2021-02-22 00:41:40 UTC
Red Hat Product Errata	RHEA-2017:1846	0	normal	SHIPPED_LIVE	libvirt bug fix and enhancement update	2017-08-01 18:02:50 UTC

Internal Links: 1463188

Description Vinzenz Feenstra [evilissimo] 2016-10-12 10:50:49 UTC

Description of problem:

When a libvirt managed qemu process gets signaled to quit, libvirt reports the same lifecylce event as if the guest operating system shuts down.


Version-Release number of selected component (if applicable):
libvirt-daemon-1.2.17-13.el7_2.6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create a virtual machine with a guest operating system (e.g. Linux)
2. Start the virtual machine 
3. virsh -r event <DOMID> --all
4. Get PID for the qemu process of that VM
5. kill -15 $PID

Actual results:
event 'lifecycle' for domain <DOMAINNAME>: Shutdown Finished

Expected results:
A different event detail that allows to distinguish a shutdown from within the Virtual Machine from a process quit via signal from the hypervisor side.

Additional info:

Comment 2 Martin Kletzander 2017-02-03 07:38:25 UTC

Just a quick test for the two cases shows:

On 'virsh shutdown domain' QEMU sends us:
  SHUTDOWN
  STOP
  SHUTDOWN

whether on 'kill $domain_pid' we get:
  SHUTDOWN

So there are two problems.  First is that the difference comes *after* the point where we consider the domain to be shut down (and we have already emitted the libvirt event).  More importantly, this is not guaranteed to happen (I just tried it once) and none of the differences (the stop or the second shutdown) make sense after SHUTDOWN was recieved.

The only way to differentiate this would be by adding information to the event in QEMU.  So I'm cloning this BZ to QEMU for the requested support.

Comment 3 Martin Kletzander 2017-02-03 07:51:57 UTC

In the meantime, if you really need to somehow at least try to distinguish clean and non-clean shutdown, you can, as a workaround, check for qemu agent disconnection event.  For SIGTERM you should not get it.

Comment 4 Nir Soffer 2017-02-23 17:57:43 UTC

This is important for RHV when using using a vm lease:
https://www.ovirt.org/develop/release-management/features/storage/vm-leases/

If sanlock cannot review the lease and terminate the vm, libvirt should return
abnormal termination. Otherwise, RHV HA mechanism will treat the event as 
normal termination within the guest, and will not start the vm on another host.

If sanlock cannot terminate qemu with SIGTERM, it will send SIGKILL - in this case
libvirt cannot depend on anything from qemu. Libvirt can use output from qeme to
detect normal shutdown, and if there is no info from qemu, treat it as abnormal
termination.

Martin, do you think we can track this issue in this bug, or we should open a new
bug focused on the sanlock use case?

Comment 5 Martin Kletzander 2017-02-24 06:52:16 UTC

(In reply to Nir Soffer from comment #4)
What you describe is slightly different.  It is of course possible to distinguish between SIGKILL'd and SIGTERM'd QEMU, that's not a problem.  If libvirt doesn't get any info from the VM and the process disappears, then the state reason is "crashed" instead of "shutdown".

What this BZ is about is that it is not distinguishable between SIGTERM and normal shutdown.  For both of these events we get the same information from qemu, so it should be 

@Vincenzo: I am wondering if, as a workaround or just for thought, it would be enough to use your guest agent that would send you info when it's being stopped as that would be a normal shutdown for you and no info would be killed process.

Comment 6 Vinzenz Feenstra [evilissimo] 2017-02-24 06:53:45 UTC

This is what we do as a temporary workaround in the long run I am expecting however libvirt and qemu to report this difference in someway.

Comment 7 Michal Skrivanek 2017-02-24 08:01:39 UTC

(In reply to Martin Kletzander from comment #5)

> @Vincenzo: I am wondering if, as a workaround or just for thought, it would
> be enough to use your guest agent that would send you info when it's being
> stopped as that would be a normal shutdown for you and no info would be
> killed process.

it's not a great workaround as it is a simple system service which can stop and restart any time

Comment 8 Martin Kletzander 2017-02-24 08:13:54 UTC

Sure, I get that you need better solution, but the fact that my idea works just proves that I understood exactly what you need.  Thanks a lot.

Comment 11 Eric Blake 2017-04-06 21:17:30 UTC

Depending on upstream reaction, qemu 2.10 may start advertising when a signal was the cause of a SHUTDOWN event:
https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg01098.html

Comment 12 Martin Kletzander 2017-04-12 14:48:42 UTC

Preliminary patch for libvirt posted here:

https://www.redhat.com/archives/libvir-list/2017-April/msg00622.html

Comment 14 Martin Kletzander 2017-05-16 20:31:03 UTC

Another version posted upstream:

https://www.redhat.com/archives/libvir-list/2017-May/msg00540.html

Comment 19 Yanqiu Zhang 2017-06-16 09:51:00 UTC

Reproduced with libvirt-2.0.0-10.el7_3.9.x86_64, qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
Steps:
1.# virsh shutdown rhel7.3
Domain rhel7.3 is being shutdown
# virsh event --all --loop
# event 'lifecycle' for domain rhel7.3: Shutdown Finished
event 'lifecycle' for domain rhel7.3: Stopped Shutdown
 
2.# kill -15 $PID
# virsh event --all --loop
event 'lifecycle' for domain rhel7.3: Shutdown Finished
event 'lifecycle' for domain rhel7.3: Stopped Shutdown

 
Verify on rhel7.4 with libvirt-3.2.0-10.el7.x86_64, qemu-kvm-rhev-2.9.0-10.el7.x86_64
Steps:
1.# virsh shutdown V (or in guest: shutdown -h; poweroff )
# virsh event V --loop --all
event 'lifecycle' for domain V: Shutdown Finished after guest request
event 'lifecycle' for domain V: Stopped Shutdown
 
2.# kill -1/2/15 $PID
# virsh event V --loop --all
event 'lifecycle' for domain V: Shutdown Finished after host request
event 'lifecycle' for domain V: Stopped Shutdown

Comment 20 Yanqiu Zhang 2017-06-16 10:33:26 UTC

mark as verified per comment 19.

Comment 21 errata-xmlrpc 2017-08-01 17:16:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 22 errata-xmlrpc 2017-08-01 23:57:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Note You need to log in before you can comment on or make changes to this bug.