Bug 1791458

Summary:	VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY event is emitted for incoming migration
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Jiri Denemark <jdenemar>
Component:	libvirt	Assignee:	Jiri Denemark <jdenemar>
Status:	CLOSED ERRATA	QA Contact:	yafu <yafu>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.1	CC:	aefrat, bzlotnik, chhu, dyuan, fjin, gwatson, jdenemar, lmen, lsurette, michal.skrivanek, mzamazal, rhodain, srevivo, tnisan, xuzhang, yafu, yanqzhan, ycui
Target Milestone:	rc	Keywords:	Upstream
Target Release:	8.2	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-6.0.0-2.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1774230
Clones:	1791886 (view as bug list)		Environment:
Last Closed:	2020-05-05 09:55:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1774230, 1791886

Description Jiri Denemark 2020-01-15 21:11:54 UTC

+++ This bug was initially created as a clone of Bug #1774230 +++

Description of problem:

During post-copy migration libvirtd on the destination host unexpectedly emits
VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY lifecycle event just before resuming the
migration in post-copy mode.

Version-Release number of selected component (if applicable):

Any libvirt version since RHEL 7.7 (bug 1647365):
libvirt-4.5.0-23.el7_7.5
libvirt-4.5.0-31.el7
libvirt-4.5.0-35.2.el8
libvirt-5.6.0-10.el8
libvirt-6.0.0-1.el8

How reproducible:

100%

Steps to Reproduce:

1. start a new domain on a source host
2. make the domain dirty memory (e.g., by running stress command):
    stress --vm 2 --vm-bytes 512M
3. start watching for lifecycle events on a destination host:
    virsh event --event lifecycle --loop --timestamp
4. migrate the domain from the source host to the destination host:
    virsh migrate --p2p --live --postcopy --postcopy-after-precopy $DOM $DEST_URI
5. check the lifecycle events reported on the destination during migration

Actual results:

2020-01-15 14:20:26.689+0000: event 'lifecycle' for domain nest: Started Migrated
2020-01-15 14:21:01.837+0000: event 'lifecycle' for domain nest: Suspended Post-copy
2020-01-15 14:21:03.266+0000: event 'lifecycle' for domain nest: Resumed Post-copy
2020-01-15 14:21:32.060+0000: event 'lifecycle' for domain nest: Resumed Migrated

Expected results:

2020-01-15 14:28:53.803+0000: event 'lifecycle' for domain nest: Started Migrated
2020-01-15 14:28:56.156+0000: event 'lifecycle' for domain nest: Resumed Post-copy
2020-01-15 14:28:56.258+0000: event 'lifecycle' for domain nest: Resumed Migrated

In other words, no "Suspended Post-copy" event should be reported.

Additional info:

This issue was nicely analyzed in the original bug 1774230:

--- Additional comment from Benny Zlotnik on 2020-01-15 09:02:25 UTC ---

Hi Jiri,

After investigating this bug and discussing the proposed patch with Milan,
there is something unclear. It seems that in post-copy migrate both source and
destination receive VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY, and it is probably
has something to do with the change from[1], as I see the following logs on
the destination:

2020-01-15 08:17:54.802+0000: 17327: debug : qemuProcessHandleMigrationStatus:1647 :
    Migration of domain 0x7fae6801c310 vmski changed state to post-copy-active
2020-01-15 08:17:54.802+0000: 17327: debug : qemuProcessHandleMigrationStatus:1663 :
    Correcting paused state reason for domain vmski to post-copy <--- I assume this emits the VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY event
2020-01-15 08:17:55.045+0000: 17327: debug : qemuProcessHandleResume:719 :
    Transitioned guest vmski into running state, reason 'post-copy', event detail 3

Is this the correct the behaviour, should the destination receive VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY as well?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1647365

--- Additional comment from Jiri Denemark on 2020-01-15 10:06:46 UTC ---

Your investigation seems to be correct. The domain is started as paused on the
destination with "migration" reason. Once migration switches to post-copy, the
code in qemuProcessHandleMigrationStatus will update the reason to "post-copy"
and emit a "suspended" event just a moment before the domain is resumed, which
should only happen on the source.

Comment 1 Jiri Denemark 2020-01-16 13:18:53 UTC

Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2020-January/msg00732.html

Comment 2 Jiri Denemark 2020-01-16 14:40:28 UTC

This is now fixed upstream by

commit bd04d63ad97c21b6955710e6473a502f49816a3c
Refs: v6.0.0-23-gbd04d63ad9
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Jan 15 15:24:55 2020 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Jan 16 15:12:19 2020 +0100

    qemu: Don't emit SUSPENDED_POSTCOPY event on destination

    When pause-before-switchover QEMU capability is enabled, we get STOP
    event before MIGRATION event with postcopy-active state. To properly
    handle post-copy migration and emit correct events commit
    v4.10.0-rc1-4-geca9d21e6c added a hack to
    qemuProcessHandleMigrationStatus which translates the paused state
    reason to VIR_DOMAIN_PAUSED_POSTCOPY and emits
    VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY event when migration state changes
    to post-copy.

    However, the code was effective on both sides of migration resulting in
    a confusing VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY event on the destination
    host, where entering post-copy mode is already properly advertised by
    VIR_DOMAIN_EVENT_RESUMED_POSTCOPY event.

    https://bugzilla.redhat.com/show_bug.cgi?id=1791458

    Signed-off-by: Jiri Denemark <jdenemar>
    Reviewed-by: Ján Tomko <jtomko>

Comment 5 yafu 2020-02-11 03:53:42 UTC

Verified with libvirt-6.0.0-4.module+el8.2.0+5642+838f3513.x86_64.

Test steps are the same with https://bugzilla.redhat.com/show_bug.cgi?id=1791886#c10 and https://bugzilla.redhat.com/show_bug.cgi?id=1791886#c11.

Comment 7 errata-xmlrpc 2020-05-05 09:55:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017