Bug 1953062

I can use the method Igor mentioned in the description to reproduce this problem:

Test env:
host:
4.18.0-304.el8.x86_64
qemu-kvm-5.2.0-15.module+el8.4.0+10650+50781ca0.x86_64
guest:
4.18.0-304.el8.x86_64

Test step:
(1) start a vm
/usr/libexec/qemu-kvm  -enable-kvm -m 1g -M q35 -monitor stdio -vnc :0 /home/images/RHEL84.qcow2   



(2) do the offline migration and 
(qemu)  migrate "exec:gzip -c > STATEFILE.gz"

(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: completed 
total time: 24100 ms
downtime: 25 ms
setup: 2 ms
transferred ram: 426188 kbytes
throughput: 144.96 mbps
remaining ram: 0 kbytes
total ram: 1065800 kbytes
duplicate: 164847 pages
skipped: 0 pages
normal: 105977 pages
normal bytes: 423908 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 6294

(3) After the first offline migration is completed, do the offline migration again

(qemu)  migrate "exec:gzip -c > STATEFILE.gz"
qemu-kvm: ../block.c:5895: bdrv_inactivate_recurse: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.
Aborted (core dumped)


(gdb) bt
#0  0x00007fea98a7737f in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fea98a61db5 in __GI_abort () at abort.c:79
#2  0x00007fea98a61c89 in __assert_fail_base
    (fmt=0x7fea98bca5f8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55ee7427c220 "!(bs->open_flags & BDRV_O_INACTIVE)", file=0x55ee7427b6a0 "../block.c", line=5895, function=<optimized out>) at assert.c:92
#3  0x00007fea98a6fa76 in __GI___assert_fail
    (assertion=assertion@entry=0x55ee7427c220 "!(bs->open_flags & BDRV_O_INACTIVE)", file=file@entry=0x55ee7427b6a0 "../block.c", line=line@entry=5895, function=function@entry=0x55ee7427d6f0 <__PRETTY_FUNCTION__.35587> "bdrv_inactivate_recurse") at assert.c:101
#4  0x000055ee740af259 in bdrv_inactivate_recurse (bs=0x55ee7569ddf0) at ../block.c:5895
#5  0x000055ee740b073f in bdrv_inactivate_all () at ../block.c:5963
#6  0x000055ee73e4c995 in qemu_savevm_state_complete_precopy_non_iterable
    (inactivate_disks=true, in_postcopy=false, f=0x55ee7579ab30) at ../migration/savevm.c:1399
#7  0x000055ee73e4c995 in qemu_savevm_state_complete_precopy
    (f=0x55ee7579ab30, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
    at ../migration/savevm.c:1451
#8  0x000055ee73deb1b6 in migration_completion (s=0x55ee756a3870) at ../migration/migration.c:3049
#9  0x000055ee73deb1b6 in migration_iteration_run (s=0x55ee756a3870) at ../migration/migration.c:3405
#10 0x000055ee73deb1b6 in migration_thread (opaque=0x55ee756a3870) at ../migration/migration.c:3573
#11 0x000055ee7415ed64 in qemu_thread_start (args=0x55ee761b5860) at ../util/qemu-thread-posix.c:521
#12 0x00007fea98e0d14a in start_thread (arg=<optimized out>) at pthread_create.c:479
#13 0x00007fea98b3cdc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Reproduce this bz when try to migrate twice time on rhelav-8.4.0 (kernel-4.18.0-304.3.el8.x86_64 & qemu-kvm-5.2.0-15.module+el8.4.0+10650+50781ca0.x86_64). Qemu core dump on the second migration
******************************************************************************************************************************
(qemu) info migrateqemu-kvm: ../block.c:5895: bdrv_inactivate_recurse: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.


I want to explain that twice migrate about offline migration is meaningless and this error is right because guest is paused after the first migration completed, please see [1]. But if you execute "cont" on src hmp after first migration completed, migration could start again for the second time and finished successfully[2]. I have tried and succeed on above environment. 
[1]
(qemu) info status 
VM status: paused (postmigrate)

[2]
(qemu) continue
(qemu) info status 
VM status: running


I also tried on rhelav-8.5.0 (kernel-4.18.0-304.3.el8.x86_64 & qemu-img-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64), didn't reproduce this bz as it would give warning if you try to migrate twice when guest is paused on src host after the first migration:
*********************************************************************
(qemu) info status 
VM status: paused (postmigrate)
(qemu) migrate -d "exec:gzip -c > STATEFILE.gz"
Error: Can't migrate the vm that was paused due to previous migration



Also tried this scenario with libvirt (libvirt-daemon-7.0.0-10.module+el8.4.0+10417+37f6984d.x86_64) on rhelav-8.4.0, libvirt will destroy vm after save vm, so libvirt won't hit such scenarios, you can't migrate for the second time:
*******************************************************************
[root@ibm-x3250m6-07 home]# virsh save rhel8.4 rhel8.4-save

Domain 'rhel8.4' saved to rhel8.4-save

[root@ibm-x3250m6-07 home]# virsh list
 Id   Name   State
--------------------

[root@ibm-x3250m6-07 home]# virsh save rhel8.4 rhel8.4-save
error: Failed to save domain 'rhel8.4' to rhel8.4-save
error: Requested operation is not valid: domain is not running



Indeed we'd better give a warning to inform guest is paused when you try to migrate for the second time on rhelav-8.4.0 like the prompt on rhelav-8.5.0.
I would close this bz as currentrelease then per above tests. Add comments if you have different opinions.