We have same bug fixed on rhel 8.0.1:
https://bugzilla.redhat.com/show_bug.cgi?id=1652572

(In reply to Li Xiaohui from comment #0)
> Description of problem:
> Guest is shared by nfs server, then migrate guest from src to dst host,
> during migration is active, stop nfs service on nfs server. Qemu core dump
> after nearly 10 mins.
> 
> 
> Version-Release number of selected component (if applicable):
> hosts info: kernel-5.14.0-63.el9.x86_64 & qemu-kvm-6.2.0-10.el9.x86_64
> 
> 
> How reproducible:
> 100%
> 
> 
> Steps to Reproduce:
> 1.Boot a guest with nfs storage shared
> 2.Migrate guest from src to dst host
> 3.During migration is active, stop nfs service on nfs server
> 
> 
> Actual results:
> After step 3, wait for nearly 10 mins, qemu on dst host hit core dump, after
> recover nfs service, qemu on src host also core dump:
> dst qemu core dump:
> (qemu) 2022-02-27T13:06:17.885961Z qemu-kvm: Failed to load
> virtio_pci/modern_queue_state:avail
> 2022-02-27T13:06:17.886034Z qemu-kvm: Failed to load
> virtio_pci/modern_state:vqs
> 2022-02-27T13:06:17.886068Z qemu-kvm: Failed to load
> virtio/extra_state:extra_state
> 2022-02-27T13:06:17.886107Z qemu-kvm: Failed to load virtio-net:virtio
> 2022-02-27T13:06:17.886145Z qemu-kvm: error while loading state for instance
> 0x0 of device '0000:00:02.2:00.0/virtio-net'
> 2022-02-27T13:06:17.886978Z qemu-kvm: load of migration failed: Input/output
> error

Did the destination actually core dump or just give these messages?
These messages appear to just be the result of the source failing first.

> src qemu core dump:
> (qemu) 2022-02-27T13:06:17.849177Z qemu-kvm:
> qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all()
> failed (-1)
> ******

Which is probably true; the source failed to flush something to the NFS server
and got an error.

> (qemu) 1.sh: line 42: 42236 Segmentation fault

Still, it shouldn't seg - do you have a backtrace or the core?

Dave

> 
> 
> 
> Expected results:
> Qemu should not hit core dump, migration should continue after nfs server
> recovery.
> 
> 
> Additional info:

Test again today on Naples hosts, get the core dump file on src host


(In reply to Dr. David Alan Gilbert from comment #2)
> (In reply to Li Xiaohui from comment #0)
...
> > 
> > 
> > Actual results:
> > After step 3, wait for nearly 10 mins, qemu on dst host hit core dump, after
> > recover nfs service, qemu on src host also core dump:
> > dst qemu core dump:
> > (qemu) 2022-02-27T13:06:17.885961Z qemu-kvm: Failed to load
> > virtio_pci/modern_queue_state:avail
> > 2022-02-27T13:06:17.886034Z qemu-kvm: Failed to load
> > virtio_pci/modern_state:vqs
> > 2022-02-27T13:06:17.886068Z qemu-kvm: Failed to load
> > virtio/extra_state:extra_state
> > 2022-02-27T13:06:17.886107Z qemu-kvm: Failed to load virtio-net:virtio
> > 2022-02-27T13:06:17.886145Z qemu-kvm: error while loading state for instance
> > 0x0 of device '0000:00:02.2:00.0/virtio-net'
> > 2022-02-27T13:06:17.886978Z qemu-kvm: load of migration failed: Input/output
> > error
> 
> Did the destination actually core dump or just give these messages?

The dst only give some messages and then quit by automatically.
this time, dst hmp print:
(qemu) 2022-03-02T12:57:54.185678Z qemu-kvm: load of migration failed: Input/output error


> These messages appear to just be the result of the source failing first.
> 
> > src qemu core dump:
> > (qemu) 2022-02-27T13:06:17.849177Z qemu-kvm:
> > qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all()
> > failed (-1)
> > ******
> 
> Which is probably true; the source failed to flush something to the NFS
> server
> and got an error.

This time, src qemu abort after nearly 10 mins:
(qemu) 2022-03-02T12:57:54.173253Z qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)
qemu-kvm: ../block/io.c:1991: int bdrv_co_write_req_prepare(BdrvChild *, int64_t, int64_t, BdrvTrackedRequest *, int): Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.
1.sh: line 42: 38286 Aborted                 (core dumped)

> 
> > (qemu) 1.sh: line 42: 42236 Segmentation fault
> 
> Still, it shouldn't seg - do you have a backtrace or the core?

Yeh, get a core dump on src host:
[root@hp-dl385g10-09 ~]# coredumpctl info 38286
           PID: 38286 (qemu-kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Wed 2022-03-02 07:57:54 EST (8min ago)
  Command Line: /usr/libexec/qemu-kvm -name mouse-vm -sandbox on -machine q35,memory-backend=pc.ram -cpu EPYC -nodefaults -chardev socket,id=qmp_>
    Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 3b818ce11e6549ebabcfeeabc40dac5b
    Machine ID: d7ff064019404f1aaf6484e65c686b6f
      Hostname: hp-dl385g10-09.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.3b818ce11e6549ebabcfeeabc40dac5b.38286.1646225874000000.zst (truncated)
     Disk Size: 22.5M
       Message: Process 38286 (qemu-kvm) of user 0 dumped core.
                
                Module /usr/libexec/qemu-kvm with build-id bf4e35b9a0dc7bad6718b694934d49423eed2ea4
                Stack trace of thread 38286:
                #0  0x00007fd8940b143c n/a (n/a + 0x0)
                ELF object binary architecture: AMD x86-64


Core dump file is uploaded to http://fileshare.englab.nay.redhat.com/pub/logs/xiaohli/x86_64/bug/bz2058982/

> 
> Dave
> 
> > 
> > 
> > 
> > Expected results:
> > Qemu should not hit core dump, migration should continue after nfs server
> > recovery.
> > 
> > 
> > Additional info:

(In reply to Li Xiaohui from comment #3)
> Test again today on Naples hosts, get the core dump file on src host
> 
> 
> (In reply to Dr. David Alan Gilbert from comment #2)
> > (In reply to Li Xiaohui from comment #0)
> ...
> > > 
> > > 
> > > Actual results:
> > > After step 3, wait for nearly 10 mins, qemu on dst host hit core dump, after
> > > recover nfs service, qemu on src host also core dump:
> > > dst qemu core dump:
> > > (qemu) 2022-02-27T13:06:17.885961Z qemu-kvm: Failed to load
> > > virtio_pci/modern_queue_state:avail
> > > 2022-02-27T13:06:17.886034Z qemu-kvm: Failed to load
> > > virtio_pci/modern_state:vqs
> > > 2022-02-27T13:06:17.886068Z qemu-kvm: Failed to load
> > > virtio/extra_state:extra_state
> > > 2022-02-27T13:06:17.886107Z qemu-kvm: Failed to load virtio-net:virtio
> > > 2022-02-27T13:06:17.886145Z qemu-kvm: error while loading state for instance
> > > 0x0 of device '0000:00:02.2:00.0/virtio-net'
> > > 2022-02-27T13:06:17.886978Z qemu-kvm: load of migration failed: Input/output
> > > error
> > 
> > Did the destination actually core dump or just give these messages?
> 
> The dst only give some messages and then quit by automatically.
> this time, dst hmp print:
> (qemu) 2022-03-02T12:57:54.185678Z qemu-kvm: load of migration failed:
> Input/output error
> 
> 
> > These messages appear to just be the result of the source failing first.
> > 
> > > src qemu core dump:
> > > (qemu) 2022-02-27T13:06:17.849177Z qemu-kvm:
> > > qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all()
> > > failed (-1)
> > > ******
> > 
> > Which is probably true; the source failed to flush something to the NFS
> > server
> > and got an error.
> 
> This time, src qemu abort after nearly 10 mins:
> (qemu) 2022-03-02T12:57:54.173253Z qemu-kvm:
> qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all()
> failed (-1)
> qemu-kvm: ../block/io.c:1991: int bdrv_co_write_req_prepare(BdrvChild *,
> int64_t, int64_t, BdrvTrackedRequest *, int): Assertion `!(bs->open_flags &
> BDRV_O_INACTIVE)' failed.
> 1.sh: line 42: 38286 Aborted                 (core dumped)
> 
> > 
> > > (qemu) 1.sh: line 42: 42236 Segmentation fault
> > 
> > Still, it shouldn't seg - do you have a backtrace or the core?
> 
> Yeh, get a core dump on src host:
> [root@hp-dl385g10-09 ~]# coredumpctl info 38286
>            PID: 38286 (qemu-kvm)
>            UID: 0 (root)
>            GID: 0 (root)
>         Signal: 6 (ABRT)
>      Timestamp: Wed 2022-03-02 07:57:54 EST (8min ago)
>   Command Line: /usr/libexec/qemu-kvm -name mouse-vm -sandbox on -machine
> q35,memory-backend=pc.ram -cpu EPYC -nodefaults -chardev socket,id=qmp_>
>     Executable: /usr/libexec/qemu-kvm
>  Control Group: /user.slice/user-0.slice/session-1.scope
>           Unit: session-1.scope
>          Slice: user-0.slice
>        Session: 1
>      Owner UID: 0 (root)
>        Boot ID: 3b818ce11e6549ebabcfeeabc40dac5b
>     Machine ID: d7ff064019404f1aaf6484e65c686b6f
>       Hostname: hp-dl385g10-09.lab.eng.pek2.redhat.com
>        Storage:
> /var/lib/systemd/coredump/core.qemu-kvm.0.3b818ce11e6549ebabcfeeabc40dac5b.
> 38286.1646225874000000.zst (truncated)
>      Disk Size: 22.5M
>        Message: Process 38286 (qemu-kvm) of user 0 dumped core.
>                 
>                 Module /usr/libexec/qemu-kvm with build-id
> bf4e35b9a0dc7bad6718b694934d49423eed2ea4
>                 Stack trace of thread 38286:
>                 #0  0x00007fd8940b143c n/a (n/a + 0x0)
>                 ELF object binary architecture: AMD x86-64
> 
> 
> Core dump file is uploaded to
> http://fileshare.englab.nay.redhat.com/pub/logs/xiaohli/x86_64/bug/bz2058982/

That core file is truncated unfortunately; it only decompresses to 2G for me.
There are 2 or 3 ways to get a better core:

  a) Try with a VM with less than 1.5GB of RAM so that it'll make a small core.
  b) try attaching to the source qemu with gdb before it crashes and instead of getting
a core then you can backtrace in the gdb.


> > 
> > Dave
> > 
> > > 
> > > 
> > > 
> > > Expected results:
> > > Qemu should not hit core dump, migration should continue after nfs server
> > > recovery.
> > > 
> > > 
> > > Additional info:

> That core file is truncated unfortunately; it only decompresses to 2G for me.
> There are 2 or 3 ways to get a better core:

Thanks.

>   a) Try with a VM with less than 1.5GB of RAM so that it'll make a small core.

Got coredump file through this method: http://fileshare.englab.nay.redhat.com/pub/logs/xiaohli/x86_64/bug/bz2058982/

[root@hp-dl385g10-09 home]# coredumpctl info 
           PID: 40909 (qemu-kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Wed 2022-03-02 22:19:33 EST (1h 55min ago)
  Command Line: /usr/libexec/qemu-kvm -name mouse-vm -sandbox on -machine q35,memory-backend=pc.ram -cpu EPYC -nodefaults -chardev socket,id=qmp_>
    Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 3b818ce11e6549ebabcfeeabc40dac5b
    Machine ID: d7ff064019404f1aaf6484e65c686b6f
      Hostname: hp-dl385g10-09.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.3b818ce11e6549ebabcfeeabc40dac5b.40909.1646277573000000.zst (present)
     Disk Size: 379.4M
       Message: Process 40909 (qemu-kvm) of user 0 dumped core.
                
                Module linux-vdso.so.1 with build-id 4bc3040c5a4ab9cffbd9e46aaaef3be564a00d3c
                ......
                Module qemu-kvm with build-id bf4e35b9a0dc7bad6718b694934d49423eed2ea4
                Stack trace of thread 40909:
                #0  0x00007f2f2a14143c __pthread_kill_implementation (libc.so.6 + 0xa643c)
                #1  0x00007f2f2a0f4d06 __GI_raise (libc.so.6 + 0x59d06)
                #2  0x00007f2f2a0c77d3 __GI_abort (libc.so.6 + 0x2c7d3)
                #3  0x00007f2f2a0c76fb __assert_fail_base (libc.so.6 + 0x2c6fb)
                #4  0x00007f2f2a0edc86 __GI___assert_fail (libc.so.6 + 0x52c86)
                #5  0x0000556c5531016b bdrv_co_write_req_prepare (qemu-kvm + 0x7dc16b)
                #6  0x0000556c5530f6d1 bdrv_aligned_pwritev (qemu-kvm + 0x7db6d1)
                #7  0x0000556c5530ee43 bdrv_co_pwritev_part (qemu-kvm + 0x7dae43)
                #8  0x0000556c552fab52 blk_co_do_pwritev_part (qemu-kvm + 0x7c6b52)
                #9  0x0000556c552fafc7 blk_aio_write_entry (qemu-kvm + 0x7c6fc7)
                #10 0x0000556c554b6786 coroutine_trampoline (qemu-kvm + 0x982786)
                #11 0x00007f2f2a0c9330 n/a (libc.so.6 + 0x2e330)
                ELF object binary architecture: AMD x86-64


>   b) try attaching to the source qemu with gdb before it crashes and instead of getting
> a core then you can backtrace in the gdb.

I can't quite tell from the coredump; I think the  path is something like:
   1) bdrv_inactivate_all fails in qemu_savevm_state_complete_precopy_non_iterable
      (we see the error)
   2) static void migration_completion(MigrationState *s) has:
            if (ret >= 0) {
                qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
                ret = qemu_savevm_state_complete_precopy(s->to_dst_file, false,
                                                         inactivate);
            }
            if (inactivate && ret >= 0) {
                s->block_inactive = true;
            }

     so that means when bdrv_inactivate_all fails, block_inactive gets set true - which hmm maybe right
     but maybe wrong

   3) Something tries to restart the VM because the migration failed
      (we have some paths that do this automatically in the migration code)
     that probably goes through the fail_invalidate path - but hmm that doesn't
     check the block_inactive flag to know?

  But, hmm, now I'm confused since the assert I think says it's expecting it to be inactive
when it isn't, which is kind of the opposite case.

We need to try and reproduce this and follow the bdrv calls; either way I don't think everything above is consistent
in the way it sets or checks block_inactive.

If NFS server is down, we have pending writes, and if we have pending writes, bdrv_inactivate_all() will return one error.  That code has changed now with Exposito series if I remember correctly.

- Migration fails and aborts destination: That is expected.
- That the sounce core dumps, it is now

I will have to take a look at why we are not testing correctly what happens when we don't sent block_inactive.

Moved the bug to the Storage team, as bdrv_inactivate_all() is on the the block layer, not migration.

Test on qemu-kvm-6.2.0-11.el9_0.7 with nfsv4, can't hit this issue.
Will test on the latest qemu7.2, and add the test result later.

(In reply to aihua liang from comment #10)
> Test on qemu-kvm-6.2.0-11.el9_0.7 with nfsv4, can't hit this issue.
> Will test on the latest qemu7.2, and add the test result later.

Test on qemu-kvm-7.2.0-10.el9 with nfsv4, and still can't hit this issue.

Test on qemu-kvm-7.2.0-10.el9 with nfsv3+hard, and still can't hit this issue.

Hi, Xiaohui

 Can you help to check it on latest qemu version? If it works ok, we can close it as currentrelease.


BR,
Aliang

Hi I tried on qemu-kvm-7.2.0-10.el9 with mount cmd [1], hit a new situation after waiting more than 20 mins:

before recovery nfs service on nfs server, dst qemu will quit automatically:
(qemu) 2023-03-06T03:47:20.107866Z qemu-kvm: load of migration failed: Input/output error

And the guest core dump on the src host, but src qemu still works, doesn't hit core dump.



Aihua, can you check again and see if we need to fix the new situation?

mount cmd [1]:
mount -t nfs -o soft,vers=4 $nfs_server_ip:/home/nfs /mnt/xiaohli

With the help of xiaohui, can reproduce the migration fail issue.

Test Env:
 qemu-kvm version:qemu-kvm-7.2.0-10.el9

Test Steps:
1. In both src and dst, mount to nfs server with nfsv4,soft
  (src)#mount -t nfs 10.73.114.14:/mnt/nfs /mnt/nfs/ -o vers=4,soft
  (dst)#mount -t nfs 10.73.114.14:/mnt/nfs /mnt/nfs/ -o vers=4,soft

2.Start guest with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
     -cpu 'Skylake-Server',+kvm_pv_unhalt \
     -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idcE8sYZ"}' \
     -chardev socket,wait=off,server=on,id=chardev_serial0,path=/var/tmp/serial-serial0-20230202-211855-PNb4QIQg \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230202-211855-PNb4QIQg,path=/var/tmp/seabios-20230202-211855-PNb4QIQg,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230202-211855-PNb4QIQg,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:df:c4:ac:9f:d2", "id": "idDHNSFZ", "netdev": "id29g0e4", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=id29g0e4,vhost=on \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 5}' \
     -monitor stdio \

 3. In dst, start guest with qemu cmdline:
     /usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
     -cpu 'Skylake-Server',+kvm_pv_unhalt \
     -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idcE8sYZ"}' \
     -chardev socket,wait=off,server=on,id=chardev_serial0,path=/var/tmp/serial-serial0-20230202-211855-PNb4QIQg \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230202-211855-PNb4QIQg,path=/var/tmp/seabios-20230202-211855-PNb4QIQg,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230202-211855-PNb4QIQg,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:df:c4:ac:9f:d2", "id": "idDHNSFZ", "netdev": "id29g0e4", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=id29g0e4,vhost=on \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 5}' \
     -monitor stdio \
     -incoming tcp:0:5000,server=on,wait=off \

 4. Migrate from src to dst
   {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.25:5000"}}

 5. Check migration status in src
   (qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 3942 ms
expected downtime: 300 ms
setup: 211 ms
transferred ram: 295557 kbytes
throughput: 561.05 mbps
remaining ram: 29665332 kbytes
total ram: 31478472 kbytes
duplicate: 380373 pages
skipped: 0 pages
normal: 72911 pages
normal bytes: 291644 kbytes
dirty sync count: 1
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 22962
precopy ram: 295557 kbytes

 6. When migration is active, stop nfs service in nfs server
   (nfs server)#systemctl stop nfs-server.service

 7. Wait about 30 minutes


 8. After 30 minutes, restore nfs service
    (nfs server)#systemctl start nfs-server.service


Test Result:
 After step7, dst qemu quit with error info
  (qemu) qemu-kvm: load of migration failed: Input/output error

 And Src qemu stopped with block_io_error.
  (qemu) qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)

  {"timestamp": {"seconds": 1678087790, "microseconds": 165579}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678087790, "microseconds": 373502}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678087790, "microseconds": 379422}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678087790, "microseconds": 390489}, "event": "STOP"}
 

  After step8, src qemu still report BLOCK_IO_ERROR at first, then enter running status.
  {"timestamp": {"seconds": 1678088151, "microseconds": 765533}, "event": "RESUME"}
{"timestamp": {"seconds": 1678088328, "microseconds": 501522}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[19]"}}
{"timestamp": {"seconds": 1678088332, "microseconds": 272685}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678088332, "microseconds": 277511}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678088332, "microseconds": 289300}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}} 

  (qemu) info status 
VM status: running


Note:
  When test with nfsv4+hard, not hit this issue, dst qemu not quit and migration continued and successfully 
 completed after nfs service restore.

Test on qemu-kvm-7.2.0-10.el9, can hit the src qemu coredump issue when redo migration after nfs service restored.

Test Env:
 qemu-kvm version:qemu-kvm-7.2.0-10.el9

Test Steps:
 1. Mount src and dst to nfs server with nfsv4,soft mode
  (src)#mount -t nfs 10.73.114.14:/mnt/nfs /mnt/nfs/ -o vers=4,soft
  (dst)#mount -t nfs 10.73.114.14:/mnt/nfs /mnt/nfs/ -o vers=4,soft

 2. Start guest with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
     -cpu 'Skylake-Server',+kvm_pv_unhalt \
     -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idcE8sYZ"}' \
     -chardev socket,wait=off,server=on,id=chardev_serial0,path=/var/tmp/serial-serial0-20230202-211855-PNb4QIQg \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230202-211855-PNb4QIQg,path=/var/tmp/seabios-20230202-211855-PNb4QIQg,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230202-211855-PNb4QIQg,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:df:c4:ac:9f:d2", "id": "idDHNSFZ", "netdev": "id29g0e4", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=id29g0e4,vhost=on \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 5}' \
     -monitor stdio \

 3. In dst, start guest with qemu cmdline:
     /usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
     -cpu 'Skylake-Server',+kvm_pv_unhalt \
     -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idcE8sYZ"}' \
     -chardev socket,wait=off,server=on,id=chardev_serial0,path=/var/tmp/serial-serial0-20230202-211855-PNb4QIQg \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230202-211855-PNb4QIQg,path=/var/tmp/seabios-20230202-211855-PNb4QIQg,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230202-211855-PNb4QIQg,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:df:c4:ac:9f:d2", "id": "idDHNSFZ", "netdev": "id29g0e4", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=id29g0e4,vhost=on \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 5}' \
     -monitor stdio \
     -incoming tcp:0:5000,server=on,wait=off \

 4. Migrate from src to dst
   {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.25:5000"}}

 5. Check migration status in src
   (qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 3942 ms
expected downtime: 300 ms
setup: 211 ms
transferred ram: 295557 kbytes
throughput: 561.05 mbps
remaining ram: 29665332 kbytes
total ram: 31478472 kbytes
duplicate: 380373 pages
skipped: 0 pages
normal: 72911 pages
normal bytes: 291644 kbytes
dirty sync count: 1
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 22962
precopy ram: 295557 kbytes

 6. When migration is active, stop nfs service in nfs server
   (nfs server)#systemctl stop nfs-server.service

 7. Wait about 30 minutes


 8. After 30 minutes, restore nfs service
    (nfs server)#systemctl start nfs-server.service

 9. Re-start dst vm, then re-do migration from src to dst.
   {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.25:5000"}}


Test Result:
 After step7, dst qemu report error info, then after some minutes, quit.
  (qemu) qemu-kvm: load of migration failed: Input/output error

 And Src qemu stopped with block_io_error at first, then it resume to running status after dst qemu quit
  (qemu) qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)

  {"timestamp": {"seconds": 1678087790, "microseconds": 165579}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678087790, "microseconds": 373502}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678087790, "microseconds": 379422}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678087790, "microseconds": 390489}, "event": "STOP"}
{"timestamp": {"seconds": 1678101523, "microseconds": 541554}, "event": "RESUME"}
{"timestamp": {"seconds": 1678101699, "microseconds": 500591}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[19]"}}
{"timestamp": {"seconds": 1678101704, "microseconds": 15716}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678101704, "microseconds": 21521}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678101704, "microseconds": 33320}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 117686}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 495633}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 496681}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 496781}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 496859}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 501416}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678101884, "microseconds": 531889}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678102064, "microseconds": 661581}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}
{"timestamp": {"seconds": 1678102064, "microseconds": 975674}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive_image1", "reason": "Input/output error", "operation": "read", "action": "report"}}

  (src qemu)info status
   VM status: running

  After step9, can migrate from src to dst at first, then dst qemu quit with error, then src qemu core dump.
  (dst qemu) qemu-kvm: Failed to load virtio_pci/modern_queue_state:avail
qemu-kvm: Failed to load virtio_pci/modern_state:vqs
qemu-kvm: Failed to load virtio/extra_state:extra_state
qemu-kvm: Failed to load virtio-net:virtio
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:01.3:00.0/virtio-net'
qemu-kvm: load of migration failed: Input/output error

  (src qemu) qemu-kvm: ../block.c:6738: int bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.
bug.txt: line 44: 504874 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox on -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' ...

  #coredumpctl debug 504874
  Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-22.scope
          Unit: session-22.scope
         Slice: user-0.slice
       Session: 22
     Owner UID: 0 (root)
       Boot ID: bd25cd699f47423e8bcbd376fdbe935c
    Machine ID: 3919555703fd4043b7f3cc2611ad4d18
      Hostname: dell-per740xd-01.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.bd25cd699f47423e8bcbd376fdbe935c.504874.1678103443000000.zst (present)
  Size on Disk: 603.4M
       Message: Process 504874 (qemu-kvm) of user 0 dumped core.
                
                Stack trace of thread 505006:
                #0  0x00007f90350a154c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #1  0x00007f9035054d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007f90350287f3 abort (libc.so.6 + 0x287f3)
                #3  0x00007f903502871b __assert_fail_base.cold (libc.so.6 + 0x2871b)
                #4  0x00007f903504dce6 __assert_fail (libc.so.6 + 0x4dce6)
                #5  0x000055b7bbaa65b4 bdrv_inactivate_recurse (qemu-kvm + 0x7eb5b4)
                #6  0x000055b7bbaa62d8 bdrv_inactivate_all (qemu-kvm + 0x7eb2d8)
                #7  0x000055b7bb7d7c34 qemu_savevm_state_complete_precopy_non_iterable (qemu-kvm + 0x51cc34)
                #8  0x000055b7bb7d81f5 qemu_savevm_state_complete_precopy (qemu-kvm + 0x51d1f5)
                #9  0x000055b7bb7cb3a7 migration_thread (qemu-kvm + 0x5103a7)
                #10 0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #11 0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #12 0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504874:
                #0  0x00007f903509c560 __GI___lll_lock_wait (libc.so.6 + 0x9c560)
                #1  0x00007f90350a2c22 __pthread_mutex_lock.5 (libc.so.6 + 0xa2c22)
                #2  0x000055b7bbc7ce0f qemu_mutex_lock_impl (qemu-kvm + 0x9c1e0f)
                #3  0x000055b7bbc959bd main_loop_wait (qemu-kvm + 0x9da9bd)
                #4  0x000055b7bb79b917 qemu_main_loop (qemu-kvm + 0x4e0917)
                #5  0x000055b7bb62193a qemu_default_main (qemu-kvm + 0x36693a)
                #6  0x00007f903503feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #7  0x00007f903503ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #8  0x000055b7bb621095 _start (qemu-kvm + 0x366095)
                
                Stack trace of thread 504888:
                #0  0x00007f90351429bf __poll (libc.so.6 + 0x1429bf)
                #1  0x00007f903556f49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c)
                #2  0x00007f903551a483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x000055b7bba8f9df iothread_run (qemu-kvm + 0x7d49df)
                #4  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504896:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504889:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504899:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504898:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504900:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504891:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504897:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504893:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504895:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504875:
                #0  0x00007f903503ee5d syscall (libc.so.6 + 0x3ee5d)
                #1  0x000055b7bbc7daff qemu_event_wait (qemu-kvm + 0x9c2aff)
                #2  0x000055b7bbc89c35 call_rcu_thread (qemu-kvm + 0x9cec35)
                #3  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #4  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #5  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504894:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504876:
                #0  0x00007f9035142abe ppoll (libc.so.6 + 0x142abe)
                #1  0x000055b7bbc7a89e fdmon_poll_wait (qemu-kvm + 0x9bf89e)
                #2  0x000055b7bbc79ade aio_poll (qemu-kvm + 0x9beade)
                #3  0x000055b7bba8f9c2 iothread_run (qemu-kvm + 0x7d49c2)
                #4  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504890:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 505011:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eea4 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x9eea4)
                #2  0x000055b7bbc7d4fc qemu_cond_timedwait_ts (qemu-kvm + 0x9c24fc)
                #3  0x000055b7bbc7d4a0 qemu_cond_timedwait_impl (qemu-kvm + 0x9c24a0)
                #4  0x000055b7bbc982e7 worker_thread (qemu-kvm + 0x9dd2e7)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504901:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb79360b qemu_wait_io_event (qemu-kvm + 0x4d860b)
                #4  0x000055b7bba3dde1 kvm_vcpu_thread_fn (qemu-kvm + 0x782de1)
                #5  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #6  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #7  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 504914:
                #0  0x00007f903509c39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x00007f903509eba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x000055b7bbc7d35f qemu_cond_wait_impl (qemu-kvm + 0x9c235f)
                #3  0x000055b7bb657ca6 vnc_worker_thread (qemu-kvm + 0x39cca6)
                #4  0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm + 0x9c2daa)
                #5  0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
                #6  0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64

Hi Kevin, Hanna

Any thoughts on this issue. Can this be fixed.

This issue is related to NFS usages, like mount protocol, mode and qemu error policy.


The different configurations may result in different results. maybe the result is expected due to using nfs v4 .
I open a dedicated bug to discuss what is the valid configuration for NFS usage.

Bug 2178024 - OS data corruption with blk_update_request I/O error when NFS is inaccessible.

Hi aliang, could you please try this with v3+hard connection?

(In reply to Vivek Goyal from comment #20)
> Any thoughts on this issue. Can this be fixed.

I think this specific problem would disappear if bdrv_inactivate_all() re-activated all block nodes on failure. Currently you end up with a mix of already inactivated nodes and still active nodes that haven't been processed (or are the failing node).

(In reply to qing.wang from comment #21)
> This issue is related to NFS usages, like mount protocol, mode and qemu
> error policy.
> 
> 
> The different configurations may result in different results. maybe the
> result is expected due to using nfs v4 .
> I open a dedicated bug to discuss what is the valid configuration for NFS
> usage.
> 
> Bug 2178024 - OS data corruption with blk_update_request I/O error when NFS
> is inaccessible.
> 
> Hi aliang, could you please try this with v3+hard connection?

nfsv3+hard works ok, but nfsv3+soft also hit this issue.

Test on qemu-kvm-6.2.0-1.el9+nfsv3 soft, hit the iroginal src qemu core dump issue(xiaohui reported in description):
  (qemu) qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)
qemu-kvm: ../block/io.c:1991: int bdrv_co_write_req_prepare(BdrvChild *, int64_t, int64_t, BdrvTrackedRequest *, int): Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.
bg.txt: line 40: 929507 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -object memory-backend-ram,size=32212254720,id=mem-machine_mem -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2 -cpu 'Skylake-Server',+kvm_pv_unhalt -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg -mon chardev=qmp_id_catch_monitor,mode=control -device ioport=1285,driver=pvpanic,id=idcE8sYZ ...

Test on  qemu-kvm-6.1.0-8.el9 nfsv3 soft, don't hit this issue.

So, it's a regression bug, and we can hit the src qemu core dump since qemu-kvm-6.2.0-1.el9.

This is what man nfs says about soft/hard mount.

       soft / hard    Determines the recovery behavior of the NFS client after
                      an NFS request times out.  If neither option  is  speci‐
                      fied  (or if the hard option is specified), NFS requests
                      are retried indefinitely.  If the soft option is  speci‐
                      fied, then the NFS client fails an NFS request after re‐
                      trans retransmissions have been sent,  causing  the  NFS
                      client to return an error to the calling application.

                      NB:  A  so-called  "soft"  timeout can cause silent data
                      corruption in certain cases. As such, use the  soft  op‐
                      tion  only  when client responsiveness is more important
                      than data integrity.  Using NFS over TCP  or  increasing
                      the value of the retrans option may mitigate some of the
                      risks of using the soft option.

So client giving error kind of makes sense with "soft" option. But coredump probably is not normal.

(In reply to aihua liang from comment #18)
> Test on qemu-kvm-7.2.0-10.el9, can hit the src qemu coredump issue when redo
> migration after nfs service restored.
> 

Making sure I understand the setup:

> Test Env:
>  qemu-kvm version:qemu-kvm-7.2.0-10.el9
> 
> Test Steps:
>  1. Mount src and dst to nfs server with nfsv4,soft mode
>   (src)#mount -t nfs 10.73.114.14:/mnt/nfs /mnt/nfs/ -o vers=4,soft
>   (dst)#mount -t nfs 10.73.114.14:/mnt/nfs /mnt/nfs/ -o vers=4,soft

So both src and dst are seeing the same NFS server...

> 
>  2. Start guest with qemu cmdline:

>      -blockdev '{"node-name": "file_ovmf_vars", "driver": "file",
> "filename":
> "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd",

>     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \

The source is referencing two files on the server...

> 
>  3. In dst, start guest with qemu cmdline:
>      /usr/libexec/qemu-kvm \

>      -blockdev '{"node-name": "file_ovmf_vars", "driver": "file",
> "filename":
> "/mnt/nfs/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd",

>      -blockdev '{"node-name": "file_image1", "driver": "file",
> "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename":
> "/mnt/nfs/rhel920-64-virtio-scsi.qcow2", "cache": {"direct": true,
> "no-flush": false}}' \
>      -blockdev '{"node-name": "drive_image1", "driver": "qcow2",
> "read-only": false, "cache": {"direct": true, "no-flush": false}, "file":
> "file_image1"}' \

and the dest is likewise set to see the same files,

>  6. When migration is active, stop nfs service in nfs server
>    (nfs server)#systemctl stop nfs-server.service
> 
>  7. Wait about 30 minutes
> 

and you are killing NFS for both src and dst (both sides should see I/O failures after hitting timeouts due to the 30 minutes waiting after NFS is forcefully stopped).

> 
>  8. After 30 minutes, restore nfs service
>     (nfs server)#systemctl start nfs-server.service
> 
>  9. Re-start dst vm, then re-do migration from src to dst.
>    {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.25:5000"}}
> 
> 
> Test Result:
>  After step7, dst qemu report error info, then after some minutes, quit.
>   (qemu) qemu-kvm: load of migration failed: Input/output error

The dst giving up on the I/O error makes sense (it couldn't access the NFS server, so the migration fails), which leaves things back to the src to try to recover when NFS comes back up.

> 
>  And Src qemu stopped with block_io_error at first, then it resume to
> running status after dst qemu quit
>   (qemu) qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable:
> bdrv_inactivate_all() failed (-1)

The bdrv_inactivate_all() failing on the source makes sense - there was I/O that could not be flushed to NFS in the time before NFS came back up.  Then...

> 
>   {"timestamp": {"seconds": 1678087790, "microseconds": 165579}, "event":
> "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name":
> "drive_image1", "reason": "Input/output error", "operation": "read",
> "action": "report"}}
> {"timestamp": {"seconds": 1678087790, "microseconds": 373502}, "event":
> "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name":
> "drive_image1", "reason": "Input/output error", "operation": "write",
> "action": "report"}}
> {"timestamp": {"seconds": 1678087790, "microseconds": 379422}, "event":
> "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name":
> "drive_image1", "reason": "Input/output error", "operation": "write",
> "action": "report"}}
> {"timestamp": {"seconds": 1678087790, "microseconds": 390489}, "event":
> "STOP"}
> {"timestamp": {"seconds": 1678101523, "microseconds": 541554}, "event":
> "RESUME"}

The gap in timestamps here is 228 minutes, although that fits with your 'wait at least 30 minutes'

> {"timestamp": {"seconds": 1678101699, "microseconds": 500591}, "event":
> "RTC_CHANGE", "data": {"offset": 0, "qom-path":
> "/machine/unattached/device[19]"}}
> {"timestamp": {"seconds": 1678101704, "microseconds": 15716}, "event":
> "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name":
> "drive_image1", "reason": "Input/output error", "operation": "write",
> "action": "report"}}

It looks like after you resume the src, there are still a few I/O errors pending from before when NFS went down that still flush through the system.  Are you waiting for those to all clear up before re-attempting the migration in step 9?

> 
>   After step9, can migrate from src to dst at first, then dst qemu quit with
> error, then src qemu core dump.
>   (dst qemu) qemu-kvm: Failed to load virtio_pci/modern_queue_state:avail
> qemu-kvm: Failed to load virtio_pci/modern_state:vqs
> qemu-kvm: Failed to load virtio/extra_state:extra_state
> qemu-kvm: Failed to load virtio-net:virtio
> qemu-kvm: error while loading state for instance 0x0 of device
> '0000:00:01.3:00.0/virtio-net'
> qemu-kvm: load of migration failed: Input/output error
> 
>   (src qemu) qemu-kvm: ../block.c:6738: int
> bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags &
> BDRV_O_INACTIVE)' failed.
> bug.txt: line 44: 504874 Aborted                 (core dumped)

And is this core dump always happening after the second migrate attempt?  I'm wondering if something in the block layer is recording the I/O failure in the first attempt (perhaps marking the device inactive when it failed to access it), but we aren't noticing it until the second attempt tries to once again inactivate devices without realizing that at least one device was already left marked inactive from the first attempt.

>                 Stack trace of thread 505006:
>                 #0  0x00007f90350a154c __pthread_kill_implementation
> (libc.so.6 + 0xa154c)
>                 #1  0x00007f9035054d46 raise (libc.so.6 + 0x54d46)
>                 #2  0x00007f90350287f3 abort (libc.so.6 + 0x287f3)
>                 #3  0x00007f903502871b __assert_fail_base.cold (libc.so.6 +
> 0x2871b)
>                 #4  0x00007f903504dce6 __assert_fail (libc.so.6 + 0x4dce6)
>                 #5  0x000055b7bbaa65b4 bdrv_inactivate_recurse (qemu-kvm +
> 0x7eb5b4)
>                 #6  0x000055b7bbaa62d8 bdrv_inactivate_all (qemu-kvm +
> 0x7eb2d8)
>                 #7  0x000055b7bb7d7c34
> qemu_savevm_state_complete_precopy_non_iterable (qemu-kvm + 0x51cc34)
>                 #8  0x000055b7bb7d81f5 qemu_savevm_state_complete_precopy
> (qemu-kvm + 0x51d1f5)
>                 #9  0x000055b7bb7cb3a7 migration_thread (qemu-kvm + 0x5103a7)
>                 #10 0x000055b7bbc7ddaa qemu_thread_start (qemu-kvm +
> 0x9c2daa)
>                 #11 0x00007f903509f802 start_thread (libc.so.6 + 0x9f802)
>                 #12 0x00007f903503f450 __clone3 (libc.so.6 + 0x3f450)
>                 

Can the NFS server be on the same machine as the src or dst, or are you using 3 machines (NFS separate from the src and dst)?

(In reply to Eric Blake from comment #28)
> >   (qemu) qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable:
> > bdrv_inactivate_all() failed (-1)
> 
> The bdrv_inactivate_all() failing on the source makes sense - there was I/O
> that could not be flushed to NFS in the time before NFS came back up. 

Note that bdrv_inactivate_all() failing means that you probably now have some block nodes inactivated, while other nodes are still active. (Unless you already got the failure for the very first node.)

This is the core problem here, I believe.

> > {"timestamp": {"seconds": 1678087790, "microseconds": 390489}, "event":
> > "STOP"}
> > {"timestamp": {"seconds": 1678101523, "microseconds": 541554}, "event":
> > "RESUME"}
> 
> The gap in timestamps here is 228 minutes, although that fits with your
> 'wait at least 30 minutes'

The RESUME is actually one thing that could save us because when resuming the VM, all block nodes are supposed to be activated again.

However, we only call bdrv_activate_all() in qmp_cont(), which we assumed would be how VMs are resumed after migration failed. As we still run into the error later, it seems that this used a different code path. Maybe block device activation should be in vm_prepare_start() instead to cover all callers.

> >   (src qemu) qemu-kvm: ../block.c:6738: int
> > bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags &
> > BDRV_O_INACTIVE)' failed.
> > bug.txt: line 44: 504874 Aborted                 (core dumped)
> 
> And is this core dump always happening after the second migrate attempt? 
> I'm wondering if something in the block layer is recording the I/O failure
> in the first attempt (perhaps marking the device inactive when it failed to
> access it), but we aren't noticing it until the second attempt tries to once
> again inactivate devices without realizing that at least one device was
> already left marked inactive from the first attempt.

Yes, that's what we seem to hit here. A write request would also have resulted in an assertion failure because you can't write to inactive block nodes, but trying to inactivate an already inactive image (as done by the second migration) does, too.

https://lists.gnu.org/archive/html/qemu-devel/2023-04/msg01634.html posted upstream for discussion.  I'm not sure whether patching just migration is sufficient, or whether we want to go whole-hog and teach vm_prepare_start() to also mandate a successful bdrv_activate_all().

Upstream pull request issued; I'm now working on backporting this downstream:
https://lists.gnu.org/archive/html/qemu-devel/2023-04/msg03157.html

Have set the ITM, thanks.

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Test on qemu-kvm-8.0.0-4.el9, the core dump issue has been resolved.

Test Steps:
1.Mount src and dst host to nfs server with nfs,soft
  #mount 10.73.196.25:/mnt/nfs /mnt/nfs -o vers=4,soft

2.Start src guest with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -machine q35,memory-backend=mem-machine_mem \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
     -cpu 'Skylake-Server',+kvm_pv_unhalt \
     -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idcE8sYZ"}' \
     -chardev socket,wait=off,server=on,id=chardev_serial0,path=/var/tmp/serial-serial0-20230202-211855-PNb4QIQg \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230202-211855-PNb4QIQg,path=/var/tmp/seabios-20230202-211855-PNb4QIQg,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230202-211855-PNb4QIQg,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel930-64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:df:c4:ac:9f:d2", "id": "idDHNSFZ", "netdev": "id29g0e4", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=id29g0e4,vhost=on \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 5}' \
     -monitor stdio \
 
3.Check block info in src
  (qemu)info block
drive_image1: /mnt/nfs/rhel930-64-virtio.qcow2 (qcow2)
    Attached to:      image1
    Cache mode:       writeback, direct

4.Start dst guest with qemu cmdline:
  /usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -machine q35,memory-backend=mem-machine_mem \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 30720 \
     -object '{"qom-type": "memory-backend-ram", "size": 32212254720, "id": "mem-machine_mem"}'  \
     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
     -cpu 'Skylake-Server',+kvm_pv_unhalt \
     -chardev socket,wait=off,server=on,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,wait=off,server=on,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20230202-211855-PNb4QIQg  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idcE8sYZ"}' \
     -chardev socket,wait=off,server=on,id=chardev_serial0,path=/var/tmp/serial-serial0-20230202-211855-PNb4QIQg \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230202-211855-PNb4QIQg,path=/var/tmp/seabios-20230202-211855-PNb4QIQg,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230202-211855-PNb4QIQg,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/mnt/nfs/rhel930-64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:df:c4:ac:9f:d2", "id": "idDHNSFZ", "netdev": "id29g0e4", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=id29g0e4,vhost=on \
     -vnc :0  \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x3", "chassis": 5}' \
     -monitor stdio \
     -incoming tcp:0:5000,server=on,wait=off \

5.Migrate from src to dst
  {"execute": "migrate","arguments":{"uri": "tcp:10.73.114.14:5000"}}

6.During migration,stop nfs server
  #(nfs-server)systemctl stop nfs-server.service

7.Wait for 30 minutes

8.Start nfs server, wait a moment, check src block info
  #(nfs-server)systemctl start nfs-server.service
  (qemu)info block

Actual Steps:
 After step7, dst quit with error.
  (qemu) qemu-kvm: load of migration failed: Input/output error
 Src return to running status with error:
  (qemu)qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)
qemu-kvm: Could not reopen qcow2 layer: Could not read qcow2 header: Input/output error
   {"timestamp": {"seconds": 1684923472, "microseconds": 653690}, "event": "STOP"}
{"timestamp": {"seconds": 1684924014, "microseconds": 238697}, "event": "RESUME"}

 After step8, src qemu can't connect to nfs server automatically, so no block info displayed.
 (qemu) info block
image1: [not inserted]
    Attached to:      image1

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368