Bug 1738451 - qemu on src host core dump after set multifd-channels and do migration twice (first migration execute migrate_cancel)
Summary: qemu on src host core dump after set multifd-channels and do migration twice ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Juan Quintela
QA Contact: Li Xiaohui
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-07 08:24 UTC by Li Xiaohui
Modified: 2020-05-05 09:47 UTC (History)
10 users (show)

Fixed In Version: qemu-kvm-4.2.0-14.module+el8.2.0+5995+d02a4eeb
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-05 09:47:43 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Li Xiaohui 2019-08-07 08:24:08 UTC
Description of problem:
set multifd-channels 4, do multifd migration, cancel during migration, then change multifd-channels 2, do multifd migration again, qemu on src host will core dump


Version-Release number of selected component (if applicable):
src&dst host info: kernel-4.18.0-125.el8.x86_64 & qemu-4.1.0-rc3
guest info: kernel-4.18.0-128.el8.x86_64


How reproducible:
2/2


Steps to Reproduce:
1.boot a guest on src host, run a stress in guest:
# stressapptest -M 4500M -s 1000000
Notes: there is 5G free mem in guest, so mem is enough
2.boot a guest on dst host with "-incoming tcp:0:4444"
3.on src&dst host, enable multifd and set multifd-channels 4
4.set speed 2GBps in src qemu
5.do multifd migration from src to dst host, during migration, execute migrate_cancel, cancel migration successfully:
(qemu) migrate -d tcp:192.168.11.22:4444
(qemu) migrate_cancel
(qemu) qemu-kvm: multifd_send_pages: channel 3 has already quit!
qemu-kvm: multifd_send_pages: channel 3 has already quit!
qemu-kvm: multifd_send_sync_main: multifd_send_pages fail
qemu-kvm: Unable to write to socket: Broken pipe
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off multifd: on dirty-bitmaps: off postcopy-blocktime: off late-block-activate: off x-ignore-shared: off 
Migration status: cancelled
total time: 0 milliseconds
6.boot a guest on dst host again, and enable multifd in dst qemu
7.set multifd-channels 2 on src&dst host
8.do multifd migration again


Actual results:
after step7, qemu in src host will core dump:
(qemu) migrate -d tcp:192.168.11.22:4444
(qemu) qemu-kvm: util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed.
./start2.sh: line 22: 10906 Aborted                 (core dumped) /usr/libexec/qemu-kvm -enable-kvm -nodefaults -machine q35 -m 8G -smp 8 -cpu Haswell-noTSX-IBRS -name debug-threads=on -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 -device virtio-scsi-pci,id=scsi0,bus=pcie.0-root-port-2 -drive file=/mnt/nfs/rhel810-scsi-0729-2.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,media=disk,cache=none,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,queues=4 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=18:66:da:5e:c2:3c,bus=pcie.0-root-port-3,vectors=10,mq=on -qmp tcp:0:3333,server,nowait -vnc :1 -device VGA -monitor stdio


Expected results:
migration go on, and finish successfully. 


Additional info:
sometimes, when execute step1~5, qemu on dst host will core dump, error like src qemu core dump

Comment 2 Juan Quintela 2019-08-21 11:19:23 UTC
Working on given better error results in cases like this.

Comment 3 Juan Quintela 2019-09-11 09:58:41 UTC
Can you reproduce with lastest rhel8.1 qemu-kvm?
I am not able to reprouce with ether::
 * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4
 * rhel8.1     Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a

I am trying to reproduce with:
- multifd on
- multifd-channels=10
- migrate_set_downtime 1ms
- migrate_set_speed 1k
- stress the test
- migrate -d <whatever>
- migrate_cancel
  destination stops as expected, and source continues
- lanunch destination again
- migrate_set_downtime 1000ms
- migrate_set_speed 1G
  (with that I on my hardware it converges, vary it or stop stressing the guest)
- migrate -d <whatever>

And it migrates correctly.
Probably the serialitation patches from bugzilla  1726898 helped here.
Later, Juan.

Comment 4 juzhang 2019-09-15 06:52:01 UTC
(In reply to Juan Quintela from comment #3)
> Can you reproduce with lastest rhel8.1 qemu-kvm?
> I am not able to reprouce with ether::
>  * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4
>  * rhel8.1     Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a
> 
> I am trying to reproduce with:
> - multifd on
> - multifd-channels=10
> - migrate_set_downtime 1ms
> - migrate_set_speed 1k
> - stress the test
> - migrate -d <whatever>
> - migrate_cancel
>   destination stops as expected, and source continues
> - lanunch destination again
> - migrate_set_downtime 1000ms
> - migrate_set_speed 1G
>   (with that I on my hardware it converges, vary it or stop stressing the
> guest)
> - migrate -d <whatever>
> 
> And it migrates correctly.
> Probably the serialitation patches from bugzilla  1726898 helped here.
> Later, Juan.

Hi Xiaohui,

Can you have a try?

Best regards,

Junyi

Comment 5 Li Xiaohui 2019-09-15 12:59:41 UTC
(In reply to juzhang from comment #4)
> (In reply to Juan Quintela from comment #3)
> > Can you reproduce with lastest rhel8.1 qemu-kvm?
> > I am not able to reprouce with ether::
> >  * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4
> >  * rhel8.1     Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a
> > 
> > I am trying to reproduce with:
> > - multifd on
> > - multifd-channels=10
> > - migrate_set_downtime 1ms
> > - migrate_set_speed 1k
> > - stress the test
> > - migrate -d <whatever>
> > - migrate_cancel
> >   destination stops as expected, and source continues
> > - lanunch destination again
> > - migrate_set_downtime 1000ms
> > - migrate_set_speed 1G
> >   (with that I on my hardware it converges, vary it or stop stressing the
> > guest)
> > - migrate -d <whatever>
> > 
> > And it migrates correctly.
> > Probably the serialitation patches from bugzilla  1726898 helped here.
> > Later, Juan.
> 
> Hi Xiaohui,
> 
> Can you have a try?
> 
> Best regards,
> 
> Junyi

Yes, will try when hosts are available

Comment 6 Li Xiaohui 2019-09-23 08:04:12 UTC
(In reply to Juan Quintela from comment #3)
> Can you reproduce with lastest rhel8.1 qemu-kvm?
> I am not able to reprouce with ether::
>  * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4
>  * rhel8.1     Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a
> 
> I am trying to reproduce with:
> - multifd on
> - multifd-channels=10
> - migrate_set_downtime 1ms
> - migrate_set_speed 1k
> - stress the test
> - migrate -d <whatever>
> - migrate_cancel
>   destination stops as expected, and source continues
> - lanunch destination again
> - migrate_set_downtime 1000ms
> - migrate_set_speed 1G
>   (with that I on my hardware it converges, vary it or stop stressing the
> guest)
Maybe there is lack of "change multifd-channels" setting after migrate again from src to dst host(According to Comment 0 's step7)
> - migrate -d <whatever>
> 
> And it migrates correctly.
> Probably the serialitation patches from bugzilla  1726898 helped here.
> Later, Juan.

Hi Juan, 
I can reproduce this issue on the latest host(kernel-4.18.0-145.el8.x86_64&qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64),
test steps like Comment 0. 
Notes: reproduce 1 time after tried 3 times.

(1)in hmp, get error:
(qemu) migrate -d tcp:10.73.73.87:4444
(qemu) qemu-kvm: util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed.

(qemu) ./start3.sh: line 22:  8642 Aborted                 (core dumped) /usr/libexec/qemu-kvm -enable-kvm ...
(2)in gdb mode, get error:
# gdb -p 8642
...
Thread 104 "live_migration" received signal SIGABRT, Aborted.
[Switching to Thread 0x7f30464f9700 (LWP 8860)]
0x00007f3294f6a8df in raise () from /lib64/libc.so.6
(gdb) c
Continuing.
[Thread 0x7f3045cf8700 (LWP 8862) exited]
[Thread 0x7f3023fff700 (LWP 8861) exited]
...

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.

Comment 7 Li Xiaohui 2019-09-23 08:09:25 UTC
Confirm again about change multifd-channels in Comment 0 's step 7, if test like Comment 0, but without step 7(just set multifd-channels to 4, let it's same with src host qemu),  qemu on src host will be stuck(guest on src host is stuck, too):
(qemu) migrate -d tcp:10.73.73.87:4444
(qemu) qemu-kvm: multifd_send_pages: channel 1 has already quit!
qemu-kvm: multifd_send_pages: channel 1 has already quit!
qemu-kvm: multifd_send_sync_main: multifd_send_pages fail

Comment 8 Li Xiaohui 2019-09-23 08:11:19 UTC
(In reply to Li Xiaohui from comment #7)
> Confirm again about change multifd-channels in Comment 0 's step 7, if test
> like Comment 0, but without step 7(just set multifd-channels to 4, let it's
> same with src host qemu),  qemu on src host will be stuck(guest on src host
> is stuck, too):
this issue will reproduce 1/5 times: qemu&guest on src host will be stuck 
> (qemu) migrate -d tcp:10.73.73.87:4444
> (qemu) qemu-kvm: multifd_send_pages: channel 1 has already quit!
> qemu-kvm: multifd_send_pages: channel 1 has already quit!
> qemu-kvm: multifd_send_sync_main: multifd_send_pages fail

Comment 10 Juan Quintela 2019-10-11 16:45:31 UTC
I think that the remaining problem is the same that bugzilla 1726898.  See comment #24 there.

Could you retest adding:

--global migration.multifd-channels=10

to the command line (the destination side is where it is necesary).  10 should be enough for any number of channels that you use, but if it fails, just put there the maximum number of channels that you are going to use.

Comment 11 Li Xiaohui 2019-10-12 05:55:49 UTC
(In reply to Juan Quintela from comment #10)
> I think that the remaining problem is the same that bugzilla 1726898.  See
> comment #24 there.
> 
> Could you retest adding:
> 
> --global migration.multifd-channels=10
> 
> to the command line (the destination side is where it is necesary).  10
> should be enough for any number of channels that you use, but if it fails,
> just put there the maximum number of channels that you are going to use.

ok, I will try later

Comment 12 Li Xiaohui 2019-10-15 06:14:50 UTC
(In reply to Juan Quintela from comment #10)
> I think that the remaining problem is the same that bugzilla 1726898.  See
> comment #24 there.
> 
> Could you retest adding:
> 
> --global migration.multifd-channels=10
> 
> to the command line (the destination side is where it is necesary).  10
> should be enough for any number of channels that you use, but if it fails,
> just put there the maximum number of channels that you are going to use.

Hi Juan,
When I use "-incoming defer" in dst qemu to retest this bz, could reproduce it by low reproduction rate(1/10 or 1/20, not sure).
Test steps like:
1.start a guest on src host;
2.start a guest on dst host with "-incoming defer";
3.set migration speed to 1M in src qemu;
4.enable multifd and set multifd-channels to 4 on src&dst host;
5.set migration listening port in dst qemu:
(qemu) migrate_incoming tcp:10.66.8.208:4444
6. migrate guest from src to dst host;
(qemu) migrate -d tcp:10.66.8.208:4444
7.during migration is active, cancel migration in src qemu:
(qemu) migrate_cancel
8.restart a guest on dst host with "-incoming defer";
9.set multifd-channels to 2 in src qemu;
10.enable multifd and set multifd-channels to 2 in dst qemu;
11.set migration listening port in dst qemu:
(qemu) migrate_incoming tcp:10.66.8.208:4444
12.migrate guest from src to dst host;
(qemu) migrate -d tcp:10.66.8.208:4444

Comment 13 Juan Quintela 2019-10-16 11:41:34 UTC
ok, thanks.  Trying to reproduce with your last recipe.

Comment 14 Juan Quintela 2019-10-29 15:10:23 UTC
Hi Xiaohli

As said on bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1726898, I think that the problem is fixed.

Could you confirm:
a- you are using -incoming defer on both launch of migration
b- to be sure that the first destination qemu has died, could you try a different port for each case?
c - if it still fails, could you give me any logs/output.

Thanks, Juan.

PD. Testing a similar way with Autotest(Lukkas) I was able to make it survive 100 times,  it is not the same test, because I don't know yet how to change the test.

Comment 15 Li Xiaohui 2019-11-18 02:50:22 UTC
Hi Juan,
reproduce bz as Comment 12 's test steps on hosts(kernel-4.18.0-148.el8.x86_64&qemu-img-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64), the qemu core dump info like followings:
(gdb) t a a bt full

Thread 15 (Thread 0x7f71645db700 (LWP 12828)):
#0  0x00007f716b3a4250 in nanosleep () at /lib64/libpthread.so.0
#1  0x00007f716feae4d7 in g_usleep () at /lib64/libglib-2.0.so.0
#2  0x0000559f128cd848 in call_rcu_thread (opaque=<optimized out>)
    at util/rcu.c:252
        tries = 2
        n = 8
        node = <optimized out>
#3  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f138305e0)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f138305e0
        start_routine = 0x559f128cd7e0 <call_rcu_thread>
        arg = 0x0
        r = <optimized out>
#4  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#5  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 14 (Thread 0x7f6fa94f7700 (LWP 13062)):
#0  0x00007f716b3a2e56 in do_futex_wait.constprop () at /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x00007f716b3a2f48 in __new_sem_wait_slow.constprop.0 ()
    at /lib64/libpthread.so.0
#2  0x0000559f128bbb34 in qemu_sem_wait (sem=sem@entry=0x559f13baca20)
    at util/qemu-thread-posix.c:319
        rc = <optimized out>
        __PRETTY_FUNCTION__ = "qemu_sem_wait"
        __func__ = "qemu_sem_wait"
#3  0x0000559f125acd08 in multifd_send_thread (opaque=0x559f13baca00)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:1115
        p = 0x559f13baca00
        local_err = 0x0
        ret = 0
        flags = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f1486ef40)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f1486ef40
        start_routine = 0x559f125acc60 <multifd_send_thread>
        arg = 0x559f13baca00
        r = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 13 (Thread 0x7f7160862700 (LWP 12844)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f13a31d60, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7160861680, reg_save_area = 0x7f7160861640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a31d60)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f13a31d60
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a54b50)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13a54b50
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f13a31d60
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 12 (Thread 0x7f7143fff700 (LWP 12845)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f13a55390, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
--Type <RET> for more, q to quit, c to continue without paging--
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7143ffe680, reg_save_area = 0x7f7143ffe640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a55390)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f13a55390
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a78180)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, --Type <RET> for more, q to quit, c to continue without paging--
__cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13a78180
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f13a55390
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 11 (Thread 0x7f71437fe700 (LWP 12846)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f13a789c0, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f71437fd680, reg_save_area = 0x7f71437fd640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
--Type <RET> for more, q to quit, c to continue without paging--
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a789c0)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f13a789c0
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a9b7b0)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13a9b7b0
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f13a789c0
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 10 (Thread 0x7f6f89ffb700 (LWP 13057)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f716b3a3072 in do_futex_wait () at /lib64/libpthread.so.0
#1  0x00007f716b3a3183 in __new_sem_wait_slow () at /lib64/libpthread.so.0
#2  0x0000559f128bba7f in qemu_sem_timedwait
    (sem=sem@entry=0x559f13925788, ms=ms@entry=10000)
    at util/qemu-thread-posix.c:289
        rc = <optimized out>
        ts = {tv_sec = 1574045029, tv_nsec = 189319000}
        __PRETTY_FUNCTION__ = "qemu_sem_timedwait"
        __func__ = "qemu_sem_timedwait"
#3  0x0000559f128b67c4 in worker_thread (opaque=0x559f13925710)
    at util/thread-pool.c:91
        req = <optimized out>
        ret = <optimized out>
        pool = 0x559f13925710
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a31340)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13a31340
        start_routine = 0x559f128b6730 <worker_thread>
        arg = 0x559f13925710
        r = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 9 (Thread 0x7f7140bff700 (LWP 12852)):
#0  0x00007f716b3a047c in pthread_cond_wait@@GLIBC_2.3.2 ()
    at /lib64/libpthread.so.0
#1  0x0000559f128bb86d in qemu_cond_wait_impl
    (cond=<optimized out>, mutex=0x559f14458418, file=0x559f12a37c37 "ui/vnc-jobs.c", line=214) at util/qemu-thread-posix.c:161
        err = <optimized out>
        __PRETTY_FUNCTION__ = "qemu_cond_wait_impl"
        __func__ = "qemu_cond_wait_impl"

#2  0x0000559f127e4d71 in vnc_worker_thread_loop
    (queue=queue@entry=0x559f144583e0) at ui/vnc-jobs.c:214
        _f = <optimized out>
        job = <optimized out>
        entry = <optimized out>
        tmp = <optimized out>
        vs = Python Exception <class 'gdb.error'> value of type `VncState' requires 75456 bytes, which is more than max-value-size: 
#3  0x0000559f127e5330 in vnc_worker_thread (arg=0x559f144583e0)
    at ui/vnc-jobs.c:324
        queue = 0x559f144583e0
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13bde120)
    at util/qemu-thread-posix.c:502
--Type <RET> for more, q to quit, c to continue without paging--
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13bde120
        start_routine = 0x559f127e5310 <vnc_worker_thread>
        arg = 0x559f144583e0
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 8 (Thread 0x7f6fa9cf8700 (LWP 13063)):
#0  0x00007f716b3a2e56 in do_futex_wait.constprop () at /lib64/libpthread.so.0
#1  0x00007f716b3a2f48 in __new_sem_wait_slow.constprop.0 ()
    at /lib64/libpthread.so.0
#2  0x0000559f128bbb34 in qemu_sem_wait (sem=sem@entry=0x559f13bacad8)
    at util/qemu-thread-posix.c:319
        rc = <optimized out>
        __PRETTY_FUNCTION__ = "qemu_sem_wait"
        __func__ = "qemu_sem_wait"
#3  0x0000559f125acd08 in multifd_send_thread (opaque=0x559f13bacab8)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:1115
        p = 0x559f13bacab8

--Type <RET> for more, q to quit, c to continue without paging--

        local_err = 0x0
        ret = 0
        flags = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f14809a90)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f14809a90
        start_routine = 0x559f125acc60 <multifd_send_thread>
        arg = 0x559f13bacab8
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 7 (Thread 0x7f7162866700 (LWP 12840)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f13979730, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7162865680, reg_save_area = 0x7f7162865640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13979730)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f13979730
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f1399c760)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f1399c760
--Type <RET> for more, q to quit, c to continue without paging--
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f13979730
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 6 (Thread 0x7f7161063700 (LWP 12843)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f13a0dfc0, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7161062680, reg_save_area = 0x7f7161062640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
--Type <RET> for more, q to quit, c to continue without paging--
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a0dfc0)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f13a0dfc0
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a31550)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13a31550
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f13a0dfc0
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7f7161864700 (LWP 12842)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
--Type <RET> for more, q to quit, c to continue without paging--
    (cpu=cpu@entry=0x559f139eabc0, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7161863680, reg_save_area = 0x7f7161863640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f139eabc0)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f139eabc0
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a0d780)
--Type <RET> for more, q to quit, c to continue without paging--
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13a0d780
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f139eabc0
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7f717079af00 (LWP 12827)):
#0  0x00007f716b0c0026 in ppoll () at /lib64/libc.so.6
#1  0x0000559f128b7175 in ppoll
    (__ss=0x0, __timeout=0x7ffcbd8a48c0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
        ts = {tv_sec = 0, tv_nsec = 56621687}
        tvsec = <optimized out>
#2  0x0000559f128b7175 in qemu_poll_ns
    (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=56621687)
    at util/qemu-timer.c:334
        ts = {tv_sec = 0, tv_nsec = 56621687}
        tvsec = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x0000559f128b8075 in os_host_main_loop_wait (timeout=56621687)
    at util/main-loop.c:236


        context = 0x559f139267f0
        ret = <optimized out>
        mlpoll = {state = 0, timeout = 4294967295, pollfds = 0x559f13925810}
        ret = <optimized out>
        timeout_ns = <optimized out>
#4  0x0000559f128b8075 in main_loop_wait (nonblocking=<optimized out>)
    at util/main-loop.c:517
        mlpoll = {state = 0, timeout = 4294967295, pollfds = 0x559f13925810}
        ret = <optimized out>
        timeout_ns = <optimized out>
#5  0x0000559f126a1169 in main_loop () at vl.c:1809
#6  0x0000559f12550fd3 in main
    (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4506
        i = <optimized out>
        snapshot = 0
        linux_boot = <optimized out>
        initrd_filename = 0x0
        kernel_filename = 0x0
        kernel_cmdline = <optimized out>
        boot_order = 0x559f1296facc "cad"
--Type <RET> for more, q to quit, c to continue without paging--
        boot_once = <optimized out>
        ds = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        accel_opts = 0x0
        olist = <optimized out>
        optind = 35
        optarg = 0x7ffcbd8a54f4 "stdio"
        loadvm = 0x0
        machine_class = 0x559f1391c120
        cpu_option = 0x7ffcbd8a51d0 "Skylake-Client,+kvm_pv_unhalt"
        vga_model = 0x0
        qtest_chrdev = 0x0
        qtest_log = 0x0
        incoming = 0x0
        userconfig = <optimized out>
        nographic = false
        display_remote = <optimized out>
        log_mask = <optimized out>
        log_file = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        ram_slots = 0
        vmstate_dump_file = 0x0
        main_loop_err = 0x0
        err = 0x0
        list_data_dirs = false
        dir = <optimized out>
        bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffcbd8a4a50}
        __func__ = "main"

Thread 3 (Thread 0x7f7162065700 (LWP 12841)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f139c6a90, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7162064680, reg_save_area = 0x7f7162064640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
--Type <RET> for more, q to quit, c to continue without paging--
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f139c6a90)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
        cpu = 0x559f139c6a90
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f139ea380)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f139ea380
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f139c6a90
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
Thread 2 (Thread 0x7f7142ffd700 (LWP 12847)):
#0  0x00007f716b0c184b in ioctl () at /lib64/libc.so.6
#1  0x0000559f125b7259 in kvm_vcpu_ioctl
    (cpu=cpu@entry=0x559f13a9bff0, type=type@entry=44672)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411
        ret = <optimized out>
        arg = 0x0
        ap = 
            {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7142ffc680, reg_save_area = 0x7f7142ffc640}}
#2  0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248
        attrs = 
          {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#3  0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a9bff0)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285
--Type <RET> for more, q to quit, c to continue without paging--
        cpu = 0x559f13a9bff0
        r = <optimized out>
#4  0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13abede0)
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13abede0
        start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn>
        arg = 0x559f13a9bff0
        r = <optimized out>
#5  0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#6  0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7f6f8b7fe700 (LWP 13061)):
#0  0x00007f716b00682f in raise () at /lib64/libc.so.6
#1  0x00007f716aff0c45 in abort () at /lib64/libc.so.6
#2  0x00007f716aff0b19 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
#3  0x00007f716affede6 in .annobin_assert.c_end () at /lib64/libc.so.6
#4  0x0000559f128bb5c6 in qemu_mutex_lock_impl
    (mutex=<optimized out>, file=<optimized out>, line=<optimized out>)
    at util/qemu-thread-posix.c:64
        err = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        __PRETTY_FUNCTION__ = "qemu_mutex_lock_impl"
        __func__ = "qemu_mutex_lock_impl"
#5  0x0000559f125ac18b in multifd_send_pages ()
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:937
        _f = <optimized out>
        i = 2
        next_channel = 2
        p = 0x559f13bacb70
        pages = 0x559f1395d340
        transferred = <optimized out>
        __func__ = "multifd_send_pages"
#6  0x0000559f125b11f2 in multifd_queue_page
    (offset=<optimized out>, block=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:984
        pages = 0x559f1395d340
        offset = 1150976
        res = -1
        tmppages = <optimized out>
        pages = 0
        pagesize_bits = <optimized out>
        pss = 
--Type <RET> for more, q to quit, c to continue without paging--
          {block = <optimized out>, page = 281, complete_round = <optimized out>}
        pages = 0
        again = true
        found = <optimized out>
#7  0x0000559f125b11f2 in ram_save_multifd_page
    (rs=<optimized out>, offset=<optimized out>, block=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:2082
        offset = 1150976
        res = -1
        tmppages = <optimized out>
        pages = 0
        pagesize_bits = <optimized out>
        pss = 
          {block = <optimized out>, page = 281, complete_round = <optimized out>}
        pages = 0
        again = true
        found = <optimized out>
#8  0x0000559f125b11f2 in ram_save_target_page
    (last_stage=false, pss=<synthetic pointer>, rs=0x7f6fa0000b80)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migra--Type <RET> for more, q to quit, c to continue without paging--
tion/ram.c:2563
        offset = 1150976
        res = -1
        tmppages = <optimized out>
        pages = 0
        pagesize_bits = <optimized out>
        pss = 
          {block = <optimized out>, page = 281, complete_round = <optimized out>}
        pages = 0
        again = true
        found = <optimized out>
#9  0x0000559f125b11f2 in ram_save_host_page
    (last_stage=<optimized out>, pss=<synthetic pointer>, rs=0x7f6fa0000b80)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:2606
        tmppages = <optimized out>
        pages = 0
        pagesize_bits = <optimized out>
        pss = 
          {block = <optimized out>, page = 281, complete_round = <optimized out>}
        pages = 0
--Type <RET> for more, q to quit, c to continue without paging--
        again = true
        found = <optimized out>
#10 0x0000559f125b11f2 in ram_find_and_save_block
    (rs=rs@entry=0x7f6fa0000b80, last_stage=last_stage@entry=false)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:2669
        pss = 
          {block = <optimized out>, page = 281, complete_round = <optimized out>}
        pages = 0
        again = true
        found = <optimized out>
#11 0x0000559f125b163a in ram_find_and_save_block
    (last_stage=false, rs=0x7f6fa0000b80)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:3528
        pages = 0
        pages = <optimized out>
        temp = <optimized out>
        rs = 0x7f6fa0000b80
        ret = <optimized out>
        i = 281
        t0 = 238615130386646
--Type <RET> for more, q to quit, c to continue without paging--
        done = 0


#12 0x0000559f125b163a in ram_save_iterate
    (f=0x559f1392bca0, opaque=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:3536
        pages = <optimized out>
        temp = <optimized out>
        rs = 0x7f6fa0000b80
        ret = <optimized out>
        i = 281
        t0 = 238615130386646
        done = 0

#13 0x0000559f1278011f in qemu_savevm_state_iterate
    (f=0x559f1392bca0, postcopy=false) at migration/savevm.c:1185
        se = 0x559f1395d6f0
        ret = 1
#14 0x0000559f1277c188 in migration_thread (opaque=0x559f138ca000)
    at migration/migration.c:3107
        s = 0x559f138ca000
        setup_start = <optimized out>
        thr_error = <optimized out>
        urgent = <optimized out>
#15 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13dc5ac0)
--Type <RET> for more, q to quit, c to continue without paging--
    at util/qemu-thread-posix.c:502
        __clframe = 
          {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x559f13dc5ac0
        start_routine = 0x559f1277bfb0 <migration_thread>
        arg = 0x559f138ca000
        r = <optimized out>
#16 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0
#17 0x00007f716b0cae53 in clone () at /lib64/libc.so.6

Comment 16 Li Xiaohui 2019-11-18 03:10:04 UTC
> Could you confirm:
> a- you are using -incoming defer on both launch of migration
> b- to be sure that the first destination qemu has died, could you try a different port for each case?
> c - if it still fails, could you give me any logs/output.

Juan, follows your advise a->b, reproduce bz, too. The core dump log is same with Comment 15

Comment 17 Juan Quintela 2019-11-19 12:18:08 UTC
Hi Xiauhi

Thanks very much.  This is the interesting bit of the trace.


Thread 1 (Thread 0x7f6f8b7fe700 (LWP 13061)):
#0  0x00007f716b00682f in raise () at /lib64/libc.so.6
#1  0x00007f716aff0c45 in abort () at /lib64/libc.so.6
#2  0x00007f716aff0b19 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
#3  0x00007f716affede6 in .annobin_assert.c_end () at /lib64/libc.so.6
#4  0x0000559f128bb5c6 in qemu_mutex_lock_impl
    (mutex=<optimized out>, file=<optimized out>, line=<optimized out>)
    at util/qemu-thread-posix.c:64
        err = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        __PRETTY_FUNCTION__ = "qemu_mutex_lock_impl"
        __func__ = "qemu_mutex_lock_impl"
#5  0x0000559f125ac18b in multifd_send_pages ()
    at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:937
        _f = <optimized out>
        i = 2
        next_channel = 2
        p = 0x559f13bacb70
        pages = 0x559f1395d340
        transferred = <optimized out>
        __func__ = "multifd_send_pages"


Especifically the "i=2/next_channel=2"
Somehow the information that you have changed the number of channels on source has been lost.
Investigating *why/how".

Thanksx, Juan.

Comment 18 Juan Quintela 2019-12-05 13:55:23 UTC
Hi

I am finishing a series today/monday that finishes this discrepancies.  The problem here is that when we cancel, we are freeing too much memory.  Recovening that.

Later, Juan.

Comment 19 Juan Quintela 2019-12-18 09:16:30 UTC
Hi

Posted fixed upstrem:

https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg03691.html

Comment 22 Juan Quintela 2019-12-24 08:31:48 UTC
Dedpended on previous patches pulled on Friday.
Will send PULL request today.
To make things easier for QE will cerate a build with the patches even before the ACK.

Comment 23 Li Xiaohui 2020-01-03 02:57:01 UTC
Hi Juan,
Since RHEL8.1.1-av has been in Testing Phase, do you have any update on this bz?
Thanks,
Li Xiaohui

Comment 24 Juan Quintela 2020-01-07 12:18:07 UTC
Hi Xiaolhi

Previous fix make postcopy-recovery fail (still investigating where the fail is inside the postcopy recovery).  I just sent another set of patches upstream that have ack upstream.  Will post the PULL request later today.

Later, Juan.

Comment 29 Ademar Reis 2020-02-05 23:02:14 UTC
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 30 Juan Quintela 2020-02-26 08:56:58 UTC
Hi

This patch series are ready upstream, will be a fast backport and will allow multifd to be much more resliient in the case of users doing cancel in the middle of a migration.
On the other hand, this feature hasn't be used on previous releases.

Later, Juan.

Comment 31 Juan Quintela 2020-03-03 13:41:16 UTC
Hi

Patches posted downstream.
All of them integrated upstream.

This is brew.
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=26970927

Comment 33 Li Xiaohui 2020-03-04 03:44:49 UTC
(In reply to Juan Quintela from comment #31)
> Hi
> 
> Patches posted downstream.
> All of them integrated upstream.
> 
> This is brew.
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=26970927

Thanks, Juan, will try later.

Comment 37 Li Xiaohui 2020-03-14 04:51:18 UTC
add case RHEL_186122 to cover this bz, and have rhel_186122 automated. 
Run this case for 100 times via automation, all pass, so make this bz verified:
========================= Test Requirement: RHEL7-175691-X86-Q35-BLOCKDEV(Migration - x86) =========================
**********************************************************************************************
RESULTS [RHEL7-175691-X86-Q35-BLOCKDEV]:
==>TOTAL : 100
==>PASS : 100 
   1: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 52 sec)
   2: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 52 sec)
   3: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 56 sec)
  ...
   98: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 32 sec)
   99: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 32 sec)
   100: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 28 sec)
==>ERROR : 0 
==>FAIL : 0 
==>CANCEL : 0 
==>SKIP : 0 
==>WARN : 0 
==>RUN TIME : 186 min 13 sec 
==>TEST LOG : /home/ipa/test_logs/rhel7_175691_x86_q35_blockdev-2020-03-13-12:00:13 
**********************************************************************************************

Comment 39 errata-xmlrpc 2020-05-05 09:47:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.