Description of problem: set multifd-channels 4, do multifd migration, cancel during migration, then change multifd-channels 2, do multifd migration again, qemu on src host will core dump Version-Release number of selected component (if applicable): src&dst host info: kernel-4.18.0-125.el8.x86_64 & qemu-4.1.0-rc3 guest info: kernel-4.18.0-128.el8.x86_64 How reproducible: 2/2 Steps to Reproduce: 1.boot a guest on src host, run a stress in guest: # stressapptest -M 4500M -s 1000000 Notes: there is 5G free mem in guest, so mem is enough 2.boot a guest on dst host with "-incoming tcp:0:4444" 3.on src&dst host, enable multifd and set multifd-channels 4 4.set speed 2GBps in src qemu 5.do multifd migration from src to dst host, during migration, execute migrate_cancel, cancel migration successfully: (qemu) migrate -d tcp:192.168.11.22:4444 (qemu) migrate_cancel (qemu) qemu-kvm: multifd_send_pages: channel 3 has already quit! qemu-kvm: multifd_send_pages: channel 3 has already quit! qemu-kvm: multifd_send_sync_main: multifd_send_pages fail qemu-kvm: Unable to write to socket: Broken pipe (qemu) info migrate globals: store-global-state: on only-migratable: off send-configuration: on send-section-footer: on decompress-error-check: on clear-bitmap-shift: 18 capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off multifd: on dirty-bitmaps: off postcopy-blocktime: off late-block-activate: off x-ignore-shared: off Migration status: cancelled total time: 0 milliseconds 6.boot a guest on dst host again, and enable multifd in dst qemu 7.set multifd-channels 2 on src&dst host 8.do multifd migration again Actual results: after step7, qemu in src host will core dump: (qemu) migrate -d tcp:192.168.11.22:4444 (qemu) qemu-kvm: util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed. ./start2.sh: line 22: 10906 Aborted (core dumped) /usr/libexec/qemu-kvm -enable-kvm -nodefaults -machine q35 -m 8G -smp 8 -cpu Haswell-noTSX-IBRS -name debug-threads=on -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 -device virtio-scsi-pci,id=scsi0,bus=pcie.0-root-port-2 -drive file=/mnt/nfs/rhel810-scsi-0729-2.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,media=disk,cache=none,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,queues=4 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=18:66:da:5e:c2:3c,bus=pcie.0-root-port-3,vectors=10,mq=on -qmp tcp:0:3333,server,nowait -vnc :1 -device VGA -monitor stdio Expected results: migration go on, and finish successfully. Additional info: sometimes, when execute step1~5, qemu on dst host will core dump, error like src qemu core dump
Working on given better error results in cases like this.
Can you reproduce with lastest rhel8.1 qemu-kvm? I am not able to reprouce with ether:: * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4 * rhel8.1 Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a I am trying to reproduce with: - multifd on - multifd-channels=10 - migrate_set_downtime 1ms - migrate_set_speed 1k - stress the test - migrate -d <whatever> - migrate_cancel destination stops as expected, and source continues - lanunch destination again - migrate_set_downtime 1000ms - migrate_set_speed 1G (with that I on my hardware it converges, vary it or stop stressing the guest) - migrate -d <whatever> And it migrates correctly. Probably the serialitation patches from bugzilla 1726898 helped here. Later, Juan.
(In reply to Juan Quintela from comment #3) > Can you reproduce with lastest rhel8.1 qemu-kvm? > I am not able to reprouce with ether:: > * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4 > * rhel8.1 Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a > > I am trying to reproduce with: > - multifd on > - multifd-channels=10 > - migrate_set_downtime 1ms > - migrate_set_speed 1k > - stress the test > - migrate -d <whatever> > - migrate_cancel > destination stops as expected, and source continues > - lanunch destination again > - migrate_set_downtime 1000ms > - migrate_set_speed 1G > (with that I on my hardware it converges, vary it or stop stressing the > guest) > - migrate -d <whatever> > > And it migrates correctly. > Probably the serialitation patches from bugzilla 1726898 helped here. > Later, Juan. Hi Xiaohui, Can you have a try? Best regards, Junyi
(In reply to juzhang from comment #4) > (In reply to Juan Quintela from comment #3) > > Can you reproduce with lastest rhel8.1 qemu-kvm? > > I am not able to reprouce with ether:: > > * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4 > > * rhel8.1 Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a > > > > I am trying to reproduce with: > > - multifd on > > - multifd-channels=10 > > - migrate_set_downtime 1ms > > - migrate_set_speed 1k > > - stress the test > > - migrate -d <whatever> > > - migrate_cancel > > destination stops as expected, and source continues > > - lanunch destination again > > - migrate_set_downtime 1000ms > > - migrate_set_speed 1G > > (with that I on my hardware it converges, vary it or stop stressing the > > guest) > > - migrate -d <whatever> > > > > And it migrates correctly. > > Probably the serialitation patches from bugzilla 1726898 helped here. > > Later, Juan. > > Hi Xiaohui, > > Can you have a try? > > Best regards, > > Junyi Yes, will try when hosts are available
(In reply to Juan Quintela from comment #3) > Can you reproduce with lastest rhel8.1 qemu-kvm? > I am not able to reprouce with ether:: > * upstream commit 25311649592f5584b1e1012d69c7327ef87473f4 > * rhel8.1 Update to qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a > > I am trying to reproduce with: > - multifd on > - multifd-channels=10 > - migrate_set_downtime 1ms > - migrate_set_speed 1k > - stress the test > - migrate -d <whatever> > - migrate_cancel > destination stops as expected, and source continues > - lanunch destination again > - migrate_set_downtime 1000ms > - migrate_set_speed 1G > (with that I on my hardware it converges, vary it or stop stressing the > guest) Maybe there is lack of "change multifd-channels" setting after migrate again from src to dst host(According to Comment 0 's step7) > - migrate -d <whatever> > > And it migrates correctly. > Probably the serialitation patches from bugzilla 1726898 helped here. > Later, Juan. Hi Juan, I can reproduce this issue on the latest host(kernel-4.18.0-145.el8.x86_64&qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64), test steps like Comment 0. Notes: reproduce 1 time after tried 3 times. (1)in hmp, get error: (qemu) migrate -d tcp:10.73.73.87:4444 (qemu) qemu-kvm: util/qemu-thread-posix.c:64: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed. (qemu) ./start3.sh: line 22: 8642 Aborted (core dumped) /usr/libexec/qemu-kvm -enable-kvm ... (2)in gdb mode, get error: # gdb -p 8642 ... Thread 104 "live_migration" received signal SIGABRT, Aborted. [Switching to Thread 0x7f30464f9700 (LWP 8860)] 0x00007f3294f6a8df in raise () from /lib64/libc.so.6 (gdb) c Continuing. [Thread 0x7f3045cf8700 (LWP 8862) exited] [Thread 0x7f3023fff700 (LWP 8861) exited] ... Program terminated with signal SIGABRT, Aborted. The program no longer exists.
Confirm again about change multifd-channels in Comment 0 's step 7, if test like Comment 0, but without step 7(just set multifd-channels to 4, let it's same with src host qemu), qemu on src host will be stuck(guest on src host is stuck, too): (qemu) migrate -d tcp:10.73.73.87:4444 (qemu) qemu-kvm: multifd_send_pages: channel 1 has already quit! qemu-kvm: multifd_send_pages: channel 1 has already quit! qemu-kvm: multifd_send_sync_main: multifd_send_pages fail
(In reply to Li Xiaohui from comment #7) > Confirm again about change multifd-channels in Comment 0 's step 7, if test > like Comment 0, but without step 7(just set multifd-channels to 4, let it's > same with src host qemu), qemu on src host will be stuck(guest on src host > is stuck, too): this issue will reproduce 1/5 times: qemu&guest on src host will be stuck > (qemu) migrate -d tcp:10.73.73.87:4444 > (qemu) qemu-kvm: multifd_send_pages: channel 1 has already quit! > qemu-kvm: multifd_send_pages: channel 1 has already quit! > qemu-kvm: multifd_send_sync_main: multifd_send_pages fail
I think that the remaining problem is the same that bugzilla 1726898. See comment #24 there. Could you retest adding: --global migration.multifd-channels=10 to the command line (the destination side is where it is necesary). 10 should be enough for any number of channels that you use, but if it fails, just put there the maximum number of channels that you are going to use.
(In reply to Juan Quintela from comment #10) > I think that the remaining problem is the same that bugzilla 1726898. See > comment #24 there. > > Could you retest adding: > > --global migration.multifd-channels=10 > > to the command line (the destination side is where it is necesary). 10 > should be enough for any number of channels that you use, but if it fails, > just put there the maximum number of channels that you are going to use. ok, I will try later
(In reply to Juan Quintela from comment #10) > I think that the remaining problem is the same that bugzilla 1726898. See > comment #24 there. > > Could you retest adding: > > --global migration.multifd-channels=10 > > to the command line (the destination side is where it is necesary). 10 > should be enough for any number of channels that you use, but if it fails, > just put there the maximum number of channels that you are going to use. Hi Juan, When I use "-incoming defer" in dst qemu to retest this bz, could reproduce it by low reproduction rate(1/10 or 1/20, not sure). Test steps like: 1.start a guest on src host; 2.start a guest on dst host with "-incoming defer"; 3.set migration speed to 1M in src qemu; 4.enable multifd and set multifd-channels to 4 on src&dst host; 5.set migration listening port in dst qemu: (qemu) migrate_incoming tcp:10.66.8.208:4444 6. migrate guest from src to dst host; (qemu) migrate -d tcp:10.66.8.208:4444 7.during migration is active, cancel migration in src qemu: (qemu) migrate_cancel 8.restart a guest on dst host with "-incoming defer"; 9.set multifd-channels to 2 in src qemu; 10.enable multifd and set multifd-channels to 2 in dst qemu; 11.set migration listening port in dst qemu: (qemu) migrate_incoming tcp:10.66.8.208:4444 12.migrate guest from src to dst host; (qemu) migrate -d tcp:10.66.8.208:4444
ok, thanks. Trying to reproduce with your last recipe.
Hi Xiaohli As said on bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1726898, I think that the problem is fixed. Could you confirm: a- you are using -incoming defer on both launch of migration b- to be sure that the first destination qemu has died, could you try a different port for each case? c - if it still fails, could you give me any logs/output. Thanks, Juan. PD. Testing a similar way with Autotest(Lukkas) I was able to make it survive 100 times, it is not the same test, because I don't know yet how to change the test.
Hi Juan, reproduce bz as Comment 12 's test steps on hosts(kernel-4.18.0-148.el8.x86_64&qemu-img-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64), the qemu core dump info like followings: (gdb) t a a bt full Thread 15 (Thread 0x7f71645db700 (LWP 12828)): #0 0x00007f716b3a4250 in nanosleep () at /lib64/libpthread.so.0 #1 0x00007f716feae4d7 in g_usleep () at /lib64/libglib-2.0.so.0 #2 0x0000559f128cd848 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:252 tries = 2 n = 8 node = <optimized out> #3 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f138305e0) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f138305e0 start_routine = 0x559f128cd7e0 <call_rcu_thread> arg = 0x0 r = <optimized out> #4 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #5 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 14 (Thread 0x7f6fa94f7700 (LWP 13062)): #0 0x00007f716b3a2e56 in do_futex_wait.constprop () at /lib64/libpthread.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #1 0x00007f716b3a2f48 in __new_sem_wait_slow.constprop.0 () at /lib64/libpthread.so.0 #2 0x0000559f128bbb34 in qemu_sem_wait (sem=sem@entry=0x559f13baca20) at util/qemu-thread-posix.c:319 rc = <optimized out> __PRETTY_FUNCTION__ = "qemu_sem_wait" __func__ = "qemu_sem_wait" #3 0x0000559f125acd08 in multifd_send_thread (opaque=0x559f13baca00) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:1115 p = 0x559f13baca00 local_err = 0x0 ret = 0 flags = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f1486ef40) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f1486ef40 start_routine = 0x559f125acc60 <multifd_send_thread> arg = 0x559f13baca00 r = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 13 (Thread 0x7f7160862700 (LWP 12844)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f13a31d60, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7160861680, reg_save_area = 0x7f7160861640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a31d60) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f13a31d60 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a54b50) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13a54b50 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f13a31d60 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 12 (Thread 0x7f7143fff700 (LWP 12845)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f13a55390, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 --Type <RET> for more, q to quit, c to continue without paging-- ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7143ffe680, reg_save_area = 0x7f7143ffe640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a55390) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f13a55390 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a78180) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, --Type <RET> for more, q to quit, c to continue without paging-- __cancel_type = <optimized out>} qemu_thread_args = 0x559f13a78180 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f13a55390 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 11 (Thread 0x7f71437fe700 (LWP 12846)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f13a789c0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f71437fd680, reg_save_area = 0x7f71437fd640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = --Type <RET> for more, q to quit, c to continue without paging-- {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a789c0) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f13a789c0 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a9b7b0) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13a9b7b0 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f13a789c0 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 10 (Thread 0x7f6f89ffb700 (LWP 13057)): --Type <RET> for more, q to quit, c to continue without paging-- #0 0x00007f716b3a3072 in do_futex_wait () at /lib64/libpthread.so.0 #1 0x00007f716b3a3183 in __new_sem_wait_slow () at /lib64/libpthread.so.0 #2 0x0000559f128bba7f in qemu_sem_timedwait (sem=sem@entry=0x559f13925788, ms=ms@entry=10000) at util/qemu-thread-posix.c:289 rc = <optimized out> ts = {tv_sec = 1574045029, tv_nsec = 189319000} __PRETTY_FUNCTION__ = "qemu_sem_timedwait" __func__ = "qemu_sem_timedwait" #3 0x0000559f128b67c4 in worker_thread (opaque=0x559f13925710) at util/thread-pool.c:91 req = <optimized out> ret = <optimized out> pool = 0x559f13925710 #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a31340) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13a31340 start_routine = 0x559f128b6730 <worker_thread> arg = 0x559f13925710 r = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 9 (Thread 0x7f7140bff700 (LWP 12852)): #0 0x00007f716b3a047c in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0 #1 0x0000559f128bb86d in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x559f14458418, file=0x559f12a37c37 "ui/vnc-jobs.c", line=214) at util/qemu-thread-posix.c:161 err = <optimized out> __PRETTY_FUNCTION__ = "qemu_cond_wait_impl" __func__ = "qemu_cond_wait_impl" #2 0x0000559f127e4d71 in vnc_worker_thread_loop (queue=queue@entry=0x559f144583e0) at ui/vnc-jobs.c:214 _f = <optimized out> job = <optimized out> entry = <optimized out> tmp = <optimized out> vs = Python Exception <class 'gdb.error'> value of type `VncState' requires 75456 bytes, which is more than max-value-size: #3 0x0000559f127e5330 in vnc_worker_thread (arg=0x559f144583e0) at ui/vnc-jobs.c:324 queue = 0x559f144583e0 #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13bde120) at util/qemu-thread-posix.c:502 --Type <RET> for more, q to quit, c to continue without paging-- __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13bde120 start_routine = 0x559f127e5310 <vnc_worker_thread> arg = 0x559f144583e0 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 8 (Thread 0x7f6fa9cf8700 (LWP 13063)): #0 0x00007f716b3a2e56 in do_futex_wait.constprop () at /lib64/libpthread.so.0 #1 0x00007f716b3a2f48 in __new_sem_wait_slow.constprop.0 () at /lib64/libpthread.so.0 #2 0x0000559f128bbb34 in qemu_sem_wait (sem=sem@entry=0x559f13bacad8) at util/qemu-thread-posix.c:319 rc = <optimized out> __PRETTY_FUNCTION__ = "qemu_sem_wait" __func__ = "qemu_sem_wait" #3 0x0000559f125acd08 in multifd_send_thread (opaque=0x559f13bacab8) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:1115 p = 0x559f13bacab8 --Type <RET> for more, q to quit, c to continue without paging-- local_err = 0x0 ret = 0 flags = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f14809a90) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f14809a90 start_routine = 0x559f125acc60 <multifd_send_thread> arg = 0x559f13bacab8 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 7 (Thread 0x7f7162866700 (LWP 12840)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f13979730, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 --Type <RET> for more, q to quit, c to continue without paging-- ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7162865680, reg_save_area = 0x7f7162865640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13979730) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f13979730 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f1399c760) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f1399c760 --Type <RET> for more, q to quit, c to continue without paging-- start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f13979730 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 6 (Thread 0x7f7161063700 (LWP 12843)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f13a0dfc0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7161062680, reg_save_area = 0x7f7161062640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} --Type <RET> for more, q to quit, c to continue without paging-- run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a0dfc0) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f13a0dfc0 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a31550) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13a31550 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f13a0dfc0 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 5 (Thread 0x7f7161864700 (LWP 12842)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl --Type <RET> for more, q to quit, c to continue without paging-- (cpu=cpu@entry=0x559f139eabc0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7161863680, reg_save_area = 0x7f7161863640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f139eabc0) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f139eabc0 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13a0d780) --Type <RET> for more, q to quit, c to continue without paging-- at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13a0d780 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f139eabc0 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 4 (Thread 0x7f717079af00 (LWP 12827)): #0 0x00007f716b0c0026 in ppoll () at /lib64/libc.so.6 #1 0x0000559f128b7175 in ppoll (__ss=0x0, __timeout=0x7ffcbd8a48c0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77 ts = {tv_sec = 0, tv_nsec = 56621687} tvsec = <optimized out> #2 0x0000559f128b7175 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=56621687) at util/qemu-timer.c:334 ts = {tv_sec = 0, tv_nsec = 56621687} tvsec = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- #3 0x0000559f128b8075 in os_host_main_loop_wait (timeout=56621687) at util/main-loop.c:236 context = 0x559f139267f0 ret = <optimized out> mlpoll = {state = 0, timeout = 4294967295, pollfds = 0x559f13925810} ret = <optimized out> timeout_ns = <optimized out> #4 0x0000559f128b8075 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:517 mlpoll = {state = 0, timeout = 4294967295, pollfds = 0x559f13925810} ret = <optimized out> timeout_ns = <optimized out> #5 0x0000559f126a1169 in main_loop () at vl.c:1809 #6 0x0000559f12550fd3 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4506 i = <optimized out> snapshot = 0 linux_boot = <optimized out> initrd_filename = 0x0 kernel_filename = 0x0 kernel_cmdline = <optimized out> boot_order = 0x559f1296facc "cad" --Type <RET> for more, q to quit, c to continue without paging-- boot_once = <optimized out> ds = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> accel_opts = 0x0 olist = <optimized out> optind = 35 optarg = 0x7ffcbd8a54f4 "stdio" loadvm = 0x0 machine_class = 0x559f1391c120 cpu_option = 0x7ffcbd8a51d0 "Skylake-Client,+kvm_pv_unhalt" vga_model = 0x0 qtest_chrdev = 0x0 qtest_log = 0x0 incoming = 0x0 userconfig = <optimized out> nographic = false display_remote = <optimized out> log_mask = <optimized out> log_file = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- ram_slots = 0 vmstate_dump_file = 0x0 main_loop_err = 0x0 err = 0x0 list_data_dirs = false dir = <optimized out> bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffcbd8a4a50} __func__ = "main" Thread 3 (Thread 0x7f7162065700 (LWP 12841)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f139c6a90, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7162064680, reg_save_area = 0x7f7162064640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 --Type <RET> for more, q to quit, c to continue without paging-- attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f139c6a90) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 cpu = 0x559f139c6a90 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f139ea380) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f139ea380 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f139c6a90 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 --Type <RET> for more, q to quit, c to continue without paging-- Thread 2 (Thread 0x7f7142ffd700 (LWP 12847)): #0 0x00007f716b0c184b in ioctl () at /lib64/libc.so.6 #1 0x0000559f125b7259 in kvm_vcpu_ioctl (cpu=cpu@entry=0x559f13a9bff0, type=type@entry=44672) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2411 ret = <optimized out> arg = 0x0 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f7142ffc680, reg_save_area = 0x7f7142ffc640}} #2 0x0000559f125b7319 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/accel/kvm/kvm-all.c:2248 attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0} run = <optimized out> ret = <optimized out> run_ret = <optimized out> #3 0x0000559f1259c56e in qemu_kvm_cpu_thread_fn (arg=0x559f13a9bff0) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/cpus.c:1285 --Type <RET> for more, q to quit, c to continue without paging-- cpu = 0x559f13a9bff0 r = <optimized out> #4 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13abede0) at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13abede0 start_routine = 0x559f1259c4b0 <qemu_kvm_cpu_thread_fn> arg = 0x559f13a9bff0 r = <optimized out> #5 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #6 0x00007f716b0cae53 in clone () at /lib64/libc.so.6 Thread 1 (Thread 0x7f6f8b7fe700 (LWP 13061)): #0 0x00007f716b00682f in raise () at /lib64/libc.so.6 #1 0x00007f716aff0c45 in abort () at /lib64/libc.so.6 #2 0x00007f716aff0b19 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 #3 0x00007f716affede6 in .annobin_assert.c_end () at /lib64/libc.so.6 #4 0x0000559f128bb5c6 in qemu_mutex_lock_impl (mutex=<optimized out>, file=<optimized out>, line=<optimized out>) at util/qemu-thread-posix.c:64 err = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- __PRETTY_FUNCTION__ = "qemu_mutex_lock_impl" __func__ = "qemu_mutex_lock_impl" #5 0x0000559f125ac18b in multifd_send_pages () at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:937 _f = <optimized out> i = 2 next_channel = 2 p = 0x559f13bacb70 pages = 0x559f1395d340 transferred = <optimized out> __func__ = "multifd_send_pages" #6 0x0000559f125b11f2 in multifd_queue_page (offset=<optimized out>, block=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:984 pages = 0x559f1395d340 offset = 1150976 res = -1 tmppages = <optimized out> pages = 0 pagesize_bits = <optimized out> pss = --Type <RET> for more, q to quit, c to continue without paging-- {block = <optimized out>, page = 281, complete_round = <optimized out>} pages = 0 again = true found = <optimized out> #7 0x0000559f125b11f2 in ram_save_multifd_page (rs=<optimized out>, offset=<optimized out>, block=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:2082 offset = 1150976 res = -1 tmppages = <optimized out> pages = 0 pagesize_bits = <optimized out> pss = {block = <optimized out>, page = 281, complete_round = <optimized out>} pages = 0 again = true found = <optimized out> #8 0x0000559f125b11f2 in ram_save_target_page (last_stage=false, pss=<synthetic pointer>, rs=0x7f6fa0000b80) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migra--Type <RET> for more, q to quit, c to continue without paging-- tion/ram.c:2563 offset = 1150976 res = -1 tmppages = <optimized out> pages = 0 pagesize_bits = <optimized out> pss = {block = <optimized out>, page = 281, complete_round = <optimized out>} pages = 0 again = true found = <optimized out> #9 0x0000559f125b11f2 in ram_save_host_page (last_stage=<optimized out>, pss=<synthetic pointer>, rs=0x7f6fa0000b80) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:2606 tmppages = <optimized out> pages = 0 pagesize_bits = <optimized out> pss = {block = <optimized out>, page = 281, complete_round = <optimized out>} pages = 0 --Type <RET> for more, q to quit, c to continue without paging-- again = true found = <optimized out> #10 0x0000559f125b11f2 in ram_find_and_save_block (rs=rs@entry=0x7f6fa0000b80, last_stage=last_stage@entry=false) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:2669 pss = {block = <optimized out>, page = 281, complete_round = <optimized out>} pages = 0 again = true found = <optimized out> #11 0x0000559f125b163a in ram_find_and_save_block (last_stage=false, rs=0x7f6fa0000b80) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:3528 pages = 0 pages = <optimized out> temp = <optimized out> rs = 0x7f6fa0000b80 ret = <optimized out> i = 281 t0 = 238615130386646 --Type <RET> for more, q to quit, c to continue without paging-- done = 0 #12 0x0000559f125b163a in ram_save_iterate (f=0x559f1392bca0, opaque=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:3536 pages = <optimized out> temp = <optimized out> rs = 0x7f6fa0000b80 ret = <optimized out> i = 281 t0 = 238615130386646 done = 0 #13 0x0000559f1278011f in qemu_savevm_state_iterate (f=0x559f1392bca0, postcopy=false) at migration/savevm.c:1185 se = 0x559f1395d6f0 ret = 1 #14 0x0000559f1277c188 in migration_thread (opaque=0x559f138ca000) at migration/migration.c:3107 s = 0x559f138ca000 setup_start = <optimized out> thr_error = <optimized out> urgent = <optimized out> #15 0x0000559f128bb4b4 in qemu_thread_start (args=0x559f13dc5ac0) --Type <RET> for more, q to quit, c to continue without paging-- at util/qemu-thread-posix.c:502 __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>} qemu_thread_args = 0x559f13dc5ac0 start_routine = 0x559f1277bfb0 <migration_thread> arg = 0x559f138ca000 r = <optimized out> #16 0x00007f716b39a2de in start_thread () at /lib64/libpthread.so.0 #17 0x00007f716b0cae53 in clone () at /lib64/libc.so.6
> Could you confirm: > a- you are using -incoming defer on both launch of migration > b- to be sure that the first destination qemu has died, could you try a different port for each case? > c - if it still fails, could you give me any logs/output. Juan, follows your advise a->b, reproduce bz, too. The core dump log is same with Comment 15
Hi Xiauhi Thanks very much. This is the interesting bit of the trace. Thread 1 (Thread 0x7f6f8b7fe700 (LWP 13061)): #0 0x00007f716b00682f in raise () at /lib64/libc.so.6 #1 0x00007f716aff0c45 in abort () at /lib64/libc.so.6 #2 0x00007f716aff0b19 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 #3 0x00007f716affede6 in .annobin_assert.c_end () at /lib64/libc.so.6 #4 0x0000559f128bb5c6 in qemu_mutex_lock_impl (mutex=<optimized out>, file=<optimized out>, line=<optimized out>) at util/qemu-thread-posix.c:64 err = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- __PRETTY_FUNCTION__ = "qemu_mutex_lock_impl" __func__ = "qemu_mutex_lock_impl" #5 0x0000559f125ac18b in multifd_send_pages () at /usr/src/debug/qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64/migration/ram.c:937 _f = <optimized out> i = 2 next_channel = 2 p = 0x559f13bacb70 pages = 0x559f1395d340 transferred = <optimized out> __func__ = "multifd_send_pages" Especifically the "i=2/next_channel=2" Somehow the information that you have changed the number of channels on source has been lost. Investigating *why/how". Thanksx, Juan.
Hi I am finishing a series today/monday that finishes this discrepancies. The problem here is that when we cancel, we are freeing too much memory. Recovening that. Later, Juan.
Hi Posted fixed upstrem: https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg03691.html
Dedpended on previous patches pulled on Friday. Will send PULL request today. To make things easier for QE will cerate a build with the patches even before the ACK.
Hi Juan, Since RHEL8.1.1-av has been in Testing Phase, do you have any update on this bz? Thanks, Li Xiaohui
Hi Xiaolhi Previous fix make postcopy-recovery fail (still investigating where the fail is inside the postcopy recovery). I just sent another set of patches upstream that have ack upstream. Will post the PULL request later today. Later, Juan.
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks
Hi This patch series are ready upstream, will be a fast backport and will allow multifd to be much more resliient in the case of users doing cancel in the middle of a migration. On the other hand, this feature hasn't be used on previous releases. Later, Juan.
Hi Patches posted downstream. All of them integrated upstream. This is brew. https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=26970927
(In reply to Juan Quintela from comment #31) > Hi > > Patches posted downstream. > All of them integrated upstream. > > This is brew. > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=26970927 Thanks, Juan, will try later.
add case RHEL_186122 to cover this bz, and have rhel_186122 automated. Run this case for 100 times via automation, all pass, so make this bz verified: ========================= Test Requirement: RHEL7-175691-X86-Q35-BLOCKDEV(Migration - x86) ========================= ********************************************************************************************** RESULTS [RHEL7-175691-X86-Q35-BLOCKDEV]: ==>TOTAL : 100 ==>PASS : 100 1: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 52 sec) 2: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 52 sec) 3: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 56 sec) ... 98: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 32 sec) 99: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 32 sec) 100: RHEL-186122-[Multiple-fds] Multifd migration cancel test (1 min 28 sec) ==>ERROR : 0 ==>FAIL : 0 ==>CANCEL : 0 ==>SKIP : 0 ==>WARN : 0 ==>RUN TIME : 186 min 13 sec ==>TEST LOG : /home/ipa/test_logs/rhel7_175691_x86_q35_blockdev-2020-03-13-12:00:13 **********************************************************************************************
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017