Bug 1956897

An idea how to test 'yank' command is to start migration and then inject firewall rules that would drop packets silently (that is DROP instead of REJECT - so that the source doesn't get notified).

Tested 'yank' on rhelav-8.5.0 (kernel-4.18.0-304.3.el8.x86_64 & qemu-img-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64)


Test scenarios: 
1.inject firewall via drop on dst host when migration is active, then migration hang, use yank to fail migration. 


Test steps:
1.Boot a guest on src host;
2.Boot a guest on dst host with '-incoming defer';
3.Set migration incoming on dst host via qmp cmd;
{"execute":"migrate-incoming","arguments":{"uri":"tcp:[::]:1234"}}
4.Start migration on src host via qmp cmd;
{"execute": "migrate","arguments":{"uri": "tcp:10.73.130.69:1234"}}
5.During migration is active, inject firewall via drop on dst host:
# iptables -A INPUT -p tcp --dport 1234 -j DROP
6.After migration hang(query migrate, only the total time of migration is increasing, other migration params stay unchanged), use yank cmd to fail migration on src host:
{ "execute": "query-yank" }
{"return": [{"type": "chardev", "id": "qmp_id_qmpmonitor1"}, {"type": "chardev", "id": "qmp_id_catch_monitor"}, {"type": "chardev", "id": "compat_monitor0"}, {"type": "chardev", "id": "serial0"}, {"type": "migration"}]}
{"execute":"yank","arguments":{"instances":[{"type":"migration"}]}}


Actual result:
After step 6, query migration status, get failed status on src host via qmp cmd, I think the result is expected:
{"execute":"query-migrate"}
{"return": {"blocked": false, "status": "failed", "error-desc": "Unable to write to socket: Broken pipe"}}

Next we need quit qemu process on dst host by manually, then close firewall on dst host, we can start migration again and vm works well on dst host after migration.



BTW, who could help answer following two questions: 
Question 1) do we need test 'yank' with network failure scenario(hit a failure of the migration network during migration is active)???
I think network failure scenario is more nearly the requirement about 'yank' as it would hit qemu hang issue but inject firewall only hit migration hang(qemu and qmp still work well) :
*******************************************************************************
+# A yank instance can be yanked with the @yank qmp command to recover from a hanging QEMU.
+#
+# Currently implemented yank instances:
+#  - nbd block device:
+#    Yanking it will shut down the connection to the nbd server without
+#    attempting to reconnect.
+#  - socket chardev:
+#    Yanking it will shut down the connected socket.
+#  - migration:
+#    Yanking it will shut down all migration connections. Unlike
+#    @migrate_cancel, it will not notify the migration process, so migration
+#    will go into @failed state, instead of @cancelled state. @yank should be
+#    used to recover from hangs.

Question 2) shall qemu on dst host quit by automatically after executing 'yank' command as migration would fail?

Besides migration, yank is also related with nbd block, chardev (You can see the last content in Comment 4 or see defails from downstream qemu-kvm-6.0 commit: 50186051f425da3ace2425371c5271d0b64e7122).

Thanks, that's a good test.  It would be better to use the "oob" capability (see https://github.com/qemu/qemu/blob/master/docs/interop/qmp-spec.txt#L116 )
that way even if there is a currently executing QMP command that's blocked, the 'yank' command should still execute.

Moving this to POST since this was included in qemu-6.0 and as noted in comment 4 is already testable.

I set ITM=14 mainly to get the release+ - feel free to use a later one for completion of "new" tests though
I did not set DTM, theoretically it could be 10 as that's about when the code was built, but that'll probably anger the dev missed bot since 10 already passed.

Danilo - I'll let you do the rest of the magic to move to ON_QA

(In reply to Dr. David Alan Gilbert from comment #6)
> Thanks, that's a good test.  It would be better to use the "oob" capability
> (see https://github.com/qemu/qemu/blob/master/docs/interop/qmp-spec.txt#L116
> )
> that way even if there is a currently executing QMP command that's blocked,
> the 'yank' command should still execute.

Thanks for the reminder. 

I will test network failure scenario when the machines are available.

Hi David,
I hit qemu core dump when do tls migration on the latest rhelav-8.5.0(qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64), could you help have a look as it give error about yank:
(qemu) qemu-kvm: ../util/yank.c:107: yank_unregister_instance: Assertion `QLIST_EMPTY(&entry->yankfns)' failed.
Aborted (core dumped)

Test steps:
1.Ca files generated as setup
2.Boot a vm as tls server on dst host;
3.Boot a vm as tls client on src host;
4.In dst host:
(qemu) migrate_set_parameter tls-creds tls0
(qemu) migrate_incoming tcp:$dst_host_ip:5801
In src host:
(qemu) migrate_set_parameter tls-creds tls0
(qemu) migrate -d tcp:$dst_host_short_name:5801


During migration, qemu on src&dst host hit core dump:
(qemu) qemu-kvm: ../util/yank.c:107: yank_unregister_instance: Assertion `QLIST_EMPTY(&entry->yankfns)' failed.
Aborted (core dumped)


And almost all tls cases are blocked by the error.

Thanks for filing https://bugzilla.redhat.com/show_bug.cgi?id=1964326 for that crash.

Upstream feature already present in qemu-6.0.
Marked as TestOnly and moved directly to ON_QA

Test 'yank' command about scenario 2) migration network hit a failure on the latest RHELAV-8.5.0 (), the result also pass.

Test environment:
hosts: kernel-4.18.0-310.el8.x86_64 & qemu-kvm-6.0.0-18.module+el8.5.0+11243+5269aaa1.x86_64

Test scenarios: 
2)migration network hit a failure on dst host when migration is active, then migration hang, use yank to fail migration. 

Test steps:
1.Boot a guest on src host;
2.Boot a guest on dst host with '-incoming defer';
3.Set migration incoming on dst host via qmp cmd;
{"execute":"migrate-incoming","arguments":{"uri":"tcp:[::]:1234"}}
4.Start migration on src host via qmp cmd;
{"execute": "migrate","arguments":{"uri": "tcp:$dst_migration_ip:1234"}}
5.During migration is active, down migration network on dst host:
# nmcli con down ens1f1
6.After migration hang(query migrate, only the total time of migration is increasing, other migration params stay unchanged), use yank cmd to fail migration on src host:
{ "execute": "query-yank" }
{"return": [{"type": "chardev", "id": "qmp_id_qmpmonitor1"}, {"type": "chardev", "id": "qmp_id_catch_monitor"}, {"type": "chardev", "id": "compat_monitor0"}, {"type": "chardev", "id": "serial0"}, {"type": "migration"}]}
{"execute":"yank","arguments":{"instances":[{"type":"migration"}]}}

Actual result:
After step 6, query migration status, get failed status on src host via qmp cmd, I think the result is expected:
{"execute":"query-migrate"}
{"return": {"blocked": false, "status": "failed", "error-desc": "Unable to write to socket: Broken pipe"}}

Next quit qemu process on dst host by manually, then restore migration network on dst host, we can start migration again and vm works well on dst host after migration.


According to Comment 4 && Comment 12, I have tested yank under two scenarios, all they pass:
1) inject firewall via drop on dst host when migration is active, then migration hang, use yank to fail migration. 
2) migration network hit a failure on dst host when migration is active, then migration hang, use yank to fail migration. 



Question 1:
Do we support yank under multifd migration?
I see multifd [1] file is added some yank codes. but when test multifd migration under scenario 2), doesn't get expected result like precopy. I will add a new comment 13 to describe the test with yank and multifd. 

Question 2:
I think Scenario 2) is a better test scenario, so I will add a case 'use yank to fail migration when migration network hit a failure' for precopy migration, is it ok? (BTW, if we support multifd with yank, I will also add one corresponding case.)


[1]
$ git show b5eea99ec2f5cf6fa0ac12a757c8873b1d2a73a4
...
diff --git a/migration/multifd.c b/migration/multifd.c
index 45c690aa11..1a1e589064 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -25,6 +25,9 @@
 #include "trace.h"
 #include "multifd.h"
 
+#include "qemu/yank.h"
+#include "io/channel-socket.h"
...

When test multifd with yank similar with Comment 12 but enabled multifd capability
1) after network down, execute 'yank' to fail migration:
Result: wait about 10 mins, get migration failed:
{"execute":"query-migrate"}
{"return": {"blocked": false, "status": "failed", "error-desc": "Unable to write to socket: No route to host"}}
2) after network down, wait several seconds then recover network, execute 'yank' to fail migration:
Result: will get migration failed at once:
{"execute":"query-migrate"}
{"return": {"blocked": false, "status": "failed", "error-desc": "Unable to write to socket: Broken pipe"}}


I'm confused if we support yank with multifd, why we can't get migration failed at once after yank, and the error-desc is different? and why when recover network then we will get migration failed at once after yank?


Notes: all qmp commands are sent under 'OOB' capability in Commment 12 & 13

Please note, that the hanging migration which this bug was cloned from actually hangs on a 'blockdev-del' of a NBD used for copying over disks/storage, so in this context.

Also note that the exact steps to make the connection stuck aren't probably known yet. It's not as simple as disabling an interface as in the case when 'blockdev-del' hung a proxy was involved and probably misbehaved.

(In reply to Peter Krempa from comment #14)
> Please note, that the hanging migration which this bug was cloned from
> actually hangs on a 'blockdev-del' of a NBD used for copying over
> disks/storage, so in this context.
> 

Thanks. 
Zixi, need your verify about yank on NBD here. 


> Also note that the exact steps to make the connection stuck aren't probably
> known yet. It's not as simple as disabling an interface as in the case when
> 'blockdev-del' hung a proxy was involved and probably misbehaved.

I could now thought above two scenarios to test yank from migration perspective. 
And I can update my test scenarios if get the exact and better steps to make connection stuck.
But if we need more test steps about NBD, Zixi maybe could help do it.

(In reply to Li Xiaohui from comment #12)
> Test 'yank' command about scenario 2) migration network hit a failure on the
> latest RHELAV-8.5.0 (), the result also pass.
> 
> Test environment:
> hosts: kernel-4.18.0-310.el8.x86_64 &
> qemu-kvm-6.0.0-18.module+el8.5.0+11243+5269aaa1.x86_64
> 
> Test scenarios: 
> 2)migration network hit a failure on dst host when migration is active, then
> migration hang, use yank to fail migration. 
> 
> Test steps:
> 1.Boot a guest on src host;
> 2.Boot a guest on dst host with '-incoming defer';
> 3.Set migration incoming on dst host via qmp cmd;
> {"execute":"migrate-incoming","arguments":{"uri":"tcp:[::]:1234"}}
> 4.Start migration on src host via qmp cmd;
> {"execute": "migrate","arguments":{"uri": "tcp:$dst_migration_ip:1234"}}
> 5.During migration is active, down migration network on dst host:
> # nmcli con down ens1f1
> 6.After migration hang(query migrate, only the total time of migration is
> increasing, other migration params stay unchanged), use yank cmd to fail
> migration on src host:
> { "execute": "query-yank" }
> {"return": [{"type": "chardev", "id": "qmp_id_qmpmonitor1"}, {"type":
> "chardev", "id": "qmp_id_catch_monitor"}, {"type": "chardev", "id":
> "compat_monitor0"}, {"type": "chardev", "id": "serial0"}, {"type":
> "migration"}]}
> {"execute":"yank","arguments":{"instances":[{"type":"migration"}]}}
> 
> Actual result:
> After step 6, query migration status, get failed status on src host via qmp
> cmd, I think the result is expected:
> {"execute":"query-migrate"}
> {"return": {"blocked": false, "status": "failed", "error-desc": "Unable to
> write to socket: Broken pipe"}}
> 
> Next quit qemu process on dst host by manually, then restore migration
> network on dst host, we can start migration again and vm works well on dst
> host after migration.
> 
> 
> According to Comment 4 && Comment 12, I have tested yank under two
> scenarios, all they pass:
> 1) inject firewall via drop on dst host when migration is active, then
> migration hang, use yank to fail migration. 
> 2) migration network hit a failure on dst host when migration is active,
> then migration hang, use yank to fail migration. 
> 
> 
> 
> Question 1:
> Do we support yank under multifd migration?
> I see multifd [1] file is added some yank codes. but when test multifd
> migration under scenario 2), doesn't get expected result like precopy. I
> will add a new comment 13 to describe the test with yank and multifd. 

If it's taking 10 minutes to report the failure in multifd that sounds like a bug in multifd's yank code.
Please take that as a separate bz just for multifd.

> Question 2:
> I think Scenario 2) is a better test scenario, so I will add a case 'use
> yank to fail migration when migration network hit a failure' for precopy
> migration, is it ok? (BTW, if we support multifd with yank, I will also add
> one corresponding case.)

I think the firewalling mechanism is probably good - as long as you tell the destination to 'drop' packets

> 
> [1]
> $ git show b5eea99ec2f5cf6fa0ac12a757c8873b1d2a73a4
> ...
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 45c690aa11..1a1e589064 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -25,6 +25,9 @@
>  #include "trace.h"
>  #include "multifd.h"
>  
> +#include "qemu/yank.h"
> +#include "io/channel-socket.h"
> ...

(In reply to Dr. David Alan Gilbert from comment #16)
> (In reply to Li Xiaohui from comment #12)
> > Test 'yank' command about scenario 2) migration network hit a failure on the
> > latest RHELAV-8.5.0 (), the result also pass.
> > 
> > Question 1:
> > Do we support yank under multifd migration?
> > I see multifd [1] file is added some yank codes. but when test multifd
> > migration under scenario 2), doesn't get expected result like precopy. I
> > will add a new comment 13 to describe the test with yank and multifd. 
> 
> If it's taking 10 minutes to report the failure in multifd that sounds like
> a bug in multifd's yank code.
> Please take that as a separate bz just for multifd.

Filed a new bz:
Bug 1970337 - Fail to get migration failure immediately if yank under multifd migration


> 
> > Question 2:
> > I think Scenario 2) is a better test scenario, so I will add a case 'use
> > yank to fail migration when migration network hit a failure' for precopy
> > migration, is it ok? (BTW, if we support multifd with yank, I will also add
> > one corresponding case.)
> 
> I think the firewalling mechanism is probably good - as long as you tell the
> destination to 'drop' packets
> 

Got it, thank you.

Verified from NBD.

yank cmmond can quit the blocked migration, when network issue happened between the migrating destination host and the nbd server.

Version:
qemu-kvm-6.0.0-18.module+el8.5.0+11243+5269aaa1.x86_64
kernel-4.18.0-310.el8.x86_64

Steps and results:
1.Boot a guest on src host;
2.Boot a guest on dst host with '-incoming defer';
3.Set migration incoming on dst host via qmp cmd;
{"execute":"migrate-incoming","arguments":{"uri":"tcp:[::]:1234"}}
4.Start migration on src host via qmp cmd;
{"execute": "migrate","arguments":{"uri": "tcp:$ip:1234"}}
5.During migration is active, inject firewall via drop on nbd server:
# iptables -A INPUT -p tcp --dport 10820 -j DROP
6.the migration hangs:
{ "execute": "query-yank" }
{"return": [{"type": "chardev", "id": "qmp_id_qmpmonitor1"}, {"type": "chardev", "id": "qmp_id_catch_monitor"}, {"type": "chardev", "id": "compat_monitor0"}, {"node-name": "nbd_image1", "type": "block-node"}, {"type": "chardev", "id": "serial0"}, {"type": "migration"}]}
{"execute":"yank","arguments":{"instances":[{"type":"migration"}]}}
{"return": {}}
{"execute":"query-migrate"}
{"return": {"blocked": false, "status": "failed", "error-desc": "Unable to write to socket: Broken pipe"}}

Drop the iptables rule and quit the qemu process, then start over the migration, it can success.

Thank you Zixi for helping testing.
I would mark this bz verified per Comment 4 & 12 & 13 & 16 & 17.

About NBD issue in Bug 1945532 - VM migration halts occasionally:
1.We aren't clear how to reproduce bz now, so shouldn't block here;
2.'yank' command support is a separate request, not a solution to bz 1945532 like said in the comment: https://bugzilla.redhat.com/show_bug.cgi?id=1945532#c69
3.We can continue to track bz 1945532, NBD QE zixchen and Storage vm migration QE aliang will go on help testing if needed (I have cc them, they will track the issue)

David, sorry for hijacking this bug and please reroute me to appropriate person, but I found out that when I set up a NDB copy job like this:

2021-07-01 15:01:12.675+0000: 29332: info : qemuMonitorSend:958 : QEMU_MONITOR_SEND_MSG: mon=0x7fd978008470 msg={"execute":"blockdev-add","arguments":{"driver":"nbd","server":{"type":"inet","host":"bart","port":"49153"},"export":"drive-sata0-0-0","node-name":"migration-sda-storage","read-only":false,"discard":"unmap"},"id":"libvirt-398"}

2021-07-01 15:01:12.678+0000: 29332: info : qemuMonitorSend:958 : QEMU_MONITOR_SEND_MSG: mon=0x7fd978008470 msg={"execute":"blockdev-mirror","arguments":{"job-id":"drive-sata0-0-0","device":"libvirt-1-format","target":"migration-sda-format","speed":9223372036853727232,"sync":"full","auto-finalize":true,"auto-dismiss":false},"id":"libvirt-400"}

and then let it run for a while and then yank it the block job does not change state. I mean, here's query-yank during the mirror job:

2021-07-01 15:01:18.251+0000: 29746: info : qemuMonitorIOWrite:436 : QEMU_MONITOR_IO_WRITE: mon=0x7fd978008470 buf={"execute":"query-yank","id":"libvirt-412"}
 len=45 ret=45 errno=0
2021-07-01 15:01:18.252+0000: 29746: debug : qemuMonitorJSONIOProcessLine:222 : Line [{"return": [{"type": "chardev", "id": "charmonitor"}, {"type": "chardev", "id": "charchannel0"}, {"node-name": "migration-sda-storage", "type": "block-node"}], "id": "libvirt-412"}]

And here's query-block-job BEFORE any yank:

2021-07-01 15:01:25.479+0000: 29746: info : qemuMonitorIOWrite:436 : QEMU_MONITOR_IO_WRITE: mon=0x7fd978008470 buf={"execute":"query-block-jobs","id":"libvirt-427"}
 len=51 ret=51 errno=0
2021-07-01 15:01:25.479+0000: 29746: debug : qemuMonitorJSONIOProcessLine:222 : Line [{"return": [{"auto-finalize": true, "io-status": "ok", "device": "drive-sata0-0-0", "auto-dismiss": false, "busy": true, "len": 21474836480, "offset": 0, "status": "running", "paused": false, "speed": 9223372036853727232, "ready": false, "type": "mirror"}], "id": "libvirt-427"}]


At this point, yank is performed:

2021-07-01 15:01:25.690+0000: 29746: info : qemuMonitorIOWrite:436 : QEMU_MONITOR_IO_WRITE: mon=0x7fd978008470 buf={"execute":"yank","arguments":{"instances":[{"type":"block-node","node-name":"migration-sda-storage"}]},"id":"libvirt-428"}
 len=125 ret=125 errno=0
2021-07-01 15:01:25.691+0000: 29746: debug : qemuMonitorJSONIOProcessLine:222 : Line [{"return": {}, "id": "libvirt-428"}]


But the mirror job seems to be not affected:

2021-07-01 15:01:25.980+0000: 29746: info : qemuMonitorIOWrite:436 : QEMU_MONITOR_IO_WRITE: mon=0x7fd978008470 buf={"execute":"query-block-jobs","id":"libvirt-429"}
 len=51 ret=51 errno=0
2021-07-01 15:01:25.981+0000: 29746: debug : qemuMonitorJSONIOProcessLine:222 : Line [{"return": [{"auto-finalize": true, "io-status": "ok", "device": "drive-sata0-0-0", "auto-dismiss": false, "busy": true, "len": 21474836480, "offset": 0, "status": "running", "paused": false, "speed": 9223372036853727232, "ready": false, "type": "mirror"}], "id": "libvirt-429"}]

(these are the very same values as before yank). Is this behaviour expected?

I don't know nbd stuff, lets bounce to eblake

(In reply to Michal Privoznik from comment #20)

> and then let it run for a while and then yank it the block job does not
> change state. I mean, here's query-yank during the mirror job:
> 

> 
> And here's query-block-job BEFORE any yank:
> 

> 
> At this point, yank is performed:
> 

> 
> But the mirror job seems to be not affected:
> 
> 2021-07-01 15:01:25.980+0000: 29746: info : qemuMonitorIOWrite:436 :
> QEMU_MONITOR_IO_WRITE: mon=0x7fd978008470
> buf={"execute":"query-block-jobs","id":"libvirt-429"}
>  len=51 ret=51 errno=0
> 2021-07-01 15:01:25.981+0000: 29746: debug :
> qemuMonitorJSONIOProcessLine:222 : Line [{"return": [{"auto-finalize": true,
> "io-status": "ok", "device": "drive-sata0-0-0", "auto-dismiss": false,
> "busy": true, "len": 21474836480, "offset": 0, "status": "running",
> "paused": false, "speed": 9223372036853727232, "ready": false, "type":
> "mirror"}], "id": "libvirt-429"}]
> 
> (these are the very same values as before yank). Is this behaviour expected?

I'm not sure; I'll ask on the upstream list while investigating it on my end.

(In reply to Eric Blake from comment #22)
> I'm not sure; I'll ask on the upstream list while investigating it on my end.

Eric, any luck yet?

(In reply to Michal Privoznik from comment #24)
> (In reply to Eric Blake from comment #22)
> > I'm not sure; I'll ask on the upstream list while investigating it on my end.
> 
> Eric, any luck yet?

Eric, any updates?

(In reply to Eric Blake from comment #22)
> (In reply to Michal Privoznik from comment #20)
> 
> > and then let it run for a while and then yank it the block job does not
> > change state. I mean, here's query-yank during the mirror job:
> > 
> 

> > (these are the very same values as before yank). Is this behaviour expected?
> 
> I'm not sure; I'll ask on the upstream list while investigating it on my end.

Here's what I tested locally, using just qemu 6.1 and nbdkit as a data sink.  If I'm missing a step, or if this is a problem with earlier qemu, I have not yet pinpointed that, but the fact that the yank appears to work with latest upstream seems like expected behavior to me, so I'm not sure why you were not seeing the block job terminate.

$ # Create 100M scratch disk to be mirrored
$ dd if=/dev/urandom of=file6 bs=1M count=100
$ # Run nbdkit as the destination for the mirror job
$ nbdkit -f memory 100M
$ # Run a rudimenary qemu process visiting a single disk
$ eqmu-system-x86_64 -m 8192 -nodefaults -nographic \
 -qmp stdio -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 \
 -object iothread,id=iothread0 \
 -device virtio-scsi-pci,bus=bridge1,addr=0x1f,id=scsi0,iothread=iothread0 \
 -drive file=file6,if=none,cache=none,id=drive_disk0,aio=native,format=raw,werror=stop,rerror=stop \
 -device scsi-hd,drive=drive_disk0,bus=scsi0.0,id=disk0
{'execute':'qmp_capabilities'}
{'execute':'query-block-jobs'}
{'execute':'blockdev-add','arguments':{'driver':'nbd','server':{
 'type':'inet','host':'localhost','port':'10809'},'export':'dst',
 'node-name':'dst','read-only':false,'discard':'unmap'}}
{'execute':'query-yank'}

At this point, there are no block jobs, but yank shows the NBD node ('dst') ready to yank.

{'execute':'blockdev-mirror','arguments':{'job-id':'mirror0','device':
'drive_disk0','target':'dst','speed':1000000,'sync':'full',
'auto-finalize':true,'auto-dismiss':false}}

Now there is a slow-running block job (1M/s over 100M disk is enough time to see progress, but also to yank before it finishes)

{'execute':'yank','arguments':{'instances':[{'type':'block-node',
'node-name':'dst'}]}}

At this point, it took a little bit of time, perhaps because of the throttling, but fairly soon, I got lots of:

{"timestamp": {"seconds": 1631223281, "microseconds": 234463}, "event": "BLOCK_JOB_ERROR", "data": {"device": "mirror0", "operation": "write", "action": "report"}}

and query-block-jobs shows:

{"return": [{"auto-finalize": true, "io-status": "ok", "device": "mirror0", "auto-dismiss": false, "busy": false, "len": 104857600, "offset": 16777216, "status": "concluded", "paused": false, "speed": 1000000, "ready": false, "type": "mirror", "error": "Input/output error"}]}

It looks like upstream qemu is thus properly handling yank of an active NBD destination to a mirror block job.  If downstream is not, it seems like we should open that as a new bug, as it is distinct from the original analysis of this bug.

Zixi, could you help check the function yank+NBD on the downstream according to Comment 20 and Comment 26?

hi Xiaohui, Tingting is the feature owner of NBD, I need info her to handle this, thanks.

(In reply to Eric Blake from comment #26)
> (In reply to Eric Blake from comment #22)
> > (In reply to Michal Privoznik from comment #20)

[...]

> It looks like upstream qemu is thus properly handling yank of an active NBD
> destination to a mirror block job.  If downstream is not, it seems like we
> should open that as a new bug, as it is distinct from the original analysis
> of this bug.

One thing that specifically needs to be considered in terms of migration is that you can have a job that is already synchronized and the VM is paused. Thus there won't be any transfer happening. Yet if you 'yank'-out the connection that job should fail as well. In this scenario theoretically libvirt should be able to figure that out beforehand, but in general usability sense, if the connection is disconnected the job should fail even when there won't be any further transfer happening.

As the issue is about block mirror, change needinfo to Aihua. Thanks.

Not sure if NBD yank take effect, it seems not from my test.

My test steps as bellows:

Test Env:
  kernel version:4.18.0-345.el8.x86_64
  qemu version:qemu-kvm-6.0.0-30.module+el8.5.0+12586+476da3e1

Test steps:
  1.Start src guest with qemu cmd:
     /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
    -chardev socket,server=on,id=qmp_id_qmpmonitor1,wait=off,path=/tmp/monitor-qmpmonitor1-20210927-235344-uQJJ6aFc  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server=on,id=qmp_id_catch_monitor,wait=off,path=/tmp/monitor-catch_monitor-20210927-235344-uQJJ6aFc  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idmerT3z \
    -chardev socket,server=on,id=chardev_serial0,wait=off,path=/tmp/serial-serial0-20210927-235344-uQJJ6aFc \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20210927-235344-uQJJ6aFc,path=/tmp/seabios-20210927-235344-uQJJ6aFc,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20210927-235344-uQJJ6aFc,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel850-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:c0:90:a7:2f:40,id=idjZ1FPm,netdev=id4rOUG1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=id4rOUG1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
    -monitor stdio \
    -qmp tcp:0:3000,server=on,wait=off \

  2. Start dst guest with qemu cmd:
      /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
    -chardev socket,server=on,id=qmp_id_qmpmonitor1,wait=off,path=/tmp/monitor-qmpmonitor1-20210927-235344-uQJJ6aFc  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server=on,id=qmp_id_catch_monitor,wait=off,path=/tmp/monitor-catch_monitor-20210927-235344-uQJJ6aFc  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idmerT3z \
    -chardev socket,server=on,id=chardev_serial0,wait=off,path=/tmp/serial-serial0-20210927-235344-uQJJ6aFc \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20210927-235344-uQJJ6aFc,path=/tmp/seabios-20210927-235344-uQJJ6aFc,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20210927-235344-uQJJ6aFc,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/mirror.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:c0:90:a7:2f:40,id=idjZ1FPm,netdev=id4rOUG1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=id4rOUG1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
    -monitor stdio \
    -qmp tcp:0:3000,server=on,wait=off \
    -incoming defer \

 3. Start ndb server and expose image in dst.
    { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet", "data": { "host": "10.73.114.14", "port": "3333" } } } }
    {"execute":"block-export-add","arguments":{"id": "export0", "node-name": "drive_image1", "type": "nbd", "writable": true}}

 4. In src, add exposed image of dst and do block mirror from src to dst.
     {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror","server":{"type":"inet","host":"10.73.114.14","port":"3333"},"export":"drive_image1"}}
{"return": {}}
     { "execute": "blockdev-mirror", "arguments": { "device": "drive_image1","target": "mirror", "sync": "full","job-id":"j1" } }
{"timestamp": {"seconds": 1632812310, "microseconds": 660507}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
{"timestamp": {"seconds": 1632812310, "microseconds": 660567}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
{"return": {}}

 5. Interrupt nbd connect by firewall setting in src.
    #iptables -A OUTPUT -d 10.73.114.14 -p tcp --dport 3333 -j REJECT

 6. Check block job status every 5 secs.
     {"execute":"query-block-jobs"}
{"return": [{"auto-finalize": true, "io-status": "ok", "device": "j1", "auto-dismiss": true, "busy": true, "len": 21474836480, "offset": 3223715840, "status": "running", "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}

 7. Yank NBD node in src.
    {'execute':'yank','arguments':{'instances':[{'type':'block-node', 'node-name':'mirror'}]}}
    {"return": {}}

 8. Check block job status.
    {"execute":"query-block-jobs"}
{"return": [{"auto-finalize": true, "io-status": "ok", "device": "j1", "auto-dismiss": true, "busy": true, "len": 21474836480, "offset": 3223715840, "status": "running", "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}

 After NBD yank, block job still in running status after one and a half hour.

 So, I'm not sure if the test steps are correct or the result is just as expected?

(In reply to aihua liang from comment #32)
> Not sure if NBD yank take effect, it seems not from my test.
> 
>  4. In src, add exposed image of dst and do block mirror from src to dst.
>     
> {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror",
> "server":{"type":"inet","host":"10.73.114.14","port":"3333"},"export":
> "drive_image1"}}
> {"return": {}}
>      { "execute": "blockdev-mirror", "arguments": { "device":
> "drive_image1","target": "mirror", "sync": "full","job-id":"j1" } }
> {"timestamp": {"seconds": 1632812310, "microseconds": 660507}, "event":
> "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
> {"timestamp": {"seconds": 1632812310, "microseconds": 660567}, "event":
> "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
> {"return": {}}
> 
>  5. Interrupt nbd connect by firewall setting in src.
>     #iptables -A OUTPUT -d 10.73.114.14 -p tcp --dport 3333 -j REJECT
> 
>  6. Check block job status every 5 secs.
>      {"execute":"query-block-jobs"}
> {"return": [{"auto-finalize": true, "io-status": "ok", "device": "j1",
> "auto-dismiss": true, "busy": true, "len": 21474836480, "offset":
> 3223715840, "status": "running", "paused": false, "speed": 0, "ready":
> false, "type": "mirror"}]}

I'm presuming that the "len" member is growing over the course of these checks, proving that something is being copied, but that there is still more to go for the mirroring to be complete.

> 
>  7. Yank NBD node in src.
>     {'execute':'yank','arguments':{'instances':[{'type':'block-node',
> 'node-name':'mirror'}]}}
>     {"return": {}}
> 
>  8. Check block job status.
>     {"execute":"query-block-jobs"}
> {"return": [{"auto-finalize": true, "io-status": "ok", "device": "j1",
> "auto-dismiss": true, "busy": true, "len": 21474836480, "offset":
> 3223715840, "status": "running", "paused": false, "speed": 0, "ready":
> false, "type": "mirror"}]}
> 
>  After NBD yank, block job still in running status after one and a half hour.
> 
>  So, I'm not sure if the test steps are correct or the result is just as
> expected?

That does look fishy, in that the firewall should cause NBD packets to time out, and then the yank should remove the NBD target entirely which should cause the job to detect errors, rather than remaining as running.  Given that your testing on downstream did not reproduce the "error": "Input/output error" that I saw when testing upstream, it looks like we are missing a backported patch downstream; we'll need a separate BZ opened to track what that is.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684