Bug 1497120 - migration+new block migration race: bdrv_co_do_pwritev: Assertion `!(bs->open_flags & 0x0800)' failed
Summary: migration+new block migration race: bdrv_co_do_pwritev: Assertion `!(bs->open...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Dr. David Alan Gilbert
QA Contact: xianwang
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1506494
TreeView+ depends on / blocked
 
Reported: 2017-09-29 08:57 UTC by Dr. David Alan Gilbert
Modified: 2018-04-11 00:37 UTC (History)
12 users (show)

(edit)
Clone Of:
: 1506494 (view as bug list)
(edit)
Last Closed: 2018-04-11 00:36:03 UTC


Attachments (Terms of Use)
Raw QMP test procedure with NBD-based migration with 'pause-before-switchover' (6.46 KB, text/plain)
2017-11-15 11:38 UTC, Kashyap Chamarthy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1104 None None None 2018-04-11 00:37 UTC

Description Dr. David Alan Gilbert 2017-09-29 08:57:13 UTC
Description of problem:
There's a race between the end of migration (and it inactivating disks) and the end of drive-mirroring that can lead to a fatal assert of the source.
It seems rare to trigger, but the theory is that it happens if there's write activity right near the end of migration as we stop the CPUs then those writes will take a little time to bubble through the drive-mirror, but if we hit the end of the migration code before then we end up calling bdrv_inactivate_all before those writes are done.

Version-Release number of selected component (if applicable):
Found upstream on >2.10, bz 1456086 might be this and hit it on 2.9

How reproducible:
Reported upstream, possible we've seen it once

Steps to Reproduce:
1. Not yet known
2.
3.

Actual results:
Source crashes with bdrv_co_do_pwritev: Assertion `!(bs->open_flags & 0x0800)' failed

Expected results:
A successful migration

Additional info:
This was reported upstream by wangjie88@huawei.com but could have also been https://bugzilla.redhat.com/show_bug.cgi?id=1456086  which was closed because we didn't find a way to reproduce it

Current discussions with kwolf are that we're going to need some changes at the visible interface to libvirt to fix this; e.g. a new migration state and extra command.

Comment 1 Dr. David Alan Gilbert 2017-10-20 09:47:45 UTC
This seems to have settled out OK upstream; see todays:

  [PATCH v4 0/7] migration: pause-before-switchover

Comment 2 Dr. David Alan Gilbert 2017-10-25 16:25:54 UTC
upstream commits 93fbd0314ec060ffaf90..0331c8cabf6168aa263a

Comment 4 Miroslav Rezanina 2017-11-02 10:09:35 UTC
Fix included in qemu-kvm-rhev-2.10.0-4.el7

Comment 6 xianwang 2017-11-06 08:27:59 UTC
Hi, Dave,
I have found some information about your patch, that in this link: http://patchwork.ozlabs.org/cover/827760/
there is two work flow as said in this link,
The precopy flow is:
active->pre-switchover->device->completed

The postcopy flow is:
active->pre-switchover->postcopy-active->completed

But I didn't know the detail test steps for this bug, could you give me the detail test steps and expected results for these two scenarios? thanks

Comment 7 xianwang 2017-11-14 08:19:22 UTC
Hi, Dave and kwolf,
according to my understand and referring to http://patchwork.ozlabs.org/cover/827760/, the following is my test steps for this scenario, could you help to review whether this bug could be verified as following steps, thanks:
version:
Host:
3.10.0-776.el7.ppc64le
qemu-kvm-rhev-2.10.0-5.el7.ppc64le
SLOF-20170724-2.git89f519f.el7.noarch

Guest:
3.10.0-776.el7.ppc64

steps:
1.Boot a guest on src host with drive:
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/RHEL74_BE.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0,lun=0,bootindex=0 \
    -qmp tcp:0:8881,server,nowait \
2.Launch listening mode on dst host, with same qemu cli as src end but different disk:
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/mount_point/block.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0,lun=0,bootindex=0 \
3.On src host, do drive-mirror job via qmp:
{"execute":"qmp_capabilities"}
{ "execute": "drive-mirror", "arguments": { "device": "drive_image1", "target": "/home/block.qcow2", "format": "qcow2", "mode": "absolute-paths", "sync": "full", "on-source-error": "stop", "on-target-error": "stop"} }
after a while, block job becomes ready automatically:
{"timestamp": {"seconds": 1510641851, "microseconds": 178028}, "event": "BLOCK_JOB_READY", "data": {"device": "drive_image1", "len": 10142023680, "offset": 10142023680, "speed": 0, "type": "mirror"}}
4.In guest, execute "stress":
# stress --cpu 4 --io 8 --vm 2 --vm-bytes 128M
5.On src host, do live migration with enabling "pause-before-switchover":
(qemu) migrate_set_capability pause-before-switchover on
(qemu) info migrate_capabilities 
xbzrle: off
rdma-pin-all: off
auto-converge: off
zero-blocks: off
compress: off
events: off
postcopy-ram: off
x-colo: off
release-ram: off
return-path: off
pause-before-switchover: on
(qemu) migrate -d tcp:10.16.42.48:5801
(qemu) info migrate
Migration status: active
in order to make "migrate status" to "pre-switchover" quickly, I enlarge the value of "downtime" and "speed"
(qemu) migrate_set_downtime 10
(qemu) migrate_set_speed 1G
(qemu) info migrate
Migration status: pre-switchover
(qemu) info status 
VM status: paused (finish-migrate)
6.On src host, do block-job-complete
{"execute":"block-job-complete","arguments":{"device":"drive_image1"}}
{"timestamp": {"seconds": 1510642464, "microseconds": 566489}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len": 10163388416, "offset": 10163388416, "speed": 0, "type": "mirror"}}
7.On src host
(qemu) migrate_continue pre-switchover

Result:
Block job and migration completed and vm works well.
On src host:
(qemu) info migrate
globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off return-path: off pause-before-switchover: on 
Migration status: completed
(qemu) info status 
VM status: paused (postmigrate)

On dst host:
(qemu) info status 
VM status: running

1)Due to after "block-job-complete", data will only write to target disk while live migration is in process, I don't know whether this steps are right, I just wonder if we don't execute "block-job-complete" before "migrate_continue pre-switchover", because once live migration complete and quit qemu of src end, block job would complete automatically.

2)For cancelling migration scenarios, which phases could cancel live migration?
at least, during post-copy mode, migration can't be cancelled.

Thanks

Comment 8 Dr. David Alan Gilbert 2017-11-14 10:49:07 UTC
(In reply to xianwang from comment #7)
> Hi, Dave and kwolf,
> according to my understand and referring to
> http://patchwork.ozlabs.org/cover/827760/, the following is my test steps
> for this scenario, could you help to review whether this bug could be
> verified as following steps, thanks:
> version:
> Host:
> 3.10.0-776.el7.ppc64le
> qemu-kvm-rhev-2.10.0-5.el7.ppc64le
> SLOF-20170724-2.git89f519f.el7.noarch
> 
> Guest:
> 3.10.0-776.el7.ppc64
> 
> steps:
> 1.Boot a guest on src host with drive:
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/home/RHEL74_BE.qcow2 \
>     -device
> scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0,lun=0,
> bootindex=0 \
>     -qmp tcp:0:8881,server,nowait \
> 2.Launch listening mode on dst host, with same qemu cli as src end but
> different disk:
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/home/mount_point/block.qcow2 \
>     -device
> scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0,lun=0,
> bootindex=0 \
> 3.On src host, do drive-mirror job via qmp:
> {"execute":"qmp_capabilities"}
> { "execute": "drive-mirror", "arguments": { "device": "drive_image1",
> "target": "/home/block.qcow2", "format": "qcow2", "mode": "absolute-paths",
> "sync": "full", "on-source-error": "stop", "on-target-error": "stop"} }
> after a while, block job becomes ready automatically:
> {"timestamp": {"seconds": 1510641851, "microseconds": 178028}, "event":
> "BLOCK_JOB_READY", "data": {"device": "drive_image1", "len": 10142023680,
> "offset": 10142023680, "speed": 0, "type": "mirror"}}

I think that's probably the wrong way to do a drive-mirror; I was expecting to see an nbd-server setup with a mirror with that.
See: https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs/live-block-operations.html#qmp-invocation-for-live-storage-migration-with-drive-mirror-nbd

(although you don't need any backing chain)

> 4.In guest, execute "stress":
> # stress --cpu 4 --io 8 --vm 2 --vm-bytes 128M

I don't think you need much of a stress test for this; just the io stress side should be enough.

> 5.On src host, do live migration with enabling "pause-before-switchover":
> (qemu) migrate_set_capability pause-before-switchover on
> (qemu) info migrate_capabilities 
> xbzrle: off
> rdma-pin-all: off
> auto-converge: off
> zero-blocks: off
> compress: off
> events: off
> postcopy-ram: off
> x-colo: off
> release-ram: off
> return-path: off
> pause-before-switchover: on
> (qemu) migrate -d tcp:10.16.42.48:5801
> (qemu) info migrate
> Migration status: active
> in order to make "migrate status" to "pre-switchover" quickly, I enlarge the
> value of "downtime" and "speed"
> (qemu) migrate_set_downtime 10
> (qemu) migrate_set_speed 1G
> (qemu) info migrate
> Migration status: pre-switchover
> (qemu) info status 
> VM status: paused (finish-migrate)

That downtime is a bit long; if you're not running the full stress then you should be able to set that downtime to a more sensible .5

> 6.On src host, do block-job-complete
> {"execute":"block-job-complete","arguments":{"device":"drive_image1"}}
> {"timestamp": {"seconds": 1510642464, "microseconds": 566489}, "event":
> "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len":
> 10163388416, "offset": 10163388416, "speed": 0, "type": "mirror"}}

Hmm is that right? Or do we need a block-job-cancel rather than complete for a migrate?

> 7.On src host
> (qemu) migrate_continue pre-switchover
> 
> Result:
> Block job and migration completed and vm works well.
> On src host:
> (qemu) info migrate
> globals: store-global-state=1, only_migratable=0, send-configuration=1,
> send-section-footer=1
> capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks:
> off compress: off events: off postcopy-ram: off x-colo: off release-ram: off
> return-path: off pause-before-switchover: on 
> Migration status: completed
> (qemu) info status 
> VM status: paused (postmigrate)
> 
> On dst host:
> (qemu) info status 
> VM status: running
> 
> 1)Due to after "block-job-complete", data will only write to target disk
> while live migration is in process, I don't know whether this steps are
> right, I just wonder if we don't execute "block-job-complete" before
> "migrate_continue pre-switchover", because once live migration complete and
> quit qemu of src end, block job would complete automatically.
>

We need to check this with kwolf.

> 2)For cancelling migration scenarios, which phases could cancel live
> migration?
> at least, during post-copy mode, migration can't be cancelled.

Yes, for the test above we should be checking cancel in precopy before and after it's entered the new pause state.

Dave
> 
> Thanks

Comment 9 Kashyap Chamarthy 2017-11-14 14:23:57 UTC
(In reply to Dr. David Alan Gilbert from comment #8)
> (In reply to xianwang from comment #7)

[...]

> > 3.On src host, do drive-mirror job via qmp:
> > {"execute":"qmp_capabilities"}
> > { "execute": "drive-mirror", "arguments": { "device": "drive_image1",
> > "target": "/home/block.qcow2", "format": "qcow2", "mode": "absolute-paths",
> > "sync": "full", "on-source-error": "stop", "on-target-error": "stop"} }
> > after a while, block job becomes ready automatically:
> > {"timestamp": {"seconds": 1510641851, "microseconds": 178028}, "event":
> > "BLOCK_JOB_READY", "data": {"device": "drive_image1", "len": 10142023680,
> > "offset": 10142023680, "speed": 0, "type": "mirror"}}
> 
> I think that's probably the wrong way to do a drive-mirror; I was expecting
> to see an nbd-server setup with a mirror with that.

Yes, you're right.  NBD server setup is required indeed.  The below link
does talk about it.

> See:
> https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs/live-block-
> operations.html#qmp-invocation-for-live-storage-migration-with-drive-mirror-
> nbd
> 
> (although you don't need any backing chain)

[...]

> > 6.On src host, do block-job-complete
> > {"execute":"block-job-complete","arguments":{"device":"drive_image1"}}
> > {"timestamp": {"seconds": 1510642464, "microseconds": 566489}, "event":
> > "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len":
> > 10163388416, "offset": 10163388416, "speed": 0, "type": "mirror"}}
> 
> Hmm is that right? Or do we need a block-job-cancel rather than complete for
> a migrate?

Yes, `block-job-cancel` is what libvirt uses in its `drive-mirror` + NBD
workflow.

While a ``drive-mirror` job is active, and it has issued a
BLOCK_JOB_READY event, signalling that synchronization of source and
destination has completed.  And you can take one of the two following
action:

  (1) By issuing `block-job-complete`, the synchronization will stop,
      and live QEMU will be pivoted to the copy.  (Event:
      BLOCK_JOB_COMPLETED) 
  
  (2) By issuing `block-job-cancel`, the synchronization will stop, but
      live QEMU will NOT be pivoted to the copy.  But what you end up
      with is a point-in-time (which is at the time of _triggering_ the
      'cancel' command) copy.  (Event: BLOCK_JOB_CANCELLED)

In the NBD workflow, after `block-job-cancel`, you also stop the NBD
server on the destination, by issuing QMP `nbd-server-stop`.

Comment 10 Kashyap Chamarthy 2017-11-15 11:38 UTC
Created attachment 1352558 [details]
Raw QMP test procedure with NBD-based migration with 'pause-before-switchover'

This was tested with upstream QEMU:

{  
  "QMP":{  
    "version":{  
      "qemu":{  
        "micro":90,
        "minor":10,
        "major":2
      },
      "package":" (v2.11.0-rc0-47-g55ed8d6)"
    },
    "capabilities":[  

    ]
  }
}

Comment 11 xianwang 2017-11-17 12:57:19 UTC
Hi, Dave and kwolf,
FYI and that link, I have re-test this scenario as following:
version:
3.10.0-768.el7.ppc64le
qemu-kvm-rhev-2.10.0-6.el7.ppc64le
SLOF-20170724-2.git89f519f.el7.noarch

localmigration:

src end:
1.boot guest:
    -drive id=drive_image1,if=none,aio=threads,cache=none,format=qcow2,file=/home/test.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,addr=07,bootindex=0 \

check:
(qemu) info block
drive_image1 (#block173): /home/test.qcow2 (qcow2)
    Attached to:      /machine/peripheral/image1/virtio-backend
    Cache mode:       writeback, direct


2.dst end:
firstly, create an qcow2 as exported disk:
#qemu-img create -f qcow2 nbd.qcow2 20G
then boot guest:
    -blockdev node-name=node-TargetDisk,driver=qcow2,file.driver=file,file.node-name=file,file.filename=/home/nbd.qcow2 \
    -device virtio-blk,id=virtio0,drive=node-TargetDisk \

check:
(qemu) info block
node-TargetDisk: /home/nbd.qcow2 (qcow2)
    Attached to:      /machine/peripheral/virtio0/virtio-backend
    Cache mode:       writeback

3.set dst as nbd server:
# telnet 127.0.0.1 8882
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":{"host":"127.0.0.1","port":"12345"}}}}
{"return": {}}
{"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk"}}
{"return": {}}

4.in src end, do drive-mirror
# telnet 127.0.0.1 8881
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute": "drive-mirror","arguments":{"device":"drive_image1","mode":"existing","job-id":"job0","target":"nbd:127.0.0.1:12345:exportname=node-TargetDisk","sync":"top","format":"qcow2"}}
{"error": {"class": "GenericError", "desc": "Image is not in qcow2 format"}}

{"execute": "drive-mirror","arguments":{"device":"drive_image1","mode":"existing","job-id":"job0","target":"nbd:127.0.0.1:12345:exportname=node-TargetDisk","sync":"top"}}
{"return": {}}
{"timestamp": {"seconds": 1510922579, "microseconds": 879307}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 879787}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 882765}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 884881}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 887108}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 889323}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 890445}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 891936}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 893544}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 893579}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 894186}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 895497}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 895883}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 896305}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 897920}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 899138}, "event": "BLOCK_JOB_ERROR", "data": {"device": "job0", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1510922579, "microseconds": 899443}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 21474836480, "offset": 0, "speed": 0, "type": "mirror", "error": "Operation not permitted"}}

on src end:
(qemu) WARNING: Image format was not specified for 'nbd://127.0.0.1:12345/node-TargetDisk' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

on dst end:
(qemu) qemu-kvm: Server is read-only, return error

please help to check it, thanks

Comment 12 Dr. David Alan Gilbert 2017-11-17 13:05:38 UTC
I think the problem here is:

 {"execute": "drive-mirror","arguments": {"device":"drive_image1","mode":"existing","job id":"job0","target":"nbd:127.0.0.1:12345:exportname=node-TargetDisk","sync":"top","format":"qcow2"}}

and the error:

{"error": {"class": "GenericError", "desc": "Image is not in qcow2 format"}}

the drive-mirror should be told the image format is raw even if the backing is really qcow2, because it's sent raw over the NBD; see kashyap's docs has:

{ "execute": "drive-mirror", "arguments": { "device": "virtio0", "target": "nbd:localhost:3333:exportname=virtio0", "sync": "full","format": "raw", "mode": "existing" } }

Comment 13 xianwang 2017-11-17 17:35:22 UTC
Hi, Dave and kwolf,
I)First:
For nbd server is not writable, the resolution maybe is that we need to add "write" permission for nbd server, but there is one little problem about qmp:
-HMP
(qemu) nbd_server_add -w node-TargetDisk  *******works well
-QMP
{"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk","writable":"true"}}
{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'writable', expected: boolean"}}
{"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk","boolean":"true"}}
{"error": {"class": "GenericError", "desc": "Parameter 'boolean' is unexpected"}}
{"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk","writable":"boolean"}}
{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'writable', expected: boolean"}}

so, I don't know what is the right qmp command

II)Second:
I have re-test four scenarios, all are local migrations, but only one scenario is pass, other three are all failed.

pass:
a)src end:
    -drive id=drive_image1,if=none,aio=threads,cache=none,format=qcow2,file=/home/test.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,addr=07,bootindex=0 \

check block and enable pause-before-switchover:
(qemu) info block
drive_image1 (#block143): /home/test.qcow2 (qcow2)
    Attached to:      /machine/peripheral/image1/virtio-backend
    Cache mode:       writeback, direct
(qemu) migrate_set_capability pause-before-switchover on
(qemu) info migrate_capabilities 
xbzrle: off
rdma-pin-all: off
auto-converge: off
zero-blocks: off
compress: off
events: off
postcopy-ram: off
x-colo: off
release-ram: off
return-path: off
pause-before-switchover: on


b)dst end:
    -blockdev node-name=node-TargetDisk,driver=qcow2,file.driver=file,file.node-name=file,file.filename=/home/nbd.qcow2 \
    -device virtio-blk-pci,id=virtio0,drive=node-TargetDisk,bus=pci.0,addr=07,bootindex=0 \

check:
(qemu) info block
node-TargetDisk: /home/nbd.qcow2 (qcow2)
    Attached to:      /machine/peripheral/virtio0/virtio-backend
    Cache mode:       writeback

setup nbd server on dst host(make dst qemu as nbd server)
(qemu) nbd_server_start 127.0.0.1:12345
(qemu) nbd_server_add -w node-TargetDisk

c)on src end, begin to do migration and then begin to do block-mirror on dst end
src end:
(qemu) migrate -d tcp:127.0.0.1:5801
(qemu) info migrate
globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
Migration status: active

then, begin to do block-mirror on src end during live migration, after block job becomes "BLOCK_JOB_READY", cancel block job and stop nbd server
src end:
{"execute": "drive-mirror","arguments":{"device":"drive_image1","mode":"existing","job-id":"job0","target":"nbd:127.0.0.1:12345:exportname=node-TargetDisk","sync":"top"}}
{"return": {}}
{"timestamp": {"seconds": 1510937156, "microseconds": 30791}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 21475229696, "offset": 21475229696, "speed": 0, "type": "mirror"}}
{"execute": "block-job-cancel","arguments":{"device":"job0"}}
{"return": {}}

dst end:
(qemu) nbd_server_stop

d)check the status of migration on src end, and switch to "pre-switchover"
on src:
(qemu) info migrate
globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off return-path: off pause-before-switchover: on 
Migration status: pre-switchover
(qemu) info status 
VM status: paused (finish-migrate)
(qemu) migrate_continue pre-switchover
(qemu) info migrate
Migration status: completed
(qemu) info status 
VM status: paused (postmigrate)

on dst:
(qemu) info status 
VM status: running
vm works well and reboot works well.

other three scenarios are similar with above, the qemu cli are all same with above one, the difference between them is as following:

1)do drive-mirror job firstly, when block job becomes "ready", then cancel block job, then begin to do live migration, result is:migration complete, vm works well on dst end without error message in guest by "demesg", but reboot guest, the file system is broken as attachment guest;

2)do drive-mirror job, cancel job before it becomes "ready", then whether do live migration before or after cancelling block job, the result is same: migration complete, but guest can't be login via vnc and there is error massage in guest via "dmesg" as following:

[  318.990587] XFS (dm-0): metadata I/O error: block 0x1aa9860 ("xfs_trans_read_buf_map") error 117 numblks 32
[  318.990588] XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
[  318.992416] XFS (dm-0): Metadata CRC error detected at xfs_dir3_block_read_verify+0x78/0x1c0 [xfs], xfs_dir3_block block 0x1aad650
[  318.992422] XFS (dm-0): metadata I/O error: block 0x1aad650 ("xfs_trans_read_buf_map") error 74 numblks 8

guest reboot failed, vnc screenshot is same as attachment guest.

1)please check whether the test steps of the pass one is right, for it is pass, does it indicate this bug is fixed? 
2)I am not sure the failed three scenarios are illegal test scenarios or they are bugs.
Thanks very much

Comment 14 Dr. David Alan Gilbert 2017-11-17 17:39:53 UTC
xianwang:
  Please copy kashyap's example - json is very fussy;

{ "execute": "nbd-server-add", "arguments": { "device": "virtio0","writable": true } }

Note there are no " around the true

Comment 15 xianwang 2017-11-18 01:02:45 UTC
(In reply to Dr. David Alan Gilbert from comment #14)
> xianwang:
>   Please copy kashyap's example - json is very fussy;
> 
> { "execute": "nbd-server-add", "arguments": { "device":
> "virtio0","writable": true } }
> 
> Note there are no " around the true

Dave,
I have tried as:
{"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":{"host":"127.0.0.1","port":"12345"}}}}
{"return": {}}
{"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk","writable":true}}
{"return": {}}

it works, thanks.

But I have another question, for "pre-switchover", what is the right order of doing "live migration" and doing "block mirror" ? 
I think we should first do block-mirror, after block job becomes to "BLOCK_JOB_COMPLETED" whether by "`block-job-complete` or `block-job-cancel`, then begin do live migration, but this failed, on contrary, first do live migration, during live migration do block-mirror, it succeed, I am confused why

Comment 16 Kashyap Chamarthy 2017-11-21 08:48:17 UTC
(In reply to xianwang from comment #15)

[...]

> I have tried as:
> {"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":
> {"host":"127.0.0.1","port":"12345"}}}}
> {"return": {}}
> {"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk",
> "writable":true}}
> {"return": {}}
> 
> it works, thanks.
> 
> But I have another question, for "pre-switchover", what is the right order
> of doing "live migration" and doing "block mirror" ? 
> I think we should first do block-mirror, after block job becomes to
> "BLOCK_JOB_COMPLETED" whether by "`block-job-complete` or
> `block-job-cancel`, then begin do live migration, but this failed, on
> contrary, first do live migration, during live migration do block-mirror, it
> succeed, I am confused why

The right order using 'pause-before-switchover' is documented in the text file I attached in comment#10.

Copying here's the full sequence:

[I didn't copy / paste the first step intentionally; it just launches source & destination QEMU -- refer the file in comment#10].


(2) [SOURCE] Set the "pause-before-switchover" capability.
-----------------------------------------------------------------------
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"pause-before-switchover","state":true}]}}
{"return": {}}
-----------------------------------------------------------------------
{"execute":"query-migrate-capabilities"}
{"return": [{"state": false, "capability": "xbzrle"}, {"state": false, "capability": "rdma-pin-all"}, {"state": false, "capability": "auto-converge"}, {"state": false, "capability": "zero-blocks"}, {"state": false, "capability": "compress"}, {"state": false, "capability": "events"}, {"state": false, "capability": "postcopy-ram"}, {"state": false, "capability": "x-colo"}, {"state": false, "capability": "release-ram"}, {"state": false, "capability": "block"}, {"state": false, "capability": "return-path"}, {"state": true, "capability": "pause-before-switchover"}, {"state": false, "capability": "x-multifd"}]}
-----------------------------------------------------------------------


(3) [DEST] Start the NBD server
-----------------------------------------------------------------------
{ "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host": "localhost", "port": "3333" } } } }
{"return": {}}
{ "execute": "nbd-server-add", "arguments": { "device": "virtio0","writable": true } }
{"return": {}}
-----------------------------------------------------------------------


(4) [SOURCE] Execute the `drive-mirror` command
-----------------------------------------------------------------------
{ "execute": "drive-mirror", "arguments": { "device": "virtio0", "target": "nbd:localhost:3333:exportname=virtio0", "sync": "full","format": "raw", "mode": "existing" } }
{"return": {}}
{"timestamp": {"seconds": 1510677793, "microseconds": 645286}, "event": "BLOCK_JOB_READY", "data": {"device": "virtio0", "len": 41126400, "offset": 41126400, "speed": 0, "type": "mirror"}}
-----------------------------------------------------------------------


(5) [SOURCE] Issue the `migrate` command (and query the migration status):
-----------------------------------------------------------------------
{"execute":"migrate","arguments":{"blk":false,"inc":false,"uri":"tcp:localhost:6666"}}
{"return": {}}
{"timestamp": {"seconds": 1510744913, "microseconds": 630373}, "event": "STOP"}

{"execute":"query-migrate"}                                                                                                                                                              
{"return": {"expected-downtime": 300, "status": "pre-switchover", "setup-time": 2, "total-time": 110111, "ram": {"total": 537403392, "postcopy-requests": 0, "dirty-sync-count": 2, "page-size": 4096, "remaining": 4141056, "mbps": 268.56432, "transferred": 59759367, "duplicate": 115920, "dirty-pages-rate": 72829, "skipped": 0, "normal-bytes": 58601472, "normal": 14307}}}
-----------------------------------------------------------------------


(6) [SOURCE] Issue the `block-job-cancel` command
-----------------------------------------------------------------------
{"execute":"block-job-cancel","arguments":{"device":"virtio0"} }
{"return": {}}
{"timestamp": {"seconds": 1510678024, "microseconds": 526240}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "virtio0", "len": 41126400, "offset": 41126400, "speed": 0, "type": "mirror"}}
-----------------------------------------------------------------------


(7) [DEST] Stop the NBD server
-----------------------------------------------------------------------
{"execute":"nbd-server-stop"}
{"return": {}}
-----------------------------------------------------------------------


(8) [DEST] Resume the guest vCPUs
-----------------------------------------------------------------------
{"execute":"cont"}
{"return": {}}
-----------------------------------------------------------------------


(9) [SOURCE] Gracefully terminate the QEMU process (repeat the same on
    destination.)
-----------------------------------------------------------------------
{"execute":"quit"}
{"return": {}}
{"timestamp": {"seconds": 1510678728, "microseconds": 574318}, "event": "SHUTDOWN", "data": {"guest": false}}
-----------------------------------------------------------------------

Comment 17 Dr. David Alan Gilbert 2017-11-21 09:31:33 UTC
(In reply to xianwang from comment #15)
> (In reply to Dr. David Alan Gilbert from comment #14)
> > xianwang:
> >   Please copy kashyap's example - json is very fussy;
> > 
> > { "execute": "nbd-server-add", "arguments": { "device":
> > "virtio0","writable": true } }
> > 
> > Note there are no " around the true
> 
> Dave,
> I have tried as:
> {"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":
> {"host":"127.0.0.1","port":"12345"}}}}
> {"return": {}}
> {"execute":"nbd-server-add","arguments":{"device":"node-TargetDisk",
> "writable":true}}
> {"return": {}}
> 
> it works, thanks.
> 
> But I have another question, for "pre-switchover", what is the right order
> of doing "live migration" and doing "block mirror" ? 
> I think we should first do block-mirror, after block job becomes to
> "BLOCK_JOB_COMPLETED" whether by "`block-job-complete` or
> `block-job-cancel`, then begin do live migration, but this failed, on
> contrary, first do live migration, during live migration do block-mirror, it
> succeed, I am confused why

Since the guest is still running during live migration you've got to mirror the changes made to the disk by the guest during the live migration itself; so you can't finish the mirror before the migration.   You have to wait for the drive mirror to declare 'block job ready' before starting the migrate command, that way you know the block code has done the first pass of copying the disk across.

Comment 18 xianwang 2017-11-21 16:04:28 UTC
Thanks for Dave's and Kashyap's reply, FYI, I have test this scenario as following:

Version:
Host:
3.10.0-768.el7.ppc64le
qemu-kvm-rhev-2.10.0-6.el7.ppc64le
SLOF-20170724-2.git89f519f.el7.noarch

Guest:
3.10.0-768.el7.ppc64le

a)src end:
    -drive id=drive_image1,if=none,aio=threads,cache=none,format=qcow2,file=/home/test.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,addr=07,bootindex=0 \

check block and enable pause-before-switchover:
(qemu) info block
drive_image1 (#block143): /home/test.qcow2 (qcow2)
    Attached to:      /machine/peripheral/image1/virtio-backend
    Cache mode:       writeback, direct
(qemu) migrate_set_capability pause-before-switchover on
(qemu) info migrate_capabilities 
xbzrle: off
rdma-pin-all: off
auto-converge: off
zero-blocks: off
compress: off
events: off
postcopy-ram: off
x-colo: off
release-ram: off
return-path: off
pause-before-switchover: on
or
-----------------------------------------------------------------------
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"pause-before-switchover","state":true}]}}
{"return": {}}
-----------------------------------------------------------------------
{"execute":"query-migrate-capabilities"}
{"return": [{"state": false, "capability": "xbzrle"}, {"state": false, "capability": "rdma-pin-all"}, {"state": false, "capability": "auto-converge"}, {"state": false, "capability": "zero-blocks"}, {"state": false, "capability": "compress"}, {"state": false, "capability": "events"}, {"state": false, "capability": "postcopy-ram"}, {"state": false, "capability": "x-colo"}, {"state": false, "capability": "release-ram"}, {"state": false, "capability": "block"}, {"state": false, "capability": "return-path"}, {"state": true, "capability": "pause-before-switchover"}, {"state": false, "capability": "x-multifd"}]}
-----------------------------------------------------------------------

b)dst end:
    -blockdev node-name=node-TargetDisk,driver=raw,file.driver=file,file.node-name=file,file.filename=/home/nbd.raw \
    -device virtio-blk-pci,id=virtio0,drive=node-TargetDisk,bus=pci.0,addr=07,bootindex=0 \

check:
(qemu) info block
node-TargetDisk: /home/nbd.raw (raw)
    Attached to:      /machine/peripheral/virtio0/virtio-backend
    Cache mode:       writeback

setup nbd server on dst host(make dst qemu as nbd server)
(qemu) nbd_server_start 10.16.69.77:12345
(qemu) nbd_server_add -w node-TargetDisk
or
-----------------------------------------------------------------------
{ "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host": "10.16.69.77", "port": "12345" } } } }
{"return": {}}
{ "execute": "nbd-server-add", "arguments": { "device": "node-TargetDisk","writable": true } }
{"return": {}}
-----------------------------------------------------------------------

c)on src end, begin to do block-mirror and after block job becomes to "BLOCK_JOB_READY" then do live migration, when live migration becomes to "pre-switchover" and cancel block job and then "migrate_continue pre-switchover"

src end:
block-mirror:
-----------------------------------------------------------------------
{ "execute": "drive-mirror", "arguments": { "device": "drive_image1", "target": "nbd:10.16.69.77:12345:exportname=node-TargetDisk", "sync": "full","format": "raw", "mode": "existing" } }
{"return": {}}
{"timestamp": {"seconds": 1510677793, "microseconds": 645286}, "event": "BLOCK_JOB_READY", "data": {"device": "virtio0", "len": 41126400, "offset": 41126400, "speed": 0, "type": "mirror"}}
-----------------------------------------------------------------------

live migration:
(qemu) migrate -d tcp:10.16.69.77:5801
(qemu) info migrate
Migration status: pre-switchover
or
-----------------------------------------------------------------------
{"execute":"migrate","arguments":{"blk":false,"inc":false,"uri":"tcp:10.16.69.77:5801"}}
{"return": {}}
{"timestamp": {"seconds": 1510744913, "microseconds": 630373}, "event": "STOP"}

{"execute":"query-migrate"}                                                                                                                                                              
{"return": {"expected-downtime": 300, "status": "pre-switchover", "setup-time": 2, "total-time": 110111, "ram": {"total": 537403392, "postcopy-requests": 0, "dirty-sync-count": 2, "page-size": 4096, "remaining": 4141056, "mbps": 268.56432, "transferred": 59759367, "duplicate": 115920, "dirty-pages-rate": 72829, "skipped": 0, "normal-bytes": 58601472, "normal": 14307}}}
-----------------------------------------------------------------------

cancel block-job:
-----------------------------------------------------------------------
{"execute": "block-job-cancel","arguments":{"device":"drive_image1"}}
{"return": {}}
{"timestamp": {"seconds": 1510678024, "microseconds": 526240}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "virtio0", "len": 41126400, "offset": 41126400, "speed": 0, "type": "mirror"}}
-----------------------------------------------------------------------

d)on dst host, stop nbd server:
(qemu) nbd_server_stop
or
-----------------------------------------------------------------------
{"execute":"nbd-server-stop"}
{"return": {}}
-----------------------------------------------------------------------

e)on src host, continue pre-switchover
(qemu) migrate_continue pre-switchover
(qemu) info migrate
Migration status: completed
or 
-----------------------------------------------------------------------
{"execute":"query-migrate"}
{"return": {"status": "completed", "setup-time": 27, "downtime": 383168, "total-time": 401383, "ram": {"total": 8607039488, "postcopy-requests": 0, "dirty-sync-count": 5, "page-size": 4096, "remaining": 0, "mbps": 12.560209, "transferred": 621304158, "duplicate": 1964677, "dirty-pages-rate": 0, "skipped": 0, "normal-bytes": 602443776, "normal": 147081}}}
-----------------------------------------------------------------------

f)on src host, check the vm status
(qemu) info status 
VM status: running
or
-----------------------------------------------------------------------
{"execute":"query-status"}
{"return": {"status": "running", "singlestep": false, "running": true}}
-----------------------------------------------------------------------

So, this bug is fixed.

Comment 19 xianwang 2017-11-22 22:42:32 UTC
(In reply to xianwang from comment #6)
> Hi, Dave,
> I have found some information about your patch, that in this link:
> http://patchwork.ozlabs.org/cover/827760/
> there is two work flow as said in this link,
> The precopy flow is:
> active->pre-switchover->device->completed
> 
> The postcopy flow is:
> active->pre-switchover->postcopy-active->completed
> 
> But I didn't know the detail test steps for this bug, could you give me the
> detail test steps and expected results for these two scenarios? thanks

Although I have added a case for scenario "precopy flow" to migration test plan, I think I should also add the second scenario "postcopy flow", I have tried two different test steps but I didn't catch the "postcopy-active" status of migration, the test flows are as following:

a)                
block mirror-->"BLOCK_JOB_READY"-->live migration-->"Migration status: pre-switchover"-->block-job-cancel-->nbd_server_stop-->(qemu)migrate_continue pre-switchover-->(qemu) migrate_start_postcopy-->"Migration status: completed"

b)block mirror-->"BLOCK_JOB_READY"-->live migration-->(qemu) migrate_start_postcopy-->"Migration status: pre-switchover"-->block-job-cancel-->nbd_server_stop-->(qemu)migrate_continue pre-switchover-->"Migration status: completed"

questions:
1)In these two test scenarios about postcopy, I can't catch "postcopy-active" status of migration, is there something wrong for these steps? or what is the right way for postcopy flow? 

2)it seems that the state of "Migration status: pre-switchover" is a state that guest and migration process are both paused; only execute "migrate_continue pre-switchover", migration could continue and will complete directly, yes? so, any operation after "migrate_continue pre-switchover" would not work including "(qemu) migrate_start_postcopy", yes? btw, what is the condition of switching to "pre-switchover" for migration? Is that condition same with the condition of going into "downtime"?

Dave and Kashyap, could you help to reply the above questions? thanks

Comment 20 Kashyap Chamarthy 2017-11-23 12:49:14 UTC
(In reply to Kashyap Chamarthy from comment #16)

[...]

> The right order using 'pause-before-switchover' is documented in the text
> file I attached in comment#10.
> 
> Copying here's the full sequence:
> 
> [I didn't copy / paste the first step intentionally; it just launches source
> & destination QEMU -- refer the file in comment#10].

Dave pointed out on IRC that I was missing the "-S" (Do not start CPU at start-up) in my notes in comment#10 for the destination QEMU command-line. 

So don't forget to note that when reusing those notes that you ought to start your destination QEMU with "-S"

[...]

Comment 21 Kashyap Chamarthy 2017-11-23 15:20:56 UTC
(In reply to Kashyap Chamarthy from comment #20)
> (In reply to Kashyap Chamarthy from comment #16)
> 
> [...]
> 
> > The right order using 'pause-before-switchover' is documented in the text
> > file I attached in comment#10.
> > 
> > Copying here's the full sequence:
> > 
> > [I didn't copy / paste the first step intentionally; it just launches source
> > & destination QEMU -- refer the file in comment#10].
> 
> Dave pointed out on IRC that I was missing the "-S" (Do not start CPU at
> start-up) in my notes in comment#10 for the destination QEMU command-line. 
> 
> So don't forget to note that when reusing those notes that you ought to
> start your destination QEMU with "-S"

 So Dave explains to me on IRC that when using "-incoming" on destination, using "-S" is not *mandatory*.  (But if you _do_ start your destination QEMU with "-S", then you need to issue the QMP command "cont" to start the vCPUs again.)

To that end, I my updated instructions are here:

https://kashyapc.fedorapeople.org/virt/qemu/Live-Migration-drive-mirror+NBD-pause-before-switchover.txt

Comment 22 Dr. David Alan Gilbert 2017-11-23 15:29:56 UTC
(In reply to xianwang from comment #19)
> (In reply to xianwang from comment #6)
> > Hi, Dave,
> > I have found some information about your patch, that in this link:
> > http://patchwork.ozlabs.org/cover/827760/
> > there is two work flow as said in this link,
> > The precopy flow is:
> > active->pre-switchover->device->completed
> > 
> > The postcopy flow is:
> > active->pre-switchover->postcopy-active->completed
> > 
> > But I didn't know the detail test steps for this bug, could you give me the
> > detail test steps and expected results for these two scenarios? thanks
> 
> Although I have added a case for scenario "precopy flow" to migration test
> plan, I think I should also add the second scenario "postcopy flow", I have
> tried two different test steps but I didn't catch the "postcopy-active"
> status of migration, the test flows are as following:

Postcopy with block migration is something that's not yet ready (it's waiting on a series by mreitz that does it a different way from block_mirror); so the test isn't really valid for postcopy yet.

> a)                
> block mirror-->"BLOCK_JOB_READY"-->live migration-->"Migration status:
> pre-switchover"-->block-job-cancel-->nbd_server_stop--
> >(qemu)migrate_continue pre-switchover-->(qemu)
> migrate_start_postcopy-->"Migration status: completed"
> 
> b)block mirror-->"BLOCK_JOB_READY"-->live migration-->(qemu)
> migrate_start_postcopy-->"Migration status:
> pre-switchover"-->block-job-cancel-->nbd_server_stop--
> >(qemu)migrate_continue pre-switchover-->"Migration status: completed"
> 
> questions:
> 1)In these two test scenarios about postcopy, I can't catch
> "postcopy-active" status of migration, is there something wrong for these
> steps? or what is the right way for postcopy flow? 

To catch a postcopy case you need to have a guest that's very busy accessing memory and/or a slow network bandwidth.

> 2)it seems that the state of "Migration status: pre-switchover" is a state
> that guest and migration process are both paused; only execute
> "migrate_continue pre-switchover", migration could continue and will
> complete directly, yes? so, any operation after "migrate_continue
> pre-switchover" would not work including "(qemu) migrate_start_postcopy",
> yes?

If performing postcopy, you'll have done a migrate_start_postcopy and then hit the pre-switchover state after it.
(Or migration finishes without needing postcopy, in which case you hit pre-switchover before giving migrate_start_postcopy).

> btw, what is the condition of switching to "pre-switchover" for
> migration? Is that condition same with the condition of going into
> "downtime"?

Yes, it's about the same time; but it gives the management layer a chance to do some cleanup just after going into downtime.

> 
> Dave and Kashyap, could you help to reply the above questions? thanks

Comment 23 xianwang 2017-11-24 02:44:45 UTC
(In reply to Dr. David Alan Gilbert from comment #22)
> (In reply to xianwang from comment #19)
> > (In reply to xianwang from comment #6)
> > > Hi, Dave,
> > > I have found some information about your patch, that in this link:
> > > http://patchwork.ozlabs.org/cover/827760/
> > > there is two work flow as said in this link,
> > > The precopy flow is:
> > > active->pre-switchover->device->completed
> > > 
> > > The postcopy flow is:
> > > active->pre-switchover->postcopy-active->completed
> > > 
> > > But I didn't know the detail test steps for this bug, could you give me the
> > > detail test steps and expected results for these two scenarios? thanks
> > 
> > Although I have added a case for scenario "precopy flow" to migration test
> > plan, I think I should also add the second scenario "postcopy flow", I have
> > tried two different test steps but I didn't catch the "postcopy-active"
> > status of migration, the test flows are as following:
> 
> Postcopy with block migration is something that's not yet ready (it's
> waiting on a series by mreitz that does it a different way from
> block_mirror); so the test isn't really valid for postcopy yet.
> 
FYI, I didn't need to add the case of "postcopy flow"? 

> > a)                
> > block mirror-->"BLOCK_JOB_READY"-->live migration-->"Migration status:
> > pre-switchover"-->block-job-cancel-->nbd_server_stop--
> > >(qemu)migrate_continue pre-switchover-->(qemu)
> > migrate_start_postcopy-->"Migration status: completed"
> > 
> > b)block mirror-->"BLOCK_JOB_READY"-->live migration-->(qemu)
> > migrate_start_postcopy-->"Migration status:
> > pre-switchover"-->block-job-cancel-->nbd_server_stop--
> > >(qemu)migrate_continue pre-switchover-->"Migration status: completed"
> > 
> > questions:
> > 1)In these two test scenarios about postcopy, I can't catch
> > "postcopy-active" status of migration, is there something wrong for these
> > steps? or what is the right way for postcopy flow? 
> 
> To catch a postcopy case you need to have a guest that's very busy accessing
> memory and/or a slow network bandwidth.
> 
> > 2)it seems that the state of "Migration status: pre-switchover" is a state
> > that guest and migration process are both paused; only execute
> > "migrate_continue pre-switchover", migration could continue and will
> > complete directly, yes? so, any operation after "migrate_continue
> > pre-switchover" would not work including "(qemu) migrate_start_postcopy",
> > yes?
> 
> If performing postcopy, you'll have done a migrate_start_postcopy and then
> hit the pre-switchover state after it.
> (Or migration finishes without needing postcopy, in which case you hit
> pre-switchover before giving migrate_start_postcopy).
> 
As you said, I have "stress --io 4" in guest, the scenario "a)" of my test is same with that you said the first one, the scenario "b)" of my test is as you said the other scenario. I think the first scenario is more reasonable;
so how do you think it's better to add the scenario "a)" to test plan or don't need to add "postcopy flow" this case?

 
> > btw, what is the condition of switching to "pre-switchover" for
> > migration? Is that condition same with the condition of going into
> > "downtime"?
> 
> Yes, it's about the same time; but it gives the management layer a chance to
> do some cleanup just after going into downtime.
> 
> > 
> > Dave and Kashyap, could you help to reply the above questions? thanks

Comment 24 xianwang 2017-11-24 02:47:31 UTC
(In reply to Dr. David Alan Gilbert from comment #22)
> (In reply to xianwang from comment #19)
> > (In reply to xianwang from comment #6)
> > > Hi, Dave,
> > > I have found some information about your patch, that in this link:
> > > http://patchwork.ozlabs.org/cover/827760/
> > > there is two work flow as said in this link,
> > > The precopy flow is:
> > > active->pre-switchover->device->completed
> > > 
> > > The postcopy flow is:
> > > active->pre-switchover->postcopy-active->completed
> > > 
> > > But I didn't know the detail test steps for this bug, could you give me the
> > > detail test steps and expected results for these two scenarios? thanks
> > 
> > Although I have added a case for scenario "precopy flow" to migration test
> > plan, I think I should also add the second scenario "postcopy flow", I have
> > tried two different test steps but I didn't catch the "postcopy-active"
> > status of migration, the test flows are as following:
> 
> Postcopy with block migration is something that's not yet ready (it's
> waiting on a series by mreitz that does it a different way from
> block_mirror); so the test isn't really valid for postcopy yet.
> 
FYI, I didn't need to add the case of "postcopy flow"? 

> > a)                
> > block mirror-->"BLOCK_JOB_READY"-->live migration-->"Migration status:
> > pre-switchover"-->block-job-cancel-->nbd_server_stop--
> > >(qemu)migrate_continue pre-switchover-->(qemu)
> > migrate_start_postcopy-->"Migration status: completed"
> > 
> > b)block mirror-->"BLOCK_JOB_READY"-->live migration-->(qemu)
> > migrate_start_postcopy-->"Migration status:
> > pre-switchover"-->block-job-cancel-->nbd_server_stop--
> > >(qemu)migrate_continue pre-switchover-->"Migration status: completed"
> > 
> > questions:
> > 1)In these two test scenarios about postcopy, I can't catch
> > "postcopy-active" status of migration, is there something wrong for these
> > steps? or what is the right way for postcopy flow? 
> 
> To catch a postcopy case you need to have a guest that's very busy accessing
> memory and/or a slow network bandwidth.
> 
> > 2)it seems that the state of "Migration status: pre-switchover" is a state
> > that guest and migration process are both paused; only execute
> > "migrate_continue pre-switchover", migration could continue and will
> > complete directly, yes? so, any operation after "migrate_continue
> > pre-switchover" would not work including "(qemu) migrate_start_postcopy",
> > yes?
> 
> If performing postcopy, you'll have done a migrate_start_postcopy and then
> hit the pre-switchover state after it.
> (Or migration finishes without needing postcopy, in which case you hit
> pre-switchover before giving migrate_start_postcopy).
> 
As you said, I have "stress --io 4" in guest, the scenario "a)" of my test is same with that you said the first one, the scenario "b)" of my test is as you said the other scenario. I think the first scenario is more reasonable;
so how do you think it's better to add the scenario "a)" to test plan or don't need to add "postcopy flow" this case?

 
> > btw, what is the condition of switching to "pre-switchover" for
> > migration? Is that condition same with the condition of going into
> > "downtime"?
> 
> Yes, it's about the same time; but it gives the management layer a chance to
> do some cleanup just after going into downtime.
> 
> > 
> > Dave and Kashyap, could you help to reply the above questions? thanks

For the postcopy scenario is need to confirm, I modify this bug to on_qa, if this issue resolved, I will verify this bug, thanks for understanding

Comment 25 Dr. David Alan Gilbert 2017-11-24 10:03:15 UTC
Yes, you don't need to do postcopy; we'll sort out postcopy+storage behaviour when Max's patches are done.

Comment 26 xianwang 2017-11-27 02:11:45 UTC
(In reply to Dr. David Alan Gilbert from comment #25)
> Yes, you don't need to do postcopy; we'll sort out postcopy+storage
> behaviour when Max's patches are done.

OK, got it, thanks for Dave's confirmation

Comment 27 xianwang 2017-11-28 04:52:22 UTC
According to comment25, this bug is fixed.

Comment 29 errata-xmlrpc 2018-04-11 00:36:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104


Note You need to log in before you can comment on or make changes to this bug.