Bug 1518738

This is going to be implemented as a filter driver rather than a blockdev-add option. But thanks for creating the BZ, this does need to be tracked.

Fix included in qemu-kvm-rhev-2.12.0-5.el7

Fix included in qemu-kvm-rhev-2.12.0-6.el7

Verified this bug on: qemu-kvm-rhev-2.12.0-6.el7.x86_64

Scenario 1:
1. copying data (a guest) from a remote address to a local image.
$ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh", "file.host":"10.73.**.**","file.path":"/home/coli/rhel76-64-virtio-scsi.qcow2","file.host_key_check":"no" }'

2. $ qemu-img  map local.qcow2 (only have backing file info, no about local.qcow2)
0x4ffde0000     0x20000         0xd30000        json:{"host": "10.73.**.**", "host_key_check": "no", "driver": "ssh", "path": "/home/coli/rhel76-64-virtio-scsi.qcow2"}
0x4ffff0000     0x10000         0x70000         json:{"host": "10.73.**.**", "host_key_check": "no", "driver": "ssh", "path": "/home/coli/rhel76-64-virtio-scsi.qcow2"}

3. boot up the local.qcow2 which has a OS.
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/local.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \

4. $ qemu-img map local.qcow2 (have local.qcow2 info via copy-on-read)
0x100000        0x10000         0x26a0000       local.qcow2
0x9b50000       0x10000         0x26b0000       local.qcow2
0x202d0000      0x90000         0x280000        local.qcow2
0x20360000      0x60000         0x3c0000        local.qcow2


Scenario 2:
1. copying data from a remote address to a local image.
$ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh", "file.host":"10.66.10.121","file.path":"/root/remote.img","file.host_key_check":"no" }'

2. $ qemu-img map local.qcow2 --output=json
[{ "start": 0, "length": 10737418240, "depth": 1, "zero": true, "data": false}]

3. read image for the 1st time
$  qemu-io --image-opts driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.filename=local.qcow2  -c 'read 0 1M'

4. $ qemu-img map local.qcow2 --output=json
[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
{ "start": 1048576, "length": 10736369664, "depth": 1, "zero": true, "data": false}]

5. read image for the 2nd time
$ qemu-io --image-opts driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.filename=local.qcow2  -c 'read 0 1M'
read 1048576/1048576 bytes at offset 0
1 MiB, 1 ops; 0.0001 sec (5.034 GiB/sec and 5154.6392 ops/sec)


Scenario 3:
qemu-iotests 216:
./check 216 -qcow2
QEMU          -- "/usr/libexec/qemu-kvm" -nodefaults -machine accel=qtest
QEMU_IMG      -- "/usr/bin/qemu-img" 
QEMU_IO       -- "/usr/bin/qemu-io"  --cache writeback -f qcow2
QEMU_NBD      -- "/usr/bin/qemu-nbd" 
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/x86_64 hp-dl385pg8-04 3.10.0-862.6.1.el7.x86_64
TEST_DIR      -- /root/rpmbuild/BUILD/qemu-2.12.0/tests/qemu-iotests/scratch
SOCKET_SCM_HELPER -- 

216        
Passed all 1 tests


Thanks.

(In reply to CongLi from comment #12)
> Verified this bug on: qemu-kvm-rhev-2.12.0-6.el7.x86_64
>
> Scenario 2:
> 1. copying data from a remote address to a local image.
> $ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh",
> "file.host":"10.66.10.121","file.path":"/root/remote.img","file.
> host_key_check":"no" }'
> 
> 2. $ qemu-img map local.qcow2 --output=json
> [{ "start": 0, "length": 10737418240, "depth": 1, "zero": true, "data":
> false}]
> 
> 3. read image for the 1st time
> $  qemu-io --image-opts
> driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.
> filename=local.qcow2  -c 'read 0 1M'

read 1048576/1048576 bytes at offset 0
1 MiB, 1 ops; 0.0029 sec (339.789 MiB/sec and 339.7893 ops/sec)

> 
> 4. $ qemu-img map local.qcow2 --output=json
> [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
> { "start": 1048576, "length": 10736369664, "depth": 1, "zero": true, "data":
> false}]
> 
> 5. read image for the 2nd time
> $ qemu-io --image-opts
> driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.
> filename=local.qcow2  -c 'read 0 1M'
> read 1048576/1048576 bytes at offset 0
> 1 MiB, 1 ops; 0.0001 sec (5.034 GiB/sec and 5154.6392 ops/sec)

The 2nd time(0.0001 sec) read is faster than the 1st time(0.0029sec) which is as expected.

(In reply to CongLi from comment #12)
> Verified this bug on: qemu-kvm-rhev-2.12.0-6.el7.x86_64
> 
> Scenario 1:
> 1. copying data (a guest) from a remote address to a local image.
> $ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh",
> "file.host":"10.73.**.**","file.path":"/home/coli/rhel76-64-virtio-scsi.
> qcow2","file.host_key_check":"no" }'
> 
> 2. $ qemu-img  map local.qcow2 (only have backing file info, no about
> local.qcow2)
> 0x4ffde0000     0x20000         0xd30000        json:{"host": "10.73.**.**",
> "host_key_check": "no", "driver": "ssh", "path":
> "/home/coli/rhel76-64-virtio-scsi.qcow2"}
> 0x4ffff0000     0x10000         0x70000         json:{"host": "10.73.**.**",
> "host_key_check": "no", "driver": "ssh", "path":
> "/home/coli/rhel76-64-virtio-scsi.qcow2"}
> 
> 3. boot up the local.qcow2 which has a OS.
>     -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/home/local.qcow2 \
>     -device scsi-hd,id=image1,drive=drive_image1 \
> 
> 4. $ qemu-img map local.qcow2 (have local.qcow2 info via copy-on-read)
> 0x100000        0x10000         0x26a0000       local.qcow2
> 0x9b50000       0x10000         0x26b0000       local.qcow2
> 0x202d0000      0x90000         0x280000        local.qcow2
> 0x20360000      0x60000         0x3c0000        local.qcow2


Sorry, please ignore the invalid test scenario.

Hi Max,

QE would like to confirm if 'copy-on-read' is ready in -blockdev and blockdev-add now.

QE met some errors when test 'copy-on-read', could you please help confirm if the usage is wrong ?

1. -blockdev:
qemu-kvm: -blockdev driver=raw,cache.direct=off,cache.no-flush=on,file.filename=/root/test.raw,node-name=my_file,file.driver=file,copy-on-read=on: Parameter 'copy-on-read' is unexpected

2. blockdev-add:
{"execute": "blockdev-add","arguments": {"node-name": "drive2","driver": "raw", "file":    {"driver":"file", "filename": "/root/test.raw","copy-on-read":"on"}}}
{"error": {"class": "GenericError", "desc": "Parameter 'file.copy-on-read' is unexpected"}}


Thanks.

Hi,

The copy-on-read filter driver basically supersedes the copy-on-read option we had for -drive (which is not available for -blockdev or blockdev-add, as you can see).  It is instead a block driver, so you would use it like so:

1. -blockdev:
-blockdev node-name=my_file,driver=copy-on-read,file.driver=raw,file.file.driver=file,file.file.filename=/root/test.raw

2. blockdev-add:
{ "execute": "blockdev-add",
  "arguments": {
      "node-name": "drive2",
      "driver": "copy-on-read",
      "file": {
          "driver": "raw",
          "file": {
              "driver": "file",
              "filename": "/root/test.raw"
          }
      }
  } }

In practice, those test cases are not very useful, however, since copy-on-read is only useful with a backing file (which raw does not provide).

So comparable useful test cases would look like this:

Image preparation:
$ qemu-img create -f qcow2 backing.qcow2 64M
Formatting 'backing.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-img create -f qcow2 -b backing.qcow2 overlay.qcow2 64M
Formatting 'overlay.qcow2', fmt=qcow2 size=67108864 backing_file=backing.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-io -c 'write 0 64M' backing.qcow2

Check the overlay's allocation state:
$ qemu-img map overlay.qcow2 
Offset          Length          Mapped to       File
0               0x4000000       0x50000         backing.qcow2

(Note the filename, so the data is still in the backing file.)

1. -blockdev:
-blockdev "{'node-name':'node0','driver':'copy-on-read','file':{'driver':'qcow2','file':{'driver':'file','filename':'overlay.qcow2'}}}"

2. blockdev-add:
{'execute':'blockdev-add','arguments':{'node-name':'node0','driver':'copy-on-read','file':{'driver':'qcow2','file':{'driver':'file','filename':'overlay.qcow2'}}}}

In both cases, execute the following over QMP:
{'execute':'human-monitor-command','arguments':{'command-line':'qemu-io node0 "read 0 64M"'}}

Close the VM and check the overlay's allocation status again:
$ qemu-img map overlay.qcow2                  
Offset          Length          Mapped to       File
0               0x4000000       0x50000         overlay.qcow2

(The filename has changed, so the data is now allocated in the overlay.)

Max

(In reply to Max Reitz from comment #16)
> Hi,
> 
> The copy-on-read filter driver basically supersedes the copy-on-read option
> we had for -drive (which is not available for -blockdev or blockdev-add, as
> you can see).  It is instead a block driver, so you would use it like so:
> 
> 1. -blockdev:
> -blockdev
> node-name=my_file,driver=copy-on-read,file.driver=raw,file.file.driver=file,
> file.file.filename=/root/test.raw
> 
> 2. blockdev-add:
> { "execute": "blockdev-add",
>   "arguments": {
>       "node-name": "drive2",
>       "driver": "copy-on-read",
>       "file": {
>           "driver": "raw",
>           "file": {
>               "driver": "file",
>               "filename": "/root/test.raw"
>           }
>       }
>   } }

Thanks Max, it works.

> 
> In practice, those test cases are not very useful, however, since
> copy-on-read is only useful with a backing file (which raw does not provide).
> So comparable useful test cases would look like this:
> 
> Image preparation:
> $ qemu-img create -f qcow2 backing.qcow2 64M
> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 cluster_size=65536
> lazy_refcounts=off refcount_bits=16
> $ qemu-img create -f qcow2 -b backing.qcow2 overlay.qcow2 64M
> Formatting 'overlay.qcow2', fmt=qcow2 size=67108864
> backing_file=backing.qcow2 cluster_size=65536 lazy_refcounts=off
> refcount_bits=16
> $ qemu-io -c 'write 0 64M' backing.qcow2
> 
> Check the overlay's allocation state:
> $ qemu-img map overlay.qcow2 
> Offset          Length          Mapped to       File
> 0               0x4000000       0x50000         backing.qcow2
> 
> (Note the filename, so the data is still in the backing file.)
> 
> 1. -blockdev:
> -blockdev
> "{'node-name':'node0','driver':'copy-on-read','file':{'driver':'qcow2',
> 'file':{'driver':'file','filename':'overlay.qcow2'}}}"
> 
> 2. blockdev-add:
> {'execute':'blockdev-add','arguments':{'node-name':'node0','driver':'copy-on-
> read','file':{'driver':'qcow2','file':{'driver':'file','filename':'overlay.
> qcow2'}}}}
> 
> In both cases, execute the following over QMP:
> {'execute':'human-monitor-command','arguments':{'command-line':'qemu-io
> node0 "read 0 64M"'}}
> 
> Close the VM and check the overlay's allocation status again:
> $ qemu-img map overlay.qcow2                  
> Offset          Length          Mapped to       File
> 0               0x4000000       0x50000         overlay.qcow2
> 
> (The filename has changed, so the data is now allocated in the overlay.)
> 
> Max


This scenario is useful and reasonable for the feature, it's helpful in the testing, I have added this scenario as a test case.


Tested the scenario above on qemu-kvm-rhev-2.12.0-7.el7.x86_64:
1. Image preparation:
$ qemu-img create -f qcow2 backing.qcow2 64
Formatting 'backing.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-img create -f qcow2 -b backing.qcow2 overlay.qcow2 64
Formatting 'overlay.qcow2', fmt=qcow2 size=67108864 backing_file=backing.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Write sth into the backing file (0~64M)
$ qemu-io -c 'write 0 64M' backing.qcow2
wrote 67108864/67108864 bytes at offset 0
64 MiB, 1 ops; 0.1065 sec (600.392 MiB/sec and 9.3811 ops/sec)

3. Check the overlay's allocation state:
$ qemu-img map overlay.qcow2
Offset          Length          Mapped to       File
0               0x4000000       0x50000         backing.qcow2   --> here is 'File' name is backing.qcow2

4. Boot up guest with the overlay file with copy-on-read.
-blockdev driver=copy-on-read,file.file.driver=file,file.file.filename=/home/overlay.qcow2,node-name=node0,file.driver=qcow2 \
    -device scsi-hd,id=image2,drive=node0 \

5. Read the info written in 0~64M from overlay file via QMP:
{'execute':'human-monitor-command','arguments':{'command-line':'qemu-io node0 "read 0 64M"'}}
{"return": ""}

6. Shutdown the vm.
 
7. Check the overlay's allocation state again:
$ qemu-img map overlay.qcow2
Offset          Length          Mapped to       File
0               0x4000000       0x50000         overlay.qcow2   --> The filename has changed, so the data is now allocated in the overlay.
...

It also works well with blockdev-add.
{"execute":"blockdev-add","arguments":{"node-name":"node0","driver":"copy-on-read","file":{"driver":"qcow2","file":{"driver":"file","filename":"/home/overlay.qcow2"}}}}


Max, thank you again.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443