Bug 1314682 - The value of "offset" reduced when mirroring which the source have no enough space
The value of "offset" reduced when mirroring which the source have no enough ...
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
All Linux
low Severity low
: rc
: ---
Assigned To: John Snow
Qianqian Zhu
Depends On: 1314635
  Show dependency treegraph
Reported: 2016-03-04 03:51 EST by jingzhao
Modified: 2017-11-30 14:54 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1314635
Last Closed: 2017-11-30 14:54:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Comment 2 John Snow 2016-09-06 20:51:54 EDT
I think this can be moved to 7.4.
Comment 4 John Snow 2017-11-29 20:01:19 EST
I just noticed that this was filed against 0.12 for RHEL6, but we've carried it along with us to extremely modern versions of QEMU. The last time I looked at this bug I couldn't find a way in the code for the offset parameter to actually decrease *anywhere* in code, so I am quite confident that this bug was fixed a very, very long time ago.

I'm looking at the way the code is written back in the 0.12 version and the offset was calculated as such:

s->common.offset = (end - cnt) * BDRV_SECTOR_SIZE;

However, "cnt" is not a fixed number of sectors left to copy -- as the drive continues to accrue new writes, this value will actually increase, so you may see offset decrease as we "lose" progress for the live operation.

This makes sense, though it's a terrible API. It is my intention for future versions of the API to separate the notion of a "percentage" progress that always increases from job statistics, which treats "bytes left to copy" and "bytes copied" differently from abstract "progress."

Modern versions of the code do not allow for decrements to the offset, so this behavior shouldn't be able to be seen nowadays.

As for the second problem reported here, mirror failing to return -EIO but returning success and leaving the base image unchanged: I'm confident this is not present in modern versions or we'd have seen more reports. 

So, based on the age of this report and its absence of an obvious equivalent in qemu-kvm-rhev, I suspect we will be able to close this after a re-test.

Thank you,
John Snow
Comment 5 Qianqian Zhu 2017-11-30 05:46:37 EST
Tested on qemu-kvm-rhev-2.10.0-9.el7.x86_64& kernel-3.10.0-800.el7.x86_64, tried below steps 7 times, can't reproduce the issue:

1. Prepare 10G block device:
# lsblk
loop0                        7:0    0    10G  0 loop /mnt/testpath
2. Launch guest with image file located in above path:
/usr/libexec/qemu-kvm \
-name rhel7_4 \
-m 1G \
-smp 2 \
-vnc :3 \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=70:e2:84:14:0e:15 \
-uuid 49a3438a-70a3-4ba8-92ce-3a05e0934608 -device virtio-scsi-pci,id=scsi \
-drive file=/mnt/testpath/rhel74-64-virtio-scsi.qcow2,if=none,format=qcow2,id=drive-virtio-scsi,werror=stop,rerror=stop \
-device scsi-hd,drive=drive-virtio-scsi,id=virtio-scsi,bus=scsi.0,scsi-id=0 \
-qmp unix:/tmp/qmp-sock,server,nowait \
-monitor stdio

3. dd files inside guest:
# dd if=/dev/urandom of=test bs=1G

4. Start block mirror:
{ "execute" : "drive-mirror", "arguments" : { "device" : "drive-virtio-scsi","target" : "/home/mirror", "format": "qcow2","sync":"full"}}

5. Query block job, check offset

6. Complete block job after reaching ready state:
{"execute":"block-job-complete","arguments" : { "device" : "drive-virtio-scsi"}}

7. Query-block, check block device file

The offset keeps increasing before and after the BLOCK_IO_ERROR:
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10202513408, "offset": 968556544, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10202513408, "offset": 974192640, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}

{"execute" : "query-block-jobs", "arguments" : {} }
{"timestamp": {"seconds": 1512037572, "microseconds": 112446}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-virtio-scsi", "nospace": true, "__com.redhat_reason": "enospc", "node-name": "#block947", "reason": "No space left on device", "operation": "write", "action": "stop"}}

{"timestamp": {"seconds": 1512037572, "microseconds": 112587}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-virtio-scsi", "nospace": true, "__com.redhat_reason": "enospc", "node-name": "#block947", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1512037572, "microseconds": 112646}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-virtio-scsi", "nospace": true, "__com.redhat_reason": "enospc", "node-name": "#block947", "reason": "No space left on device", "operation": "write", "action": "stop"}}
{"execute" : "query-block-jobs", "arguments" : {} }{"timestamp": {"seconds": 1512037573, "microseconds": 386970}, "event": "STOP"}

{"execute" : "query-block-jobs", "arguments" : {} }
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1097400320, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1097400320, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1097400320, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1097400320, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1221459968, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1256456192, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1267269632, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1283915776, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-scsi", "busy": true, "len": 10387062784, "offset": 1371471872, "paused": false, "speed": 0, "ready": false, "type": "mirror"}]}
{"execute" : "query-block-jobs", "arguments" : {} }

Block file correct after complete the block job.
{"execute" : "query-block", "arguments" : {} }
{"return": [{"io-status": "nospace", "device": "drive-virtio-scsi", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/mirror", "cluster-size": 65536, "format": "qcow2", "actual-size": 10380251136, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "#block817", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": false, "writeback": true}, "file": "/home/mirror", "encryption_key_missing": false}, "qdev": "virtio-scsi", "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "qdev": "/machine/unattached/device[20]", "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "type": "unknown"}]}

Based on above result, I agree that this bz can be closed.
Comment 6 John Snow 2017-11-30 14:54:24 EST
My apologies that I didn't notice this was filed against 0.12 earlier.

Note You need to log in before you can comment on or make changes to this bug.