Description of problem: QEMU as NBD server always aligns file sizes up to the next sector boundary, but then uses byte-aligned bdrv_block_status() to determine the result for sparse reads or block status commands from the client. As a result, when serving a raw file that is not aligned to sector boundaries, qemu will see a transition from data to hole in the middle of the final sector exposed to the client, and fragments the reply to the client. QEMU as NBD client rejects the block status results as non-compliant (how can status change in anything smaller than the minimum alignment that the client is allowed to address?), killing the connection. Similarly, QEMU as NBD client tries to write beyond the export size of an nbd server that has a non-sector-aligned size (qemu always rounds up, and thus cannot be such a server, but nbdkit can be used for the purpose). Access to the padding bytes is always going to be fishy, but at the bare minimum, qemu should recognize that a file size that is not sector-aligned but with no corresponding minimum byte alignment implies that the server can handle sub-sector requests. Version-Release number of selected component (if applicable): qemu-kvm-rhev-2.12.0-9.el7 How reproducible: 100% Steps to Reproduce: 1. printf %01000d 0 > file 2. qemu-nbd -f raw -v -t -x '' file & 3. qemu-img map --output=json nbd://localhost:10809 4. kill $! 5. nbdkit file file=file & 6. qemu-io -f raw -c 'r 0 513' nbd://localhost:10809 7. kill $! Actual results: step 3: qemu-img: Protocol error: server sent status chunk with invalid length qemu-img: Could not read file metadata: Invalid argument step 6: nbdkit: file.0: error: invalid request: offset and length are out of range read failed: Invalid argument Expected results: step 3: [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true, "offset": 0}] (or maybe show the tail of the file as a hole, but at least not fail) step 6: read 513/513 bytes at offset 0 513 bytes, 1 ops; 0.0407 sec (12.290 KiB/sec and 24.5314 ops/sec) Additional info: Tangentially related to bug 1601310 in that both are in regards to handling of unaligned POSIX files. Upstream patches proposed: https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg00305.html
Don't forget to also backport: 89aa0d87634 Author: Vladimir Sementsov-Ogievskiy <vsementsov> Date: Fri Apr 27 17:20:01 2018 +0300 nbd/client: fix nbd_negotiate_simple_meta_context Initialize received variable. Otherwise, is is possible for server to answer without any contexts, but we will set context_id to something random (received_id is not initialized too) and return 1, which is wrong.
(In reply to Eric Blake from comment #0) > Upstream patches proposed: > https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg00305.html These patches were reviewed and discussed, but apparently never merged. What's the current status/plan?
Upstream still does not have patches for these issues; unaligned images are enough of a corner case that it hasn't been my highest priority, but may be fixed in time for qemu 4.0
RHV should not have unaligned raw images. I know about these cases in RHV: - memory dump files - we pad the file to next block size - empty qcow2 images - should not effect qemu-nbd - raw uploads - I think we always round the disk size up, and user cannot truncate disks, so uploading unaligned image should result in padded image (sparse or preallocated). So I think we should be safe, but bug 1649788 tells we are not.
Reproduced this issue as below: Tested with: qemu-kvm-3.1.0-20.module+el8+2904+e658c755 kernel-4.18.0-80.el8 Steps: # printf %1000d 1 > f1 # qemu-nbd -f raw -p 9000 f1 -t # qemu-img convert nbd:localhost:9000 f2 -f raw qemu-img: Protocol error: server sent status chunk with invalid length
(In reply to Eric Blake from comment #4) > Upstream still does not have patches for these issues; unaligned images are > enough of a corner case that it hasn't been my highest priority, but may be > fixed in time for qemu 4.0 Most of the issues related to unaligned images were indeed fixed for upstream qemu 4.0. There are still some issues in the qemu server with alignment when backing files have a narrower alignment than the active layer, which won't be fixed until 4.1, but where the 4.0 client has been taught to work around those problems rather than the 3.1 behavior of failing the connection.
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks
I tested this issue with: qemu-kvm-5.1.0-10.module+el8.3.0+8254+568ca30d kernel-4.18.0-221 I got the expected results: # qemu-img map --output=json nbd://localhost:10809 [{ "start": 0, "length": 1000, "depth": 0, "zero": false, "data": true, "offset": 0}, { "start": 1000, "length": 24, "depth": 0, "zero": true, "data": true, "offset": 1000}] # kill $! # nbdkit file file=file & [2] 28418 [1] Done qemu-nbd -f raw -v -t -x '' file # qemu-io -f raw -c 'r 0 513' nbd://localhost:10809 read 513/513 bytes at offset 0 513 bytes, 1 ops; 00.00 sec (1.548 MiB/sec and 3163.4958 ops/sec) [2]+ Done nbdkit file file=file
Test with latest qemu version qemu-kvm-5.1.0-13.module+el8.3.0+8382+afc3bbea.x86_64, this issue is fixed, so close the bug. Version: qemu-kvm-5.1.0-13.module+el8.3.0+8382+afc3bbea.x86_64 Test Steps: 1. printf %01000d 0 > file 2. qemu-nbd -f -p 10850 raw -v -t -x '' file & 3. qemu-img map --output=json nbd://localhost:10850 [{ "start": 0, "length": 1000, "depth": 0, "zero": false, "data": true, "offset": 0}, { "start": 1000, "length": 24, "depth": 0, "zero": true, "data": true, "offset": 1000}] 4. kill $! 5. nbdkit file file=file & 6. qemu-io -f raw -c 'r 0 513' nbd://localhost:10809 read 513/513 bytes at offset 0 513 bytes, 1 ops; 00.01 sec (48.148 KiB/sec and 96.1076 ops/sec) [1]+ Done nbdkit file file=file 7. kill $! Actual result: Results as above, after step 3 & 6. Expected result: Same as actual result.