Description of problem: This qemu commit reduced the maximum disk size supported from 2⁶³ - 512 bytes to 2⁶³ - 2³⁰ bytes: https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8b1170012b1de6649c66ac1887f4df7e312abf3b Since nbdkit was testing qemu’s older limit, we must adjust the streaming plugin and a couple of tests which embedded this limit. Note this isn't really a bug in nbdkit. Version-Release number of selected component (if applicable): nbdkit-1.22.0-2.scrmod+el8.4.0+9296+87860477 qemu-15:5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106 How reproducible: 100% Steps to Reproduce: $ nbdkit memory $(( 2**63 - 512 )) --run 'qemu-img info "$uri"' Actual results: qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large The above command used to work with earlier qemu. Expected results: There are no expected good results, since qemu has reduced the maximum size of disks that it supports. Instead we have fixed the nbdkit tests so they test the reduced maximum size. Additional info: nbdkit upstream fix: https://github.com/libguestfs/nbdkit/commit/c3ec8c951e39a0f921252c162c236f23c588d2bd
For QE: There is one visible change in nbdkit-streaming-plugin: $ mkfifo pipe $ cat pipe > out & [1] 1697288 $ qemu-img create -f qcow2 qcow2 1G Formatting 'qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 $ nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" ' qemu-img: Could not open 'nbd+unix://?socket=/tmp/nbdkitRIsyC3/socket': Could not refresh total sector count: File too large $ rm pipe qcow2 out This should be fixed by this nbdkit change. Also previously the nbdkit build was broken: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=34115176 We're expecting that builds will NOT be broken after this change in nbdkit.
This change will be made as a side-effect of the rebase (bug 1906049).
Reproduce the bug with builds: qemu-kvm-5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106.x86_64 nbdkit-1.22.0-2.module+el8.4.0+8855+a9e237a9.x86_64 Steps: 1.Use nbdkit memory plugin to create the largest possible RAM disk and use qemu-img to check the disk: # nbdkit memory $(( 2**63 - 1 )) --run 'qemu-img info "$uri"' qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large 2.Combine the nbdkit-streaming-plugin with qemu-img # qemu-img create -f qcow2 qcow2 1G Formatting 'qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 # nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" ' qemu-img: Could not open 'nbd+unix://?socket=/tmp/nbdkito619UR/socket': Could not refresh total sector count: File too large Verify the bug with builds: qemu-kvm-5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106.x86_64 nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64 Steps: 1.Use nbdkit memory plugin to create the largest possible RAM disk and use qemu-img to check the disk info: # nbdkit memory $(( 2**63 - 1 )) --run 'qemu-img info "$uri"' qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large Result 1: qemu-img cannot check the disk info. 2.Combine the nbdkit-streaming-plugin with qemu-img # qemu-img create -f qcow2 qcow2 1G Formatting 'qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 # nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" ' (0.00/100%) Result 2:The command blocked,shows (0.00/100%) more than 30mins. Hi,Rjones, Pls help to check the verify results,Did I miss any information during the test?if not,I think the bug isn't be fixed. Thanks!
The test of nbdkit streaming plugin is a correct reproducer and verification. The nbdkit memory ... --run 'qemu-img ...' commands are wrong however: With qemu <= 5.2.0 (eg. the version of qemu in RHEL AV 8.4.0) the maximum size qemu can handle is $(( 2**63 - 512 )). So this works: qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 $ nbdkit memory $(( 2**63 - 512 )) --run 'qemu-img info "$uri"' image: nbd://localhost:10809 file format: raw virtual size: 8 EiB (9223372036854775296 bytes) disk size: unavailable With qemu > 5.2.0 (eg. wrb package) the same command fails: qemu-kvm-5.2.0-3.scrmod+el8.4.0+9533+93b6ae37.wrb210120.x86_64 $ nbdkit memory $(( 2**63 - 512 )) --run 'qemu-img info "$uri"' qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large qemu > 5.2.0 reduced the largest disk size it can handle to $(( 2**63 - 2**30 )) so this works instead: $ nbdkit memory $(( 2**63 - 2**30 )) --run 'qemu-img info "$uri"' image: nbd://localhost:10809 file format: raw virtual size: 8 EiB (9223372035781033984 bytes) disk size: unavailable but anything larger, eg. $(( 2**63 - 2**30 + 512 )) will fail. Note the version of nbdkit does not matter when testing this part, because this isn't a bug in nbdkit, it's a change in the behaviour of qemu.
Hi,Rjones Thanks for you kindly reply,I still have some problem: 1.As you said,the test of nbdkit streaming plugin is correct,but the command cannot execute successfully. The command will hang on more than 30mins without result. # nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" ' (0.00/100%) 2.For the largest disk size,I only find related info in nbdkit-memory-plugin man-page.Should qemu update info in man-page? Thanks!
(In reply to liuzi from comment #10) > Hi,Rjones > Thanks for you kindly reply,I still have some problem: Oh my apologies I didn't see your other question from earlier. > 1.As you said,the test of nbdkit streaming plugin is correct,but the command > cannot execute successfully. > The command will hang on more than 30mins without result. > > # nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 > qcow2 -O raw "$uri" ' > (0.00/100%) You missed out the mkfifo and cat steps :-) The reason it was stuck at 0/100 is because there was no "cat" process writing to the pipe. If you follow the full steps in comment 1 then it should work. > 2.For the largest disk size,I only find related info in nbdkit-memory-plugin > man-page.Should qemu update info in man-page? I don't know if qemu want to document this. They seem to have made the change unintentionally in the first place. Something for the qemu team to answer anyway.
Thanks for Rjones reply.I will send email to qemu team for the largest disk problem. Verify the bug with builds: qemu-kvm-5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106.x86_64 nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64 Steps: 1.Use nbdkit memory plugin to create the largest possible RAM disk and use qemu-img to check the disk info: #nbdkit memory $(( 2**63 - 2**30 )) --run 'qemu-img info "$uri"' image: nbd://localhost:10809 file format: raw virtual size: 8 EiB (9223372035781033984 bytes) disk size: unavailable 2.Combine the nbdkit-streaming-plugin with qemu-img # mkfifo pipe # cat pipe > out & [1] 642654 # qemu-img create -f qcow2 qcow2 1G Formatting 'qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 # nbdkit memory $(( 2**63 - 2**30 )) --run 'qemu-img info "$uri"' image: nbd://localhost:10809 file format: raw virtual size: 8 EiB (9223372035781033984 bytes) disk size: unavailable Result: The command can be finished without error info.so change the bug from ON_QA to Verified.
In the end the change was intentional, see Vladimir's reply here: https://lists.nongnu.org/archive/html/qemu-devel/2021-01/msg01253.html
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098