Bug 1913740

Summary:	Add workaround for qemu reduced maximum disk size
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Richard W.M. Jones <rjones>
Component:	nbdkit	Assignee:	Richard W.M. Jones <rjones>
Status:	CLOSED ERRATA	QA Contact:	liuzi <zili>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	eblake, juzhou, mxie, rjones, tyan, tzheng, virt-maint, xiaodwan, zili
Target Milestone:	rc	Keywords:	Triaged
Target Release:	8.4
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	nbdkit-1.24.0-1.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1966511 (view as bug list)		Environment:
Last Closed:	2021-05-25 06:46:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1906049
Bug Blocks:	1966511

Description Richard W.M. Jones 2021-01-07 13:51:13 UTC

Description of problem:

This qemu commit reduced the maximum disk size supported from
2⁶³ - 512 bytes to 2⁶³ - 2³⁰ bytes:

  https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8b1170012b1de6649c66ac1887f4df7e312abf3b

Since nbdkit was testing qemu’s older limit, we must adjust the
streaming plugin and a couple of tests which embedded this limit.

Note this isn't really a bug in nbdkit.

Version-Release number of selected component (if applicable):

nbdkit-1.22.0-2.scrmod+el8.4.0+9296+87860477
qemu-15:5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106

How reproducible:

100%

Steps to Reproduce:

$ nbdkit memory $(( 2**63 - 512 )) --run 'qemu-img info "$uri"'

Actual results:

qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large

The above command used to work with earlier qemu.

Expected results:

There are no expected good results, since qemu has reduced the
maximum size of disks that it supports.

Instead we have fixed the nbdkit tests so they test the reduced
maximum size.

Additional info:

nbdkit upstream fix:

https://github.com/libguestfs/nbdkit/commit/c3ec8c951e39a0f921252c162c236f23c588d2bd

Comment 1 Richard W.M. Jones 2021-01-07 14:03:29 UTC

For QE:

There is one visible change in nbdkit-streaming-plugin:

  $ mkfifo pipe
  $ cat pipe > out &
  [1] 1697288
  $ qemu-img create -f qcow2 qcow2 1G
  Formatting 'qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
  $ nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" '
  qemu-img: Could not open 'nbd+unix://?socket=/tmp/nbdkitRIsyC3/socket': Could not refresh total sector count: File too large
  $ rm pipe qcow2 out

This should be fixed by this nbdkit change.

Also previously the nbdkit build was broken:

  https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=34115176

We're expecting that builds will NOT be broken after this change in nbdkit.

Comment 2 Richard W.M. Jones 2021-01-08 09:40:19 UTC

This change will be made as a side-effect of the rebase (bug 1906049).

Comment 8 liuzi 2021-01-29 06:34:24 UTC

Reproduce the bug with builds:
qemu-kvm-5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106.x86_64
nbdkit-1.22.0-2.module+el8.4.0+8855+a9e237a9.x86_64

Steps:
1.Use nbdkit memory plugin to create the largest possible RAM disk and use qemu-img to check the disk:
#  nbdkit memory $(( 2**63 - 1 )) --run 'qemu-img info "$uri"'
qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large

2.Combine the nbdkit-streaming-plugin with qemu-img
# qemu-img create -f qcow2 qcow2 1G
Formatting 'qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
# nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" '
qemu-img: Could not open 'nbd+unix://?socket=/tmp/nbdkito619UR/socket': Could not refresh total sector count: File too large




Verify the bug with builds:
qemu-kvm-5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64

Steps:
1.Use nbdkit memory plugin to create the largest possible RAM disk and use qemu-img to check the disk info:
# nbdkit memory $(( 2**63 - 1 )) --run 'qemu-img info "$uri"'
qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large

Result 1: qemu-img cannot check the disk info.

2.Combine the nbdkit-streaming-plugin with qemu-img
# qemu-img create -f qcow2 qcow2 1G
Formatting 'qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
# nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" '
    (0.00/100%)

Result 2:The command blocked,shows (0.00/100%) more than 30mins.


Hi,Rjones,
Pls help to check the verify results,Did I miss any information during the test?if not,I think the bug isn't be fixed.
Thanks!

Comment 9 Richard W.M. Jones 2021-01-29 08:32:53 UTC

The test of nbdkit streaming plugin is a correct reproducer and verification.

The nbdkit memory ... --run 'qemu-img ...' commands are wrong however:

With qemu <= 5.2.0 (eg. the version of qemu in RHEL AV 8.4.0) the maximum
size qemu can handle is $(( 2**63 - 512 )).

So this works:

  qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64

  $ nbdkit memory $(( 2**63 - 512 )) --run 'qemu-img info "$uri"'
  image: nbd://localhost:10809
  file format: raw
  virtual size: 8 EiB (9223372036854775296 bytes)
  disk size: unavailable

With qemu > 5.2.0 (eg. wrb package) the same command fails:

  qemu-kvm-5.2.0-3.scrmod+el8.4.0+9533+93b6ae37.wrb210120.x86_64

  $ nbdkit memory $(( 2**63 - 512 )) --run 'qemu-img info "$uri"'
  qemu-img: Could not open 'nbd://localhost:10809': Could not refresh total sector count: File too large

qemu > 5.2.0 reduced the largest disk size it can handle to $(( 2**63 - 2**30 ))
so this works instead:

  $ nbdkit memory $(( 2**63 - 2**30 )) --run 'qemu-img info "$uri"'
  image: nbd://localhost:10809
  file format: raw
  virtual size: 8 EiB (9223372035781033984 bytes)
  disk size: unavailable

but anything larger, eg. $(( 2**63 - 2**30 + 512 )) will fail.

Note the version of nbdkit does not matter when testing this part, because this
isn't a bug in nbdkit, it's a change in the behaviour of qemu.

Comment 10 liuzi 2021-01-29 12:12:06 UTC

Hi,Rjones
Thanks for you kindly reply,I still have some problem:
1.As you said,the test of nbdkit streaming plugin is correct,but the command cannot execute successfully.
The command will hang on more than 30mins  without result.

# nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2 qcow2 -O raw "$uri" '
    (0.00/100%)

2.For the largest disk size,I only find related info in nbdkit-memory-plugin man-page.Should qemu update info in man-page?

Thanks!

Comment 11 Richard W.M. Jones 2021-01-29 12:19:40 UTC

(In reply to liuzi from comment #10)
> Hi,Rjones
> Thanks for you kindly reply,I still have some problem:

Oh my apologies I didn't see your other question from earlier.

> 1.As you said,the test of nbdkit streaming plugin is correct,but the command
> cannot execute successfully.
> The command will hang on more than 30mins  without result.
> 
> # nbdkit -U - streaming write=pipe --run ' qemu-img convert -p -n -f qcow2
> qcow2 -O raw "$uri" '
>     (0.00/100%)

You missed out the mkfifo and cat steps :-)  The reason it was stuck
at 0/100 is because there was no "cat" process writing to the pipe.

If you follow the full steps in comment 1 then it should work.

> 2.For the largest disk size,I only find related info in nbdkit-memory-plugin
> man-page.Should qemu update info in man-page?

I don't know if qemu want to document this.  They seem to have made
the change unintentionally in the first place.  Something for the qemu
team to answer anyway.

Comment 12 liuzi 2021-02-02 03:24:48 UTC

Thanks for Rjones reply.I will send email to qemu team for the largest disk problem.

Verify the bug with builds:
qemu-kvm-5.2.0-2.scrmod+el8.4.0+9296+87860477.wrb210106.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64

Steps:
1.Use nbdkit memory plugin to create the largest possible RAM disk and use qemu-img to check the disk info:
#nbdkit memory $(( 2**63 - 2**30 )) --run 'qemu-img info "$uri"'
image: nbd://localhost:10809
file format: raw
virtual size: 8 EiB (9223372035781033984 bytes)
disk size: unavailable


2.Combine the nbdkit-streaming-plugin with qemu-img
# mkfifo pipe
# cat pipe > out &
[1] 642654
# qemu-img create -f qcow2 qcow2 1G
Formatting 'qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
# nbdkit memory $(( 2**63 - 2**30 )) --run 'qemu-img info "$uri"'
image: nbd://localhost:10809
file format: raw
virtual size: 8 EiB (9223372035781033984 bytes)
disk size: unavailable


Result: The command can be finished without error info.so change the bug from ON_QA to Verified.

Comment 13 Richard W.M. Jones 2021-02-02 08:43:00 UTC

In the end the change was intentional, see Vladimir's reply here:

https://lists.nongnu.org/archive/html/qemu-devel/2021-01/msg01253.html

Comment 15 errata-xmlrpc 2021-05-25 06:46:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098