RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2029751 - nbdkit-cow-filter does not allow cow-block-size=4096
Summary: nbdkit-cow-filter does not allow cow-block-size=4096
Keywords:
Status: CLOSED ERRATA
Alias: None
Deadline: 2022-02-14
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: nbdkit
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: mxie@redhat.com
URL:
Whiteboard:
Depends On: 2011709
Blocks: 2011713
TreeView+ depends on / blocked
 
Reported: 2021-12-07 08:40 UTC by Richard W.M. Jones
Modified: 2022-05-17 13:12 UTC (History)
9 users (show)

Fixed In Version: nbdkit-1.28.3-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-17 12:50:41 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-104942 0 None None None 2021-12-07 08:48:10 UTC
Red Hat Product Errata RHEA-2022:2408 0 None None None 2022-05-17 12:51:08 UTC

Description Richard W.M. Jones 2021-12-07 08:40:22 UTC 
Description of problem:

Because of a bug in a boundary check, nbdkit-cow-filter does not
allow the valid setting cow-block-size=4096.  This affects virt-v2v
because to get better performance we'd like to use this block size.

Version-Release number of selected component (if applicable):

nbdkit-1.28.2-2.el9.x86_64

How reproducible:

100%

Steps to Reproduce:

The cow-block-size parameter is available:

$ nbdkit --filter=cow null --help | grep block-size
cow-block-size=<N>       Set COW block size.

If you try to set it to 4096, it gives an error:

$ nbdkit --filter=cow null cow-block-size=4096 --run true
nbdkit: error: cow-block-size is out of range (4096..2G) or not a power of 2

If you set it to another power of 2 value, it's OK:

$ nbdkit --filter=cow null cow-block-size=8192 --run true
Comment 1 Richard W.M. Jones 2021-12-07 08:40:59 UTC 
Upstream fix:

https://gitlab.com/nbdkit/nbdkit/-/commit/c6fe9cb5b4a45e026f6c45ff3f92a510731d87b9
Comment 2 Richard W.M. Jones 2021-12-07 08:44:52 UTC 
Add bug 2011709 because we'll probably pick this fix up in a rebase.
Comment 3 mxie@redhat.com 2021-12-17 04:49:10 UTC 
Test the bug with nbdkit-1.28.3-2.el9.x86_64

Steps:
1.Check the man page of nbdkit-cow-filter
# man nbdkit-cow-filter  |grep cow-block-size -A 3
                                   [cow-block-size=N]
                                   [cow-on-cache=false|true]
                                   [cow-on-read=false|true|/PATH]

--
       cow-block-size=N
           Set the block size used by the filter.  This has to be a power of two and the minimum block size is 4K.
           The maximum block size depends on the plugin, but a block size larger than a few megabytes is not
           usually a good idea.

2. # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=0
....
nbdkit: error: cow-block-size is out of range (4096..2G) or not a power of 2
....

3.Convert a guest from VMware with nbdkit and add cow-block-size=4096/4k to command line

3.1 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=4096 
.....
real	6m53.898s
user	0m12.463s
sys	1m40.819s


3.2 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=4k
....
real	7m34.770s
user	0m10.748s
sys	1m26.901s


4.Convert a guest from VMware with nbdkit and add cow-block-size=8196/8K to command line
4.1 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=8192
.....
real	7m37.623s
user	0m11.539s
sys	1m32.833s

# time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=8K
....
real	7m20.561s
user	0m10.992s
sys	1m28.543s


Hi rjones,
   
  It's strange that setting cow-block-size=4k takes slightly longer to convert than setting cow-block-size=4096, I get same result after trying many times, is it normal?
Comment 6 Richard W.M. Jones 2021-12-17 10:09:28 UTC 
It's pretty strange and I don't understand it.  However the block size is
definitely set to 4096 in both cases.  In the logs you can see the cow
filter getting a 512 byte request from the client, and making a 4096 byte request
to the underlying plugin (in both logs):

nbdkit: vddk.2: debug: cow: pread count=512 offset=0
nbdkit: vddk.2: debug: vddk: pread count=4096 offset=0

If this is consistently reproducible, I can take a closer look when I get back from holiday.
Comment 9 mxie@redhat.com 2022-01-20 03:52:30 UTC 
Verify the bug with nbdkit-1.28.4-2.el9.x86_64

Steps:
1.Check the man page of nbdkit-cow-filter
# man nbdkit-cow-filter  |grep cow-block-size -A 3
                                   [cow-block-size=N]
                                   [cow-on-cache=false|true]
                                   [cow-on-read=false|true|/PATH]

--
       cow-block-size=N
           Set the block size used by the filter.  This has to be a power of two and the minimum block size is 4K.
           The maximum block size depends on the plugin, but a block size larger than a few megabytes is not
           usually a good idea.

2. # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=0
....
nbdkit: error: cow-block-size is out of range (4096..2G) or not a power of 2
....

3.Convert a guest from VMware with nbdkit and add cow-block-size=4096/4K/4k to command line

3.1 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=4096 
.....
real	6m44.493s
user	0m12.028s
sys	1m33.559s


3.2 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=4K
....
real	7m19.641s
user	0m12.294s
sys	1m33.509s

3.3 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=4k
....
real	7m21.094s
user	0m12.076s
sys	1m32.004s


4.Convert a guest from VMware with nbdkit and add cow-block-size=8196/8K to command line
4.1 # time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=8192
.....
real	7m37.161s
user	0m11.950s
sys	1m32.963s

# time nbdkit -rfv -U - --exportname --filter=retry  vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-6300  file='[esx7.0-matrix] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' libdir=/home/vddk7.0.2 thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbdssl  compression=skipz --run 'qemu-img convert $nbd /home/esx7.0-win11-x86_64' --filter=cacheextents --filter=delay rdelay=20ms --filter=cow cow-on-read=true cow-block-size=8K
....
real	7m38.320s
user	0m11.771s
sys	1m33.122s


Hi rjones,
   
  I still can reproduce the problem 'cow-block-size=4k/4K takes slightly longer to convert than setting cow-block-size=4096, do you think this problem deserves our attention?
Comment 10 Richard W.M. Jones 2022-01-20 08:47:30 UTC 
I think just to verify this bug on its own, all that is needed is:

  $ nbdkit --filter=cow null cow-block-size=4096 --run true

(which should exit with no error).  Previously trying to set this gave
an error.

The performance benefits of cow-block-size=4096 are difficult to measure
in the way you're doing for a few reasons:

 - The actual performance benefit was with fstrim (not copying).  It's very
   difficult to test fstrim on its own, it can only really be measured as
   part of the whole virt-v2v process.  See the commit message for more details:
   https://github.com/libguestfs/virt-v2v/commit/351d61f768c51287917a04b9fbedf24d79f5deb4

 - Virt-v2v now uses nbdcopy, not qemu-img convert.

I would leave performance testing in the bug / email thread that we already
have and not try to do it here.
Comment 11 mxie@redhat.com 2022-01-20 09:36:24 UTC 
# nbdkit --filter=cow null cow-block-size=4096 --run true
Result:exit with no error

Move the bug from ON_QA to VERIFIED  according to comment9 and comment10
Comment 13 errata-xmlrpc 2022-05-17 12:50:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: nbdkit), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2408
Note You need to log in before you can comment on or make changes to this bug.