New version is nbdkit-1.28.1-1.el9

Note this is a rebase along the upstream stable branch and so contains
only bug fixes (except for the new feature added in bug 2018463).

Setting Verified Tested to move the process along.  There is a build
which passed the internal tests:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1776921

nbdkit-1.28.3-1.el9

This is a simple rebase along the stable branch, containing only bug fixes:
https://gitlab.com/nbdkit/nbdkit/-/commits/stable-1.28/

nbdkit-1.28.4-1.el9

This is another simple rebase along the stable branch, bringing
only bug fixes:
https://gitlab.com/nbdkit/nbdkit/-/commits/stable-1.28/

As before this is an update along the stable branch bringing in
a few more upstream fixes before RHEL 9.0.

Verify the bug with nbdkit-1.28.5-1.el9.x86_64

Steps:
1.Rebuild nbdkit rpm packages from src package

$ rpmbuild --rebuild nbdkit-1.28.5-1.el9.src.rpm 
....
libtool: warning: remember to run 'libtool --finish /usr/lib64/nbdkit/filters'
....
+ exit 0

2.Check if '-D cow.verbose=1' can restore the old debug output

2.1 # nbdkit -fv --filter=cow memory 10k --run 'nbdsh -u $uri -c "h.pwrite(bytearray(1024), 0)"' |& grep 'debug: cow: blk'

2.2 # nbdkit -fv --filter=cow memory 10k --run 'nbdsh -u $uri -c "h.pwrite(bytearray(1024), 0)"' -D cow.verbose=1 |& grep 'debug: cow: blk'
nbdkit: memory.0: debug: cow: blk_read_multiple block 0 (offset 0) run of length 1 is not allocated
nbdkit: memory.0: debug: cow: blk_write block 0 (offset 0)

3.Check default block size for nbdkit-cache-filter and nbdkit-cow-filter 

# man nbdkit-cache-filter |grep 64K -B 3
       cache-min-block-size=SIZE
           Set the minimum block size used by the cache.  This must be a power of 2 and ≥ 4096.

           The default is 64K, or the block size of the filesystem which contains the temporary file storing the cache (whichever


# man nbdkit-cow-filter |grep 64K -B 4
       cow-block-size=N
           Set the block size used by the filter.  This has to be a power of two and the minimum block size is 4K.  The maximum
           block size depends on the plugin, but a block size larger than a few megabytes is not usually a good idea.

          The default is 64K.

4.Convert a guest with vddk.stats=1 by nbdkit
# # nbdkit --exit-with-parent --threads 16 -D nbdkit.backend.datapath=0 -D vddk.stats=1 -D vddk.datapath=0 --verbose --filter multi-conn --filter cow --filter cacheextents --filter retry vddk server=10.73.198.169 vm=moref=vm-6306 file='[esx7.0-matrix] esx7.0-rhel8.5-x86_64_1/esx7.0-rhel8.5-x86_64.vmdk' user=root libdir=/home/vddk7.0.2 thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 multi-conn-mode=disable cow-block-size=4096 cow-on-read=/tmp/v2v.6l3Qwq/convert password=+/home/passwd --run 'qemu-img convert $nbd /home/esx7.0-rhel8.5-x86_64'
.....
nbdkit: debug: VDDK function stats (-D vddk.stats=1):
nbdkit: debug: VixDiskLib_...                        µs calls           bytes
nbdkit: debug:   Exit                           1002565     1
nbdkit: debug:   ConnectEx                       707986     1
nbdkit: debug:   ReadAsync                       385188 22530     17209233408
nbdkit: debug:   Open                            286491     1
nbdkit: debug:   InitEx                          129163     1
nbdkit: debug:   QueryAllocatedBlocks             35957    19
nbdkit: debug:   Disconnect                        4481     1
nbdkit: debug:   GetInfo                           3411     1
nbdkit: debug:   Close                             2472     1
nbdkit: debug:   FreeBlockList                      324    19
nbdkit: debug:   Wait                                37     1
nbdkit: debug:   FreeInfo                            28     1
nbdkit: debug:   FreeConnectParams                   23     1
nbdkit: debug:   GetTransportMode                    18     1
nbdkit: debug:   AllocateConnectParams                9     1
nbdkit: debug: retry: unload filter
nbdkit: debug: cacheextents: unload filter
nbdkit: debug: cow: unload filter
nbdkit: debug: multi-conn: unload filter



Hi Richard,

    nbdkit 1.28 release note said 'nbdkit-cow-filter...In addition the default block size for both filters is now 64K. Both changes greatly improve performance', my understanding is 64k should be the best size for performance for cow filter，but I found v2v use 'cow-block-size=4096' in conversion, is it expected?

   # cat virt-v2v-1.45.97-2.log |grep 'cow-block-size=4096'
Jan 26 11:16:41  LANG=C 'nbdkit' '--exit-with-parent' '--foreground' '--pidfile' '/tmp/v2vnbdkit.P1jGNB/nbdkit1.pid' '--unix' '/tmp/v2v.UgyuQD/in0' '--threads' '16' '--selinux-label' 'system_u:object_r:svirt_socket_t:s0' '-D' 'nbdkit.backend.datapath=0' '-D' 'vddk.stats=1' '-D' 'vddk.datapath=0' '--verbose' '--filter' 'multi-conn' '--filter' 'cow' '--filter' 'cacheextents' '--filter' 'retry' 'vddk' 'server=10.73.198.169' 'vm=moref=vm-6306' 'file=[esx7.0-matrix] esx7.0-rhel8.5-x86_64_1/esx7.0-rhel8.5-x86_64.vmdk' 'user=vsphere.local\Administrator' 'libdir=/home/vddk7.0.2' 'thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' 'multi-conn-mode=disable' 'cow-block-size=4096' 'cow-on-read=/tmp/v2v.UgyuQD/convert' 'password=+/home/passwd'

>     nbdkit 1.28 release note said 'nbdkit-cow-filter...In addition the default block size for both filters is now 64K. Both changes greatly improve performance', my understanding is 64k should be the best size for performance for cow filter，but I found v2v use 'cow-block-size=4096' in conversion, is it expected?

The release notes for 1.28 were wrong.  The optimal block size for nbdkit-cow-filter
depends on how the filter is used, but for our use case in virt-v2v using a smaller
block size gave better performance by avoiding read-modify-write cycles in the
fstrim step:

https://github.com/libguestfs/virt-v2v/commit/351d61f768c51287917a04b9fbedf24d79f5deb4

Thanks rjones, move the bug from ON_QA to VERIFIED according to comment8 and comment9

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: nbdkit), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2408