Bug 2135765

Setting ITR=9.2.0 and DTM=15 just to give a target. 

This needs to be completed before 15-Jan in order to be ready for Comprehensive Test Cycle 2

Upstream, there were not sufficient changes to justify a whole
new stable release.  So for RHEL I have decided to simply rebase
along the stable branch to the latest stable version (1.32.5).
Changes here are fairly minimal and should be only bug fixes:

https://gitlab.com/nbdkit/nbdkit/-/commits/stable-1.32/

Since we are updating from 1.30 to 1.32, please see the release notes here:
https://libguestfs.org/nbdkit-release-notes-1.32.1.html

Test the bug with nbdkit-server-1.32.5-2.el9.x86_64

Steps:
1. Rebuild nbdkit rpm packages from src package
#  rpmbuild --rebuild nbdkit-1.32.5-2.el9.src.rpm 
....
+ exit 0

2.Test new options 'create=(true|false), create-size and create-mode' of nbdkit-ssh-plugin
2.1 # nbdkit -U - ssh host=10.73.212.36 /var/tmp/new-disk user=root password=+/home/esxpwd create=true create-mode=0777 create-size=100M --run 'nbdinfo --can connect $uri'

Result2.1: disk can be created successfully on remote host

# ll /var/tmp/new-disk
-rwxrwxrwx    1 root     root     104857600 Jan  6 02:23 /var/tmp/new-disk

2.2 # nbdkit -U - ssh host=10.73.212.36 /var/tmp/newdisk user=root password=+/home/esxpwd create=false create-mode=0777 create-size=100M --run 'nbdinfo --can connect $uri'
nbdkit: ssh[1]: error: cannot open file for writing: SFTP server: No such file
nbdinfo: nbd_connect_uri: handshake: server has no export named '': No such file or directory

Result2.2: disk won't be create when create=false 


3.Test basic function of nbdkit-luks-filter

3.1 Prepare a LUKSv1-encrypted image which is converted from VMware by v2v, then use nbdkit luks filer to read the disk image
# # nbdkit file esx6.7-rhel7.8-swap_lvm_luks-redhat123-sda --filter=luks passphrase=+/home/diskpw --filter=partition partition=1 --run 'nbdcopy $nbd vmware-luks-disk.img' 
nbdkit: file[1]: error: this disk does not contain a LUKS header
nbdcopy: nbd://localhost:10809: nbd_connect_uri: recv: server disconnected unexpectedly
nbdkit: nbdkit command was killed by signal 11

Result3.1: luks filter can't read LUKSv1-encrypted disk image 

3.2 Prepare a luks encrypted image which is created by qemu-img, then use nbdkit luks filer to read the disk image

3.2.1 # qemu-img create -f luks --object secret,data=LETMEPASS,id=sec0 -o key-secret=sec0 encrypted.img 100M
Formatting 'encrypted.img', fmt=luks size=104857600 key-secret=sec0

3.2.2 # nbdkit file encrypted.img --filter=luks passphrase=LETMEPASS --run 'nbdcopy $nbd data.img'

3.2.3 # nbdkit file data.img --filter=luks passphrase=LETMEPASS --run 'nbdcopy $nbd data-b.img'
nbdkit: file[1]: error: this disk does not contain a LUKS header
nbdcopy: nbd://localhost:10809: nbd_connect_uri: recv: server disconnected unexpectedly
nbdkit: nbdkit command was killed by signal 11

Result3.2: according to step3.2.3, seems luks filter doesn't read LUKS-encrypted disk image correctly


4.Test basic function of nbdkit-scan-filter

4.1 Check thread_model of vddk plugin
# nbdkit vddk --dump-plugin | grep thread
max_thread_model=parallel
thread_model=parallel

4.2 Test scan filter with vddk plugin
#  nbdkit -rf -U  --exportname='' --exit-with-parent --filter=multi-conn  --filter=scan  --filter=cow --filter=cacheextents --filter=retry  vddk server=10.73.212.149 user=root password=+/home/passwd vm=moref=vm-145 file='[esx8.0-matrix] esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmdk' libdir=/home/vddk8.0.0  thumbprint='D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94' cow-on-read=true  cow-on-cache=true scan-ahead=true  scan-clock=true scan-size=2048 scan-forever=true   --run 'nbdinfo $uri'
nbdkit: vddk[1]: error: scan: warning: underlying plugin does not support the PARALLEL thread model, not scanning
nbdkit: vddk[1]: error: scan: warning: underlying plugin does not support the PARALLEL thread model, not scanning
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd+unix:///?socket=/root/--exportname=
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: true
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: true
	can_fua: false
	can_multi_conn: true
	can_trim: false
	can_zero: false
	block_size_minimum: 512
	block_size_preferred: 65536
	block_size_maximum: 4294967295

Result4.2： scan filter can't be used even if thread_model of plugin is parallel


5.Test new option burstiness for nbdkit-rate-filter 

#  nbdkit --filter=rate memory 64M rate=1M connection-rate=500K burstiness=20 --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 67108864 (64M)
	content: data
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

Hi Richard,

   Please help to check the problems of step3.1, step3.2 and step4.2, thanks!

For 3.1 and 3.2:

  nbdkit: nbdkit command was killed by signal 11

This should never happen :-(  Please file a new bug about it.  It
would be very useful if you could collect the stack trace from
the crash [coredumpctl gdb ; t a a bt], and even better if there
was a way for me to get a copy of the disk image.

For 4.2:

  nbdkit: vddk[1]: error: scan: warning: underlying plugin does not support the PARALLEL thread model, not scanning

Since the scan filter is new (added May 2022) and it always required the
plugin to support PARALLEL threads, and since VDDK has never supported PARALLEL,
I don't think this is a regression?

(In reply to Richard W.M. Jones from comment #7)

> For 4.2:
> 
>   nbdkit: vddk[1]: error: scan: warning: underlying plugin does not support
> the PARALLEL thread model, not scanning
> 
> Since the scan filter is new (added May 2022) and it always required the
> plugin to support PARALLEL threads, and since VDDK has never supported
> PARALLEL,
> I don't think this is a regression?

Remove retry filter from the command, then scan filter can be used with vddk plugin

#  nbdkit --filter=scan  --filter=cow  --filter=multi-conn --filter=cacheextents vddk server=10.73.212.149 user=root password=+/home/passwd vm=moref=vm-145 file='[esx8.0-matrix] esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmdk' libdir=/home/vddk8.0.0  thumbprint='D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94' cow-on-read=true  cow-on-cache=true scan-ahead=true  scan-clock=true scan-size=2048 scan-forever=true   --run 'nbdinfo $uri' 
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: false
	block_size_minimum: 512
	block_size_preferred: 65536
	block_size_maximum: 4294967295

OK I see, in fact VDDK plugin is PARALLEL (since nbdkit 1.30).

However adding other filters, such as nbdkit-retry-filter, may
reduce the overall thread model.

In any case this isn't a regression or a bug in the scan filter,
it has this limitation and always had it.

Verify the bug with nbdkit-1.32.5-3.el9.x86_64

Steps:
1. Rebuild nbdkit rpm packages from src package
#  rpmbuild --rebuild nbdkit-1.32.5-3.el9.src.rpm 
....
+ exit 0

2.Test new options 'create=true create-size and create-mode' of nbdkit-ssh-plugin

#  nbdkit ssh host=10.73.224.33 /var/tmp/newdisk user=root password=+/home/esxpwd create=true --run 'nbdinfo --can connect $uri'
nbdkit: error: if using create=true, you must specify the size of the new remote file using create-size=SIZE

# nbdkit ssh host=10.73.224.33 /var/tmp/new-disk user=root password=+/home/esxpwd create=true create-mode=0644 create-size=100M --run 'nbdinfo --can connect $uri'

# ll /var/tmp/new-disk 
-rw-r--r-- 1 root root 104857600 Jan 12 04:16 /home/new-disk

Result 2.1: disk can be created successfully on remote xen host


3.Test basic function of nbdkit-readahead-filter

#  nbdkit vddk --filter=readahead --dump-plugin | grep thread
max_thread_model=parallel
thread_model=parallel

# nbdkit  --filter=readahead  --filter=scan --filter=cache --filter=multi-conn --filter=cacheextents vddk server=10.73.212.149 user=root password=+/home/passwd vm=moref=vm-145 file='[esx8.0-matrix] esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmdk' libdir=/home/vddk8.0.0  thumbprint='D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94' scan-ahead=true  scan-clock=true scan-size=2048 scan-forever=true   --run 'nbdinfo $uri' 
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: false
	can_zero: true
	block_size_minimum: 512
	block_size_preferred: 65536
	block_size_maximum: 4294967295

#nbdkit  --filter=readahead  --filter=cache file encrypted.img  --filter=luks passphrase=LETMEPASS --run 'nbdcopy $nbd data.img' 

Result3: no error shows when use readahead filter with vddk and file plugins


4.Test basic function of nbdkit-stats-filter

# nbdkit --filter=exitlast --filter=stats memory 25G statsfile=example.txt

# nbdcopy data.img nbd://localhost

# cat example.txt 
total: 400 ops, 411.709456 s, 100.00 MiB, 248.72 KiB/s
write: 400 ops, 0.133221 s, 100.00 MiB, 750.63 MiB/s op, 248.72 KiB/s total
  Request size and alignment breakdown:
    18 bits: 100.0% (400 reqs, 100.00 MiB total)
         18 bit aligned: 100.0% (400)
         19 bit aligned:  50.0% (200)
         20 bit aligned:  25.0% (100)
         21 bit aligned:  12.5% (50)
         22 bit aligned:   6.2% (25)
         23+ bit-aligned:  3.2% (13)

Result4: can see block size and alignment of requests in the file where write the stats.


5.  Check thread_model of blocksize filter
#  nbdkit vddk --filter=blocksize --dump-plugin | grep thread
max_thread_model=parallel
thread_model=parallel

# nbdkit --filter=blocksize vddk server=10.73.212.149 user=root password=+/home/passwd vm=moref=vm-145 file='[esx8.0-matrix] esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmdk' libdir=/home/vddk8.0.0  thumbprint='D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94'  --run 'nbdinfo $uri' 
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: false
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: false
	can_trim: false
	can_zero: true
	block_size_minimum: 1
	block_size_preferred: 4096
	block_size_maximum: 4294967295


# nbdkit --filter=blocksize --filter=cache vddk server=10.73.212.149 user=root password=+/home/passwd vm=moref=vm-145 file='[esx8.0-matrix] esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmdk' libdir=/home/vddk8.0.0  thumbprint='D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94'  --run 'nbdinfo $uri' 
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: false
	can_zero: true
	block_size_minimum: 1
	block_size_preferred: 65536
	block_size_maximum: 4294967295

Result5: block_size_preferred will be increased if use blocksize filter with cache filter


Hi Richard,
   Please help to check result5, is it expected? Thanks!

The blocksize filter just passes through the preferred block size
from the underlying chain:
https://gitlab.com/nbdkit/nbdkit/-/blob/09a61a0a00d3036ebfbc2fbadb0ba3e10a778ff3/filters/blocksize/blocksize.c#L186

Adding the cache filter into the chain changes the preferred block
size because the cache filter uses 64K blocks by default:
https://gitlab.com/nbdkit/nbdkit/-/blob/09a61a0a00d3036ebfbc2fbadb0ba3e10a778ff3/filters/cache/cache.c#L75

If you want you can change the cache filter block size, eg:

$ nbdkit memory 1M --filter=blocksize \
    --run 'nbdinfo --json $uri | jq -r ".exports[0].\"block_size_preferred\"" '
4096

$ nbdkit memory 1M --filter=blocksize --filter=cache \
    --run 'nbdinfo --json $uri | jq -r ".exports[0].\"block_size_preferred\"" '
65536

$ nbdkit memory 1M --filter=blocksize --filter=cache cache-min-block-size=32K \
    --run 'nbdinfo --json $uri | jq -r ".exports[0].\"block_size_preferred\"" '
32768

$ nbdkit memory 1M --filter=blocksize --filter=cache cache-min-block-size=4K --run 'nbdinfo --json $uri | jq -r ".exports[0].\"block_size_preferred\"" '
4096

Thanks for explanation, move the bug from ON_QA to VERIFIED according to comment10 and comment11

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nbdkit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2347