Bug 2168629

Summary: [RHEL 9.3] Rebase nbdkit to 1.34
Product: Red Hat Enterprise Linux 9 Reporter: Richard W.M. Jones <rjones>
Component: nbdkitAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: mxie <mxie>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: eblake, lersek, mxie, rjones, tyan, tzheng, virt-maint, vwu, xiaodwan
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nbdkit-1.34.2-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:28:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1841820, 2173054    

Description Richard W.M. Jones 2023-02-09 15:23:24 UTC 
Description of problem:

For RHEL 9.2 the original plan was to rebase to nbdkit 1.34
(https://bugzilla.redhat.com/show_bug.cgi?id=2135765).  We
didn't actually manage to do that.  However it is likely there
will be a new upstream stable release of nbdkit quite soon
and we should consider rebasing to this.

There are no release notes yet, but they will be prepared
as part of the upstream release.

Version-Release number of selected component (if applicable):

libnbd 1.34.x
Comment 1 Richard W.M. Jones 2023-02-09 16:37:35 UTC 
In RHEL 9.2 we only carry 4 downstream patches:
https://gitlab.com/nbdkit/nbdkit/-/commits/rhel-9.2/
Comment 2 Richard W.M. Jones 2023-04-12 16:46:43 UTC 
This bug is in a confusing state.  Shouldn't it be added to an erratum (automatically)?
Comment 4 Richard W.M. Jones 2023-04-13 13:03:18 UTC 
Preliminary release notes for 1.34 are available here:
https://gitlab.com/nbdkit/nbdkit/-/blob/master/docs/nbdkit-release-notes-1.34.pod
Comment 5 Richard W.M. Jones 2023-04-14 09:57:26 UTC 
Final release notes for 1.34 are available here:
https://libguestfs.org/nbdkit-release-notes-1.34.1.html
Comment 6 mxie@redhat.com 2023-04-27 09:25:23 UTC 
Test the bug with nbdkit-server-1.34.1-1.el9.x86_64

Steps:
1.Test new parameters 'fd' and 'dirfd' for nbdkit-file-plugin

1.1 Check help output of nbdkit file plugin 
1.1.1 
# nbdkit file --help
nbdkit [-4|--ipv4-only] [-6|--ipv6-only]
       [-D|--debug PLUGIN|FILTER|nbdkit.FLAG=N]
....
....
nbdkit [-V|--version]

Please read the nbdkit(1) manual page for full usage.

plugin: file (nbdkit file plugin)
(/usr/lib64/nbdkit/plugins/nbdkit-file-plugin.so)
[file=]<FILENAME>       The filename to serve
fd=<FILE_DESCRIPTOR>    Serve file attached to file descriptor
dir=<DIRNAME>           A directory containing files to serve
dirfd=<FILE_DESCRIPTOR> Serve dir attached to file descriptor
cache=<MODE>            Set use of caching (default, none)
fadvise=<LEVEL>         Set fadvise hint (normal, random, sequential)


1.1.2 
# nbdkit file -h
nbdkit: invalid option -- 'h'
nbdkit [-4|--ipv4-only] [-6|--ipv6-only]
       [-D|--debug PLUGIN|FILTER|nbdkit.FLAG=N]
.....

nbdkit [-V|--version]

Please read the nbdkit(1) manual page for full usage.


Result 1.1.1 :  The help info of  both nbdkit and file plugin will be shown
Result 1.1.2 :  Only help info of nbdkit is shown


1.2 Test Option 'fd' with file plugin

1.2.1 Set number of fd lower than 3 
# nbdkit file fd=2  --run 'nbdinfo $uri' 
nbdkit: error: file descriptor must be > 2 because stdin, stdout and stderr are reserved for nbdkit

1.2.2 Set  a nonexistent fd 
# nbdkit file fd=7  --run 'nbdinfo $uri' 
nbdkit: error: fd is not regular or block device: 7

1.2.3 set a valid fd for a disk
# exec 5> /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2 
# nbdkit file fd=5  --run 'nbdinfo $uri' 
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 0
	content: empty
.....

1.2.4 Write info to the fd of step1.2.3
# echo hello >&5
# nbdkit file fd=5  --run 'nbdinfo $uri' 
nbdkit: file.1: error: pread: Bad file descriptor
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 6
.....

Result 1.2.4: not sure if result is expected

1.3 Test Option 'dirfd' with file plugin

1.3.1 Set number of fd lower than 3 
# nbdkit file dirfd=2
nbdkit: error: file descriptor must be > 2 because stdin, stdout and stderr are reserved for nbdkit

1.3.2  Set  a nonexistent dirfd 
# nbdkit file dirfd=9
nbdkit: error: dirfd is not a directory: 9

1.3.3  set a valid dirfd for a disk
# exec 6< /root/dir_file
# nbdkit file dirfd=6 --run 'nbdinfo --list $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="nbdkit-1.34.1-1.el9.src.rpm":
	export-size: 2446366
	uri: nbd://localhost:10809/nbdkit-1.34.1-1.el9.src.rpm
.....

1.3.4 Add fd and dirfd at the same time in command line
# nbdkit file fd=5 dirfd=6 
nbdkit: error: file|dir|fd|dirfd parameter can only appear once on the command line

1.3.5 Create a empty directory and add file descriptor to the directory
# exec 7< /root/empty
# nbdkit file dirfd=7 --run 'nbdinfo --list $uri'
protocol: newstyle-fixed without TLS, using structured packets

Result 1.2 and 1.3: no problems found except not sure if the result 1.3.5 is expected


2. Test new option blocksize-write-disconnect of nbdkit-blocksize-policy-filter

2.1 Set the value of blocksize-minimum smaller than the value of blocksize-write-disonnect
#   nbdkit vddk server=10.73.212.149 user=root password=+/home/passwd vm=moref=vm-145 file='[esx8.0-matrix] esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmdk' libdir=/home/vddk8.0.0  thumbprint='D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94'  --filter=blocksize-policy  blocksize-minimum=32K blocksize-preferred=64k blocksize-write-disconnect=32k --run 'nbdinfo $uri' 
nbdkit: error: blocksize-write-disonnect must be larger than blocksize-minimum

2.2 Copy the script from  https://gitlab.com/nbdkit/nbdkit/-/blob/master/tests/test-blocksize-write-disconnect.sh, then test blocksize-write-disonnect option
#./test-blocksize-write-disconnect.sh 
....
Testing size=33554432
nbdkit: eval[1]: debug: blocksize-policy: pwrite count=33554432 offset=0 fua=0
nbdkit: eval[1]: error: client pwrite request rejected: count 33554432 is larger than maximum size 16777216
nbdkit: eval[1]: debug: sending error reply: Invalid argument
Testing size=33554433
nbdkit: eval[1]: debug: blocksize-policy: pwrite count=33554433 offset=0 fua=0
nbdkit: eval[1]: error: disconnecting client due to oversize write request
nbdkit: eval[1]: debug: blocksize-policy: finalize
nbdkit: eval[1]: debug: eval: finalize
nbdkit: eval[1]: debug: blocksize-policy: close
nbdkit: eval[1]: debug: eval: close
nbdkit: eval[1]: debug: calling: /tmp/nbdkitQJs9bu/missing close ""
....


3. Test new features of nbdkit curl plugin

3.1 Check new update of man page
3.1.1 The description of -r option is updated
# man nbdkit-curl-plugin |grep 'pass the -r' -A 2
       them).  To force nbdkit to use a readonly connection, pass the -r flag.  Using
       the -r flag also enables NBD multi-conn, which usually performs much better
       (if the client supports it).
       
3.1.2 Clarify documentation for followlocation=false is updated
# man nbdkit-curl-plugin |grep 'followlocation' -A 7
       followlocation=false
           (nbdkit ≥ 1.26)

           Do not follow redirects from the server.  The default is true (follow
           redirects).

           You can follow redirects but avoid redirecting to a less secure protocol
           (eg. HTTPS redirecting to FTP) by using the "protocols" parameter instead.

3.2   Test new option 'connections'
3.2.1 Check option info in man page
# man nbdkit-curl-plugin |grep 'Open up to "N" curl connections to the web server' -B 3 -A 3
       connections=N
           (nbdkit ≥ 1.34)

           Open up to "N" curl connections to the web server.  The default is 4.
           Curl connections are shared between all NBD clients, so you may wish to
           increase this if you expect many simultaneous NBD clients (or a single
           client using many multi-conn connections).

3.2.1 Not set connections option
#time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false  --run 'nbdcopy -p "$uri" null:'
█ 100% [****************************************]

real	3m10.341s
user	0m22.805s
sys	1m1.232s


3.2.2  Set connections=4  
# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=4 --run 'nbdcopy -p "$uri" null:'
█ 100% [****************************************]

real	3m14.616s
user	0m23.052s
sys	1m1.337s

3.2.3 Set connections=5
# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=5 --run 'nbdcopy -p "$uri" null:'
█ 100% [****************************************]

real	3m9.474s
user	0m23.235s
sys	1m1.120s

3.2.4 Set connections=6 
# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=6 --run 'nbdcopy -p "$uri" null:'
█ 100% [****************************************]

real	3m29.656s
user	0m24.313s
sys	1m2.534s

3.2.5 Test the scenario with nbdkit-server-1.32.5-4.el9.x86_64
# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false --run 'nbdcopy -p "$uri" null:'
█ 100% [****************************************]

real	3m5.926s
user	0m22.068s
sys	1m0.403s


Result 3.2 :  Performance doesn't seem to change with higher connection option values, and performance doesn't improve after adding feature "curl handle pool"


3.3 Test new options about http-version

3.3.1 Check new options about http-version in man page
# man nbdkit-curl-plugin |grep 'http-version=none' -A 12
       http-version=none
       http-version=1.0
       http-version=1.1
       http-version=2.0
       http-version=2TLS
       http-version=2-prior-knowledge
       http-version=3
       http-version=3only
           (nbdkit ≥ 1.34)

           Force curl to use a particular HTTP protocol.  The default is "none"
           meaning curl will negotiate the best protocol with the server.  The other
           settings are mainly for testing.  See CURLOPT_HTTP_VERSION(3) for details.

3.3.2 Test http-version=1.0, http-version=1.1, http-version=2.0, http-version=2TLS, http-version=2-prior-knowledge
For example:
# time nbdkit -r curl url='https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false connections=4 http-version=2.0 --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 12884901888 (12G)
	content: DOS/MBR boot sector
.....

real	0m1.358s
user	0m0.080s
sys	0m0.018s

Result: no error is shown

3.3.3 Test http-version=3
# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=3 http-version=3 --run 'nbdcopy -p "$uri" null:'
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda]: URL using bad/illegal format or missing URL: HTTP/3 requested for non-HTTPS URL
nbdcopy: nbd://localhost:10809: nbd_connect_uri: handshake: server has no export named '': No such file or directory

real	0m0.032s
user	0m0.017s
sys	0m0.015s

# time nbdkit -r curl url='https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false connections=4 http-version=3 --run 'nbdinfo $uri'
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix]: Timeout was reached: Operation timed out after 300517 milliseconds with 0 out of 0 bytes received
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix]: Timeout was reached: Operation timed out after 300515 milliseconds with 0 out of 0 bytes received
nbdinfo: nbd_opt_go: server replied with error to opt_go request: No such file or directory for the default export
nbdinfo: suggestion: to list all exports on the server, use --list
protocol: newstyle-fixed without TLS, using structured packets

real	10m1.070s
user	0m0.127s
sys	0m0.025s


Result 3.3.3:  Pop up error when use nbdkit curl to access above http url (scheme: http ) and https url (version: HTTP/2) with http-version=3 , and  nbdkit curl will take a long time to return an error  when connect with https url, not sure if it is a problem

3.3.4 Test http-version=3only
# nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=4 http-version=3only --run 'nbdcopy -p "$uri" null:'
nbdkit: error: unknown http-version: 3only

Result  3.3.4:  option 'http-version= 3only ' can't be recongized by nbdkit curl plugin


3.4  Test new options about ssl-version

3.4.1 Check new options about  ssl-version in man page
# man nbdkit-curl-plugin |grep 'ssl-version=max-default' -A 6
       ssl-version=max-default
       ssl-version=max-tlsv1.0
       ssl-version=max-tlsv1.1
       ssl-version=max-tlsv1.2
       ssl-version=max-tlsv1.3
           Set the SSL ciphers and TLS version.  For further information see
           CURLOPT_SSL_CIPHER_LIST(3) and CURLOPT_SSLVERSION(3).

3.4.2 Test  ssl-version=max-default, ssl-version=max-tlsv1.2 and  ssl-version=max-tlsv1.3
For example:
# nbdkit -r curl url='https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false connections=4 http-version=2.0  ssl-version=max-tlsv1.3 --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 12884901888 (12G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
.....

Result: no error is shown

3.4.3 Test ssl-version=max-tlsv1.0 and  ssl-version=max-tlsv1.1

3.4.3.1 Test ssl-version=max-tlsv1.0 and  ssl-version=max-tlsv1.1 with https url
# nbdkit -r curl url='https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false connections=4 http-version=2.0  ssl-version=max-tlsv1.0 --run 'nbdinfo $uri'
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix]: SSL connect error: error:0A0000BF:SSL routines::no protocols available
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix]: SSL connect error: error:0A0000BF:SSL routines::no protocols available
nbdinfo: nbd_opt_go: server replied with error to opt_go request: No such file or directory for the default export
nbdinfo: suggestion: to list all exports on the server, use --list
protocol: newstyle-fixed without TLS, using structured packets

# nbdkit -r curl url='https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false connections=4 http-version=2.0  ssl-version=max-tlsv1.1 --run 'nbdinfo $uri'
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix]: SSL connect error: error:0A0000BF:SSL routines::no protocols available
nbdkit: curl[1]: error: problem doing HEAD request to fetch size of URL [https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix]: SSL connect error: error:0A0000BF:SSL routines::no protocols available
nbdinfo: nbd_opt_go: server replied with error to opt_go request: No such file or directory for the default export
nbdinfo: suggestion: to list all exports on the server, use --list
protocol: newstyle-fixed without TLS, using structured packets

3.4.3.2 Test ssl-version=max-tlsv1.0 and  ssl-version=max-tlsv1.1 with http url
#  nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=3 ssl-version=max-tlsv1.0  --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
.....

#  nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=3 ssl-version=max-tlsv1.1  --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 17179869184 (16G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
.....

Result 3.4.3:  Can't set ssl-version=max-tlsv1.0 and  ssl-version=max-tlsv1.1 for https url

3.4.4  Set invalid value for ssl-version option
# nbdkit -r curl url='https://10.73.212.149/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false connections=4 http-version=2.0  ssl-version=max-tlsv1.4 --run 'nbdinfo $uri'
nbdkit: error: unknown ssl-version: max-tlsv1.4

Result 3.4.4:  the error is expected


3.5 Check new feature  'NBD CONNECTIONS AND CURL HANDLES'

3.5.1 Check feature info in man page
# man nbdkit-curl-plugin |grep ' nbdkit ≤ 1.32 used a simple model' -B 1 -A 9
NBD CONNECTIONS AND CURL HANDLES
       nbdkit ≤ 1.32 used a simple model where a new NBD connection would create a
       new libcurl handle.  In practice this meant there was a 1-to-1 relationship
       between NBD connections and HTTP connections to the remote web server
       (assuming http: or https: URL).

       nbdkit ≥ 1.34 changed to using a fixed pool of libcurl handles shared across
       all NBD connections.  You can control the maximum number of curl handles in
       the pool with the "connections" parameter (default 4).  Note that if there are
       more than 4 NBD connections, they will share the 4 web server connections,
       unless you adjust "connections".


3.5.2 # nbdkit -f -v -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=4 -D curl.pool=1 --run 'nbdinfo $uri'
.....

Result 3.5.2: there is no info about pool in debug log



Hi Richard,

  Please help to check if below results are expected, thanks!

Result 1.1.1 :  The help info of  both nbdkit and file plugin will be shown
Result 1.1.2 :  Only help info of nbdkit is shown
Result 1.2 and 1.3: no problems found except not sure if  the result 1.3.5 is expected
Result 1.2.4: not sure if error 'pread: Bad file descriptor' is expected
Result 3.2 :  Performance doesn't seem to change with higher connection option values, and performance doesn't improve after adding feature "curl handle pool"
Result 3.3.3:  Pop up error when use nbdkit curl to access above http url (scheme: http ) and https url (version: HTTP/2) with http-version=3 , and  nbdkit curl will take a long time to return an error  when connect with https url, not sure if it is a problem
Result  3.3.4:  option 'http-version= 3only ' can't be recongized by nbdkit curl plugin
Result 3.4.3:  Can't set ssl-version=max-tlsv1.0 and  ssl-version=max-tlsv1.1 for https url
Result 3.5.2:  there is no info about pool in debug log
Comment 7 Richard W.M. Jones 2023-04-27 09:58:05 UTC 
> Result 1.1.1 :  The help info of  both nbdkit and file plugin will be shown

This is how it has always worked.  'nbdkit --help' shows the general help page,
and 'nbdkit plugin --help' shows help for nbdkit and the plugin.   Been like that
since version 0.1.

> Result 1.1.2 :  Only help info of nbdkit is shown

The problem is that '-h' is not a known option, use --help.

> 1.2.4 Write info to the fd of step1.2.3
> # echo hello >&5
> # nbdkit file fd=5  --run 'nbdinfo $uri' 
> nbdkit: file.1: error: pread: Bad file descriptor

I think the problem here is with the test.  The first echo .. >&5 writes
something to fd 5 which is not open.  This is an error and you should see:

$ echo hello >&5
bash: 5: Bad file descriptor

fd 5 is still closed / invalid so nbdkit correctly prints an error about
the Bad file descriptor.

Did you type the exec command first to open the fd, eg:

$ exec 5> /tmp/hello
$ echo hello >&5
$ nbdkit file fd=5 --run 'nbdinfo --size $uri'
6
Comment 8 Richard W.M. Jones 2023-04-27 09:59:38 UTC 
> 1.3.5 Create a empty directory and add file descriptor to the directory
> # exec 7< /root/empty                                                           
> # nbdkit file dirfd=7 --run 'nbdinfo --list $uri'                               
> protocol: newstyle-fixed without TLS, using structured packets
>
> Result 1.2 and 1.3: no problems found except not sure if the result 1.3.5 is
> expected

Yes, this is expected.  It's just the "null case" where there are no files
so no exports.
Comment 9 Richard W.M. Jones 2023-04-27 10:03:01 UTC 
> 2.2 Copy the script from
> https://gitlab.com/nbdkit/nbdkit/-/blob/master/tests/test-blocksize-write-disconnect.sh,
> then test blocksize-write-disonnect option
> #./test-blocksize-write-disconnect.sh                                           

The tests are run as part of the package build, so it may be better just to check
that the tests were run by looking at build.log:

https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=66264
=> latest build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2469742
=> build on x86-64: https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/nbdkit/1.34.1/1.el9/data/logs/x86_64/build.log

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.qs8Ndd
...
PASS: test-blocksize-write-disconnect.sh
...
============================================================================
Testsuite summary for nbdkit 1.34.1
============================================================================
# TOTAL: 291
# PASS:  249
# SKIP:  42
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

(Unfortunately we don't collect build artifacts so we are unable to see why
some tests were skipped.)
Comment 10 Richard W.M. Jones 2023-04-27 10:10:00 UTC 
> Result 3.2 :  Performance doesn't seem to change with higher connection option
> values, and performance doesn't improve after adding feature "curl handle pool"

There's a few things here.  Firstly connections=4 is the default.  You could try
smaller settings.  connections=1 may make things worse.

But the main problem is that nbdcopy uses 4 threads by default, so you have to
try changing both nbdkit curl connections=X and nbdcopy -C X, eg:

  $ time nbdkit -r curl URL connections=16 --run 'nbdcopy -p -C 16 "$uri" null:'

(You can also try increasing one and decreasing the other, they don't
have to be the same.)

However I don't expect this to make a massive difference.  It is very
sensitive to the network and web server.  The web server may also limit
the number of connections you can make.
Comment 11 Richard W.M. Jones 2023-04-27 10:14:21 UTC 
> 3.3.3 Test http-version=3
> URL using bad/illegal format or missing URL: HTTP/3 requested for non-HTTPS URL

So QUIC (HTTP/3) only supports unencrypted connections, you have to use https://...

However I doubt the server supports QUIC at all.  Support in web servers is pretty
rare outside Google's own services.

Also QUIC runs over UDP (not TCP) so you may have to open UDP ports on the
server or client.

In all, I would be surprised if it is possible to test this feature ourselves.
Comment 12 Richard W.M. Jones 2023-04-27 10:16:49 UTC 
> Result  3.3.4:  option 'http-version= 3only ' can't be recongized by nbdkit
> curl plugin

This requires curl >= 7.88.0 which is not available in RHEL 9, so this
feature is disabled in our build.  Refer to:

https://curl.se/libcurl/c/CURLOPT_HTTP_VERSION.html
https://gitlab.com/nbdkit/nbdkit/-/blob/e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/curl.c#L307
Comment 13 Richard W.M. Jones 2023-04-27 10:19:50 UTC 
> 3.4.3.1 Test ssl-version=max-tlsv1.0 and  ssl-version=max-tlsv1.1 with https
> SSL connect error: error:0A0000BF:SSL routines::no protocols available   

TLS v1.0 and v1.1 are disabled in RHEL openssl, so won't work.
Comment 14 Richard W.M. Jones 2023-04-27 10:21:39 UTC 
> Result 3.5.2: there is no info about pool in debug log

You should see these messages:

https://gitlab.com/nbdkit/nbdkit/-/blob/e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/pool.c#L103
https://gitlab.com/nbdkit/nbdkit/-/blob/e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/pool.c#L130
https://gitlab.com/nbdkit/nbdkit/-/blob/e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/pool.c#L171

However -D curl.pool=1 is an undocumented debugging option for developers
so it's not really necessary to test it.
Comment 15 mxie@redhat.com 2023-04-27 11:41:12 UTC 
(In reply to Richard W.M. Jones from comment #14)
> > Result 3.5.2: there is no info about pool in debug log
> 
> You should see these messages:
> 
> https://gitlab.com/nbdkit/nbdkit/-/blob/
> e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/pool.c#L103
> https://gitlab.com/nbdkit/nbdkit/-/blob/
> e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/pool.c#L130
> https://gitlab.com/nbdkit/nbdkit/-/blob/
> e397f06433cc4715b0e89c6a81dd9014a5db8ca9/plugins/curl/pool.c#L171
> 
> However -D curl.pool=1 is an undocumented debugging option for developers
> so it's not really necessary to test it.

Thanks, I found below info in nbdkit curl debug log

nbdkit: curl[1]: debug: curl: open returned handle 0x7f0ee80097a0
nbdkit: curl[1]: debug: curl: prepare readonly=1
nbdkit: curl[1]: debug: curl: get_size
nbdkit: curl[1]: debug: get_handle: 0
nbdkit: curl[1]: debug: put_handle: 0
Comment 16 mxie@redhat.com 2023-04-27 12:06:39 UTC 
(In reply to Richard W.M. Jones from comment #10)
> > Result 3.2 :  Performance doesn't seem to change with higher connection option
> > values, and performance doesn't improve after adding feature "curl handle pool"
> 
> There's a few things here.  Firstly connections=4 is the default.  You could
> try
> smaller settings.  connections=1 may make things worse.
> 
> But the main problem is that nbdcopy uses 4 threads by default, so you have
> to
> try changing both nbdkit curl connections=X and nbdcopy -C X, eg:
> 
>   $ time nbdkit -r curl URL connections=16 --run 'nbdcopy -p -C 16 "$uri"
> null:'
> 
> (You can also try increasing one and decreasing the other, they don't
> have to be the same.)
> 
> However I don't expect this to make a massive difference.  It is very
> sensitive to the network and web server.  The web server may also limit
> the number of connections you can make.

Thanks, performance improves with higher connection option values

# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=1 --run 'nbdcopy -p -C 1 "$uri" null:'
█ 100% [****************************************]-----------------------------------]

real	7m9.986s
user	0m19.642s
sys	0m55.126s


# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=4 --run 'nbdcopy -p -C 4 "$uri" null:'
█ 100% [****************************************]

real	3m40.575s
user	0m24.561s
sys	1m4.010s

# time nbdkit -r curl url='http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section3/libvirtmanual/mxie/esx8.0-rhel8.8-x86_64-sda' sslverify=false connections=10 --run 'nbdcopy -p -C 10 "$uri" null:'
█ 100% [****************************************]

real	3m27.367s
user	0m33.422s
sys	1m16.007s
Comment 17 mxie@redhat.com 2023-04-27 12:13:04 UTC 
(In reply to Richard W.M. Jones from comment #7)
> > 1.2.4 Write info to the fd of step1.2.3
> > # echo hello >&5
> > # nbdkit file fd=5  --run 'nbdinfo $uri' 
> > nbdkit: file.1: error: pread: Bad file descriptor
> 
> I think the problem here is with the test.  The first echo .. >&5 writes
> something to fd 5 which is not open.  This is an error and you should see:
> 
> $ echo hello >&5
> bash: 5: Bad file descriptor
> 
> fd 5 is still closed / invalid so nbdkit correctly prints an error about
> the Bad file descriptor.
> 
> Did you type the exec command first to open the fd, eg:
> 
> $ exec 5> /tmp/hello
> $ echo hello >&5
> $ nbdkit file fd=5 --run 'nbdinfo --size $uri'
> 6

Yes, I did, the error will show when run 'nbdinfo $uri'

# exec 5> /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2 
# echo hello >&5# nbdkit file fd=5 --run 'nbdinfo --size $uri'
6
# nbdkit file fd=5 --run 'nbdinfo $uri'
nbdkit: file.1: error: pread: Bad file descriptor
protocol: newstyle-fixed without TLS, using structured packets
export="":
        export-size: 6
        uri: nbd://localhost:10809/
        contexts:
                base:allocation
        is_rotational: false
        is_read_only: false
        can_cache: true
        can_df: true
        can_fast_zero: false
        can_flush: true
        can_fua: true
        can_multi_conn: true
        can_trim: true
	can_zero: true
Comment 18 Richard W.M. Jones 2023-04-27 14:09:31 UTC 
(In reply to mxie from comment #17)
> # exec 5> /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2 
> # echo hello >&5

This command probably overwrites /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2
which maybe not what you intended?

> # nbdkit file fd=5 --run 'nbdinfo --size $uri'
> 6
> # nbdkit file fd=5 --run 'nbdinfo $uri'
> nbdkit: file.1: error: pread: Bad file descriptor
> protocol: newstyle-fixed without TLS, using structured packets
> export="":
>         export-size: 6

I think what is happening is that the original 'exec 5> ...' command
opens the file descriptor for write only.  So we can get the size (nbdinfo --size)
but as soon as we try to read from it, it will fail.

This works instead:

$ exec 5<> /tmp/hello
$ echo hello >&5
$ nbdkit file fd=5 --run 'nbdinfo --size $uri'
6
$ nbdkit file fd=5 --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 6
	content: ASCII text

Note:
(1) Using 5<> to open for read/write
(2) The content is "ASCII text" which is correct because the file contains "hello\n"
Comment 19 mxie@redhat.com 2023-04-27 14:38:32 UTC 
(In reply to Richard W.M. Jones from comment #18)
> (In reply to mxie from comment #17)
> > # exec 5> /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2 
> > # echo hello >&5
> 
> This command probably overwrites
> /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2
> which maybe not what you intended?
> 
> > # nbdkit file fd=5 --run 'nbdinfo --size $uri'
> > 6
> > # nbdkit file fd=5 --run 'nbdinfo $uri'
> > nbdkit: file.1: error: pread: Bad file descriptor
> > protocol: newstyle-fixed without TLS, using structured packets
> > export="":
> >         export-size: 6
> 
> I think what is happening is that the original 'exec 5> ...' command
> opens the file descriptor for write only.  So we can get the size (nbdinfo
> --size)
> but as soon as we try to read from it, it will fail.
> 
> This works instead:
> 
> $ exec 5<> /tmp/hello
> $ echo hello >&5
> $ nbdkit file fd=5 --run 'nbdinfo --size $uri'
> 6
> $ nbdkit file fd=5 --run 'nbdinfo $uri'
> protocol: newstyle-fixed without TLS, using structured packets
> export="":
> 	export-size: 6
> 	content: ASCII text
> 
> Note:
> (1) Using 5<> to open for read/write
> (2) The content is "ASCII text" which is correct because the file contains
> "hello\n"

Thanks, you're right, the error disappeared after opening file descriptor for read and write

# exec 6<> /home/RHEL-9.3.0-20230417.0-x86_64-ovmf.qcow2 
# echo hello >&6
# nbdkit file fd=6 --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 6
	content: ASCII text
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true
Comment 23 mxie@redhat.com 2023-07-07 10:48:46 UTC 
Verify the bug with nbdkit-server-1.34.1-1.el9.x86_64

Steps:
1.Build nbdkit packages from src rpm package with regular user
$ rpmbuild --rebuild nbdkit-1.34.1-1.el9.src.rpm 
...
PASS: test-blocksize-write-disconnect.sh
...
+ exit 0

2. Check that nbdkit --dump-config now reports exit_with_parent=yes
#  nbdkit --dump-config
binary=/usr/sbin/nbdkit
bindir=/usr/bin
exit_with_parent=yes
......

3. Check that some debug information of the nbdkit command is now displayed in bold, but the bold information is not clear  when the background of the terminal box is black, please refer to the screenshot 'bold-debug-info-of-nbdkit.png'

# nbdkit -vr curl url='https://10.73.212.129/folder/esx8.0-rhel9.3-x86_64_1/esx8.0-rhel9.3-x86_64-flat.vmdk?dcPath=data&dsName=esx8.0-matrix' password=+/home/passwd user=root sslverify=false  --run 'nbdinfo $uri'
....
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 12884901888 (12G)
	content: DOS/MBR boot sector
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: true
	can_cache: false
	can_df: true
	can_fast_zero: false
	can_flush: false
	can_fua: false
	can_multi_conn: true
	can_trim: false
	can_zero: false

Hi Richard,
   Please help to check step3, is it a problem?
Comment 25 Richard W.M. Jones 2023-07-07 11:40:56 UTC 
Dark grey background does make that difficult to read ...
Comment 26 Richard W.M. Jones 2023-07-07 11:43:10 UTC 
(In reply to Richard W.M. Jones from comment #25)
> Dark grey background does make that difficult to read ...

So I think it's worth filing this as a separate bug for now, so we can
look at it later.  I'm not sure I'll have time to fix this before
RHEL 9.3, but it's not really a blocker (in fact, I think it may even
exist already in older versions).
Comment 27 mxie@redhat.com 2023-07-13 05:51:16 UTC 
The problem found already has been tracked by bug, move the bug from ON_QA to VERIFIED
Comment 29 errata-xmlrpc 2023-11-07 08:28:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nbdkit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6374