Bug 2168628

Summary: [RHEL 9.3] Rebase libnbd to 1.16
Product: Red Hat Enterprise Linux 9 Reporter: Richard W.M. Jones <rjones>
Component: libnbdAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Vera <vwu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: lersek, mxie, rjones, tyan, tzheng, virt-maint, vwu, xiaodwan
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libnbd-1.16.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:30:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Richard W.M. Jones 2023-02-09 15:23:19 UTC 
Description of problem:

For RHEL 9.2 the original plan was to rebase to libnbd 1.16
(https://bugzilla.redhat.com/show_bug.cgi?id=2135764).  We
didn't actually manage to do that.  However it is likely there
will be a new upstream stable release of libnbd quite soon
and we should consider rebasing to this.

There are no release notes yet, but they will be prepared
as part of the upstream release.

Version-Release number of selected component (if applicable):

libnbd 1.16.x
Comment 1 Richard W.M. Jones 2023-02-09 16:37:55 UTC 
In RHEL 9.2 we do not carry any downstream patches.
Comment 2 Richard W.M. Jones 2023-04-13 13:03:45 UTC 
Preliminary release notes for 1.16 are available here:
https://gitlab.com/nbdkit/libnbd/-/blob/master/docs/libnbd-release-notes-1.16.pod
Comment 3 Richard W.M. Jones 2023-04-18 15:11:43 UTC 
Final release notes are now available:
https://libguestfs.org/libnbd-release-notes-1.16.1.html
Comment 5 Vera 2023-05-06 07:30:28 UTC 
Tested the new features of 1.16 with the versions:
libnbd-1.16.0-1.el9.x86_64
qemu-kvm-8.0.0-1.el9.x86_64
libvirt-9.2.0-1.el9.x86_64
nbdkit-1.34.1-1.el9.x86_64

Steps:
S1:New APIs
S1.1. New API nbd_supports_vsock(3) can be used to test if the client has vsock support 

# rpm -q libnbd
libnbd-1.16.0-1.el9.x86_64
# nbdsh

Welcome to nbdsh, the shell for interacting with
Network Block Device (NBD) servers.

The ‘nbd’ module has already been imported and there
is an open NBD handle called ‘h’.

h.get_size()                       # Get size of the remote disk.
buf = h.pread(512, 0)              # Read the first sector.
exit() or Ctrl-D                   # Quit the shell
help(nbd)                          # Display documentation

nbd> h.get_version()
'1.16.0'
nbd> h.supports_vsock()
True
nbd> 

# nbdkit --vsock --port 18022 pattern size=512 --run 'nbdinfo "$uri"'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 512
	content: data
	uri: nbd+vsock://1:18022/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: true
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: false
	can_fua: false
	can_multi_conn: true
	can_trim: false
	can_zero: false

S1.2 New nbd_stats_bytes_sent(3), nbd_stats_bytes_received(3), nbd_stats_chunks_sent(3) and nbd_stats_chunks_received(3) APIs

nbd> h.stats_bytes_sent()
0
nbd> 
nbd> h.stats_bytes_received()
0
nbd> h.stats_chunks_sent()
0
nbd> h.stats_chunks_received()
0
nbd> h.get_size()
5368709120
nbd> h.stats_bytes_sent()
44
nbd> h.stats_bytes_received()
90
nbd> h.stats_chunks_sent()
2
nbd> h.stats_chunks_received()
1
nbd> 
nbd> h.shutdown()
72
nbd> h.stats_bytes_received()
90
nbd> h.stats_chunks_sent()
3
nbd> h.stats_chunks_received()
1
nbd> 

S2:nbdinfo 
S2.1 nbdinfo has colorized output (use --no-color to disable)
# nbdinfo nbd://localhost
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true
# nbdinfo nbd://localhost --no-color
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

S2.2 nbdinfo has new --is tls and --can structured-reply options

# nbdinfo 'nbds://10.66.146.143:10812?tls-certificates=/var/tmp/pki'
protocol: newstyle-fixed with TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: DOS/MBR boot sector
	uri: nbds://10.66.146.143:10812/?tls-certificates=/var/tmp/pki
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: false
	can_trim: true
	can_zero: true
	block_size_minimum: 1
	block_size_preferred: 4096
	block_size_maximum: 33554432
# nbdinfo --can structured-reply 'nbds://10.66.146.143:10812?tls-certificates=/var/tmp/pki'
# echo $?
0
# 
# nbdinfo --is tls 'nbds://10.66.146.143:10812?tls-certificates=/var/tmp/pki'
# echo $?
0

S3 nbdsh(1) options -c and -u can now be interleaved

# nbdsh -c 'print("before connecting ...")' -u nbd://localhost
before connecting ...
# 
# nbdsh -c 'print("before connecting ...")' -u nbd://localhost:18022
before connecting ...
nbdsh: command line script failed: nbd_connect_uri: connect: localhost:18022: could not connect to remote host: Connection refused

S4 nbdcopy
S4.1
# nbdcopy -p RHEL9.2.qcow2 nbd://localhost
█ 100% [****************************************]
# nbdinfo nbd://localhost
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 10737418240 bytes
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

S4.2 Use nbdcopy to copy the file between multiple NBD servers
# nbdinfo nbd://localhost:10815
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10815/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

# nbdcopy -p nbd://localhost nbd://localhost:10815
█ 100% [****************************************]
# nbdinfo nbd://localhost:10815
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 10737418240 bytes
	uri: nbd://localhost:10815/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true


Marking as Verified:Tested.
Comment 8 Vera 2023-08-17 03:50:05 UTC 
Verified the new features of 1.16 with the versions:
libnbd-1.16.0-1.el9.x86_64
qemu-kvm-8.0.0-11.el9.x86_64
libvirt-9.5.0-5.el9.x86_64
nbdkit-1.34.2-1.el9.x86_64

Steps:
S1:New APIs
S1.1. New API nbd_supports_vsock(3) can be used to test if the client has vsock support 

# rpm -q libnbd
libnbd-1.16.0-1.el9.x86_64
# nbdsh

Welcome to nbdsh, the shell for interacting with
Network Block Device (NBD) servers.

The ‘nbd’ module has already been imported and there
is an open NBD handle called ‘h’.

h.get_size()                       # Get size of the remote disk.
buf = h.pread(512, 0)              # Read the first sector.
exit() or Ctrl-D                   # Quit the shell
help(nbd)                          # Display documentation

nbd> h.get_version()
'1.16.0'
nbd> 
nbd> h.supports_vsock()
True
nbd> 

# nbdkit --vsock --port 18022 pattern size=512 --run 'nbdinfo "$uri"'
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 512
	content: data
	uri: nbd+vsock://1:18022/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: true
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: false
	can_fua: false
	can_multi_conn: true
	can_trim: false
	can_zero: false

S1.2 New nbd_stats_bytes_sent(3), nbd_stats_bytes_received(3), nbd_stats_chunks_sent(3) and nbd_stats_chunks_received(3) APIs

nbd> h.stats_bytes_sent()
44
nbd> h.stats_bytes_received()
90
nbd> h.stats_chunks_sent()
2
nbd> h.stats_chunks_received()
1
nbd> h.get_size()
10737418240
nbd> h.stats_bytes_sent()
44
nbd> h.stats_bytes_received()
90
nbd> h.stats_chunks_sent()
2
nbd> h.stats_chunks_received()
1
nbd> 
nbd> h.shutdown()
nbd> h.stats_bytes_received()
90
nbd> h.stats_chunks_sent()
3
nbd> h.stats_chunks_received()
1
nbd> 
nbd> 

S2:nbdinfo 
S2.1 nbdinfo has colorized output (use --no-color to disable)
# nbdinfo nbd://localhost
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true
# nbdinfo nbd://localhost --no-color
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true


S2.2 nbdinfo has new --is tls and --can structured-reply options

# nbdinfo 'nbds://10.66.146.143?tls-certificates=/var/tmp/pki'
protocol: newstyle-fixed with TLS, using structured packets
export="":
	export-size: 6442450944 (6G)
	content: DOS/MBR boot sector
	uri: nbds://10.66.146.143:10809/?tls-certificates=/var/tmp/pki
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: false
	can_trim: true
	can_zero: true
	block_size_minimum: 1
	block_size_preferred: 4096
	block_size_maximum: 33554432
# nbdinfo --can structured-reply 'nbds://10.66.146.143?tls-certificates=/var/tmp/pki'
# echo $?
0
# nbdinfo --is tls 'nbds://10.66.146.143?tls-certificates=/var/tmp/pki'
# echo $?
0


S3 nbdsh(1) options -c and -u can now be interleaved

# nbdsh -c 'print("before connecting ...")' -u nbd://localhost
before connecting ...
# nbdsh -c 'print("before connecting ...")' -u nbd://localhost:18022
before connecting ...
nbdsh: command line script failed: nbd_connect_uri: connect: localhost:18022: could not connect to remote host: Connection refused


S4 nbdcopy
S4.1
#  nbdcopy -p RHEL9.2.qcow2 nbd://localhost
█ 100% [****************************************]
#  nbdinfo nbd://localhost
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 10737418240 bytes
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

S4.2 Use nbdcopy to copy the file between multiple NBD servers
# nbdinfo nbd://localhost:10810
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10810/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

# nbdcopy -p nbd://localhost nbd://localhost:10810
█ 100% [****************************************]
# nbdinfo nbd://localhost:10810
protocol: newstyle-fixed without TLS, using structured packets
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 10737418240 bytes
	uri: nbd://localhost:10810/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

Marking as Verified.
Comment 10 errata-xmlrpc 2023-11-07 08:30:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libnbd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6394