Bug 2011708

Summary:	Rebase to libnbd 1.10 before RHEL 9 GA
Product:	Red Hat Enterprise Linux 9	Reporter:	Richard W.M. Jones <rjones>
Component:	libnbd	Assignee:	Richard W.M. Jones <rjones>
Status:	CLOSED ERRATA	QA Contact:	Vera <vwu>
Severity:	high	Docs Contact:
Priority:	medium
Version:	9.0	CC:	chhu, juzhou, kkiwi, lersek, mxie, rjones, tyan, tzheng, virt-maint, xiaodwan
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libnbd-1.10.5-1.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-17 12:51:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Richard W.M. Jones 2021-10-07 07:33:50 UTC

Description of problem:

Since RHEL 9 Beta was released, libnbd has released a new
stable branch (1.10).  Because of the number of backports
in RHEL 9 Beta, updating to the new stable branch for GA
is not actually a very large change.

Here are the release notes for the new stable branch:
https://libguestfs.org/libnbd-release-notes-1.10.1.html

This update is optional, we don't need to do it if it would
cause too much extra work for QE.

Version-Release number of selected component (if applicable):

libnbd-1.8.2-3.el9 -> libnbd 1.10.x

Comment 4 Richard W.M. Jones 2021-10-30 15:16:29 UTC

Release notes for 1.10:
https://libguestfs.org/libnbd-release-notes-1.10.1.html

For bugs fixed along the stable-1.10 branch:
https://gitlab.com/nbdkit/libnbd/-/commits/stable-1.10/

Comment 5 Richard W.M. Jones 2021-12-02 21:12:12 UTC

This contains some fixes for obscure memory leaks in the Python bindings, see:
https://gitlab.com/nbdkit/libnbd/-/commits/stable-1.10/

Comment 6 Vera 2022-01-28 03:09:29 UTC

Verified with the versions:
libnbd-1.10.3-1.el9.x86_64
nbdkit-1.28.4-2.el9.x86_64
nbdfuse-1.10.3-1.el9.x86_64
python3-libnbd-1.10.3-1.el9.x86_64

Steps:
Scenario1:nbdinfo
1.1 IPv6 numeric URIs are supported.
# nbdinfo 'nbd://[2620:52:0:4254:2d8:61ff:fe3d:da54]:10809'
protocol: newstyle-fixed without TLS
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://[2620:52:0:4254:2d8:61ff:fe3d:da54]:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

1.2 <nbdinfo(1)> <--list> (which lists all exports of a server) no longer exists early if one of the exports is inaccessible or there is a recoverable error
# nbdkit -U - file dir=/tmp/test --run 'nbdinfo --list "$uri"'
protocol: newstyle-fixed without TLS
export="1":
	export-size: 0
	uri: nbd+unix:///1?socket=/tmp/nbdkitNujo64/socket
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true
export="好":
	export-size: 0
	uri: nbd+unix:///%E5%A5%BD?socket=/tmp/nbdkitNujo64/socket
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: false
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

1.3 Use nbdinfo to display information and metadata about an NBD server.
# nbdinfo nbd://localhost
protocol: newstyle-fixed without TLS
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

# nbdinfo --json nbd://localhost | jq
{
  "protocol": "newstyle-fixed",
  "TLS": false,
  "exports": [
    {
      "export-name": "",
      "content": "data",
      "uri": "nbd://localhost:10809/",
      "contexts": [
        "base:allocation"
      ],
      "is_rotational": false,
      "is_read_only": false,
      "can_cache": true,
      "can_df": true,
      "can_fast_zero": true,
      "can_flush": true,
      "can_fua": true,
      "can_multi_conn": true,
      "can_trim": true,
      "can_zero": true,
      "export-size": 10737418240,
      "export-size-str": "10G"
    }
  ]
}


Scenario2: nbdcopy
2.1 nbdcopy new <--request-size> option to allow controlling the size of requests made to NBD servers.  Adjusting this can affect performance

# nbdcopy --request-size=1048576 -p fedora-30.qcow2 nbd://localhost
█ 100% [****************************************]
# nbdinfo nbd://localhost
protocol: newstyle-fixed without TLS
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 6442450944 bytes
	uri: nbd://localhost:10809/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

2.2 nbdcopy new I<-v> flag to enable libnbd and other debugging.
# nbdcopy -p nbd://localhost test.img -v |& tee >  nbdcopy.log
█ 100% [****************************************]
# grep test nbdcopy.log 
nbdcopy: dst: file_ops "test.img"

2.3  Copy between a local file and an NBD server (in both directions)

# nbdcopy --request-size=1048576 -p fedora-30.qcow2 nbd://localhost:10810
█ 100% [****************************************]
# nbdcopy nbd://localhost:10810 verify.img -p
█ 100% [****************************************]
# ls -al |grep verify
-rw-r--r--.  1 root root 10737418240 Jan 27 05:53 verify.img

2.4 Use nbdcopy to copy the file between multiple NBD servers


# nbdinfo nbd://localhost:10811
protocol: newstyle-fixed without TLS
export="":
	export-size: 10737418240 (10G)
	content: data
	uri: nbd://localhost:10811/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true
# nbdinfo nbd://localhost:10810
protocol: newstyle-fixed without TLS
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 6442450944 bytes
	uri: nbd://localhost:10810/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true

# nbdcopy -p nbd://localhost:10810 nbd://localhost:10811
█ 100% [****************************************]
# nbdinfo nbd://localhost:10811
protocol: newstyle-fixed without TLS
export="":
	export-size: 10737418240 (10G)
	content: QEMU QCOW2 Image (v3), 6442450944 bytes
	uri: nbd://localhost:10811/
	contexts:
		base:allocation
	is_rotational: false
	is_read_only: false
	can_cache: true
	can_df: true
	can_fast_zero: true
	can_flush: true
	can_fua: true
	can_multi_conn: true
	can_trim: true
	can_zero: true


2.5 nbdfuse now supports nbdcopy-style C<[ CMD ]>, eg: nbdfuse dir/ramdisk [ nbdkit memory 1G ]

# nbdfuse /dir/ramdsk [ nbdkit memory 1G ] &
[1] 120666
# ls -lh /dir
total 0
-rw-rw-rw-. 1 root root 1.0G Jan 27 06:02 ramdsk


Scenario3: nbdfuse

3.1 nbdfuse new I<-v> flag to enable libnbd and other debugging

# nbdfuse nbdfuse-v/ramdisk --command nbdkit -s memory 2G -v |& tee > /nbdfuse-v.log
# ls -lh /dir
total 0
-rw-rw-rw-. 1 root root 1.0G Jan 27 06:02 ramdsk

3.2 Turn a qcow2 file into raw file
# nbdfuse /var/tmp/mp/fedora-30.raw [ qemu-nbd -f qcow2 fedora-30.qcow2 ] &
[1] 120803
# ll /var/tmp/mp
total 0
-rw-rw-rw-. 1 root root 6442450944 Jan 27 06:06 fedora-30.raw



Scenario4:nbdsh
4.1 Use nbdsh to test simple nbd API calls
# nbdsh

Welcome to nbdsh, the shell for interacting with
Network Block Device (NBD) servers.

The ‘nbd’ module has already been imported and there
is an open NBD handle called ‘h’.

h.connect_tcp("remote", "10809")   # Connect to a remote server.
h.get_size()                       # Get size of the remote disk.
buf = h.pread(512, 0, 0)           # Read the first sector.
exit() or Ctrl-D                   # Quit the shell
help(nbd)                          # Display documentation

nbd> help(nbd)

nbd> h.connect_command (["nbdkit", "-s", "memory", "size=2G"])
nbd> h.get_size()
2147483648
nbd> buf = b"hello"
nbd> h.pwrite (buf, 0, 0)
nbd> buf2 = h.pread (100, 0)
nbd> print (buf2)
b'hello\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
nbd> buf=nbd.Buffer.from_bytearray(bytearray(b"abc"))
nbd> print(buf.is_zero(offset=0))
False
nbd> buf=nbd.Buffer.from_bytearray(bytearray(b"\0a\02\0"))
nbd> print(buf.is_zero(offset=4))
True
nbd> print(buf.is_zero(offset=0))
False
nbd> print(buf.is_zero(offset=1))
False
nbd> buf=nbd.Buffer.from_bytearray(bytearray(b""))
nbd> print(buf.is_zero(offset=0))
True
nbd> print(buf.is_zero(offset=1))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib64/python3.9/site-packages/nbd.py", line 157, in is_zero
    return libnbdmod.aio_buffer_is_zero(self._o, offset, size)
IndexError: offset out of range
nbd> 


Scenario 5: rebuilding src rpm
Steps:
1. Download src package ;
-rw-r--r--.  1 root root 1260697 Jan 17 12:02 libnbd-1.10.3-1.el9.src.rpm

2.  Resolve the dependence problem before rebuilding,then build libnbd rpm packages from src package with non-root user;

# rpmbuild --rebuild libnbd-1.10.3-1.el9.src.rpm 
Installing libnbd-1.10.3-1.el9.src.rpm
warning: user mockbuild does not exist - using root
warning: group mock does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mock does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mock does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mock does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mock does not exist - using root
setting SOURCE_DATE_EPOCH=1642377600
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.uC4G7j
+ umask 022
+ cd /root/rpmbuild/BUILD
+ /usr/lib/rpm/redhat/gpgverify --keyring=/root/rpmbuild/SOURCES/libguestfs.keyring --signature=/root/rpmbuild/SOURCES/libnbd-1.10.3.tar.gz.sig --data=/root/rpmbuild/SOURCES/libnbd-1.10.3.tar.gz
gpgv: Signature made Mon Jan 17 11:03:28 2022 EST
gpgv:                using RSA key F7774FB1AD074A7E8C8767EA91738F73E1B768A0
gpgv:                issuer "rich"
gpgv: Good signature from "Richard W.M. Jones <rjones>"
gpgv:                 aka "Richard W.M. Jones <rich>"
+ cd /root/rpmbuild/BUILD
+ rm -rf libnbd-1.10.3
...........
..........
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.JdqXH0
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd libnbd-1.10.3
+ /usr/bin/rm -rf /root/rpmbuild/BUILDROOT/libnbd-1.10.3-1.el9.x86_64
+ RPM_EC=0
++ jobs -p
+ exit 0
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.1dZZea
+ umask 022
+ cd /root/rpmbuild/BUILD
+ rm -rf libnbd-1.10.3
+ RPM_EC=0
++ jobs -p
+ exit 0


3. check the rpms under the ~/RPMS/x86_64 and compare with the rpms on the brewweb.

# ls -al
total 1404
drwxr-xr-x. 2 root root   4096 Jan 27 21:17 .
drwxr-xr-x. 4 root root     34 Jan 27 21:17 ..
-rw-r--r--. 1 root root 136891 Jan 27 21:17 libnbd-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 303974 Jan 27 21:17 libnbd-debuginfo-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 125386 Jan 27 21:17 libnbd-debugsource-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 381379 Jan 27 21:17 libnbd-devel-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  32495 Jan 27 21:17 nbdfuse-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  37669 Jan 27 21:17 nbdfuse-debuginfo-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  58513 Jan 27 21:17 ocaml-libnbd-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  68236 Jan 27 21:17 ocaml-libnbd-debuginfo-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  95514 Jan 27 21:17 ocaml-libnbd-devel-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  75775 Jan 27 21:17 python3-libnbd-1.10.3-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  92370 Jan 27 21:17 python3-libnbd-debuginfo-1.10.3-1.el9.x86_64.rpm


Moving to verified.

Comment 8 errata-xmlrpc 2022-05-17 12:51:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libnbd), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2409