Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2228131

Summary: Update nbdkit curl plugin to use "multi" interface, which enhances performance
Product: Red Hat Enterprise Linux 9 Reporter: Richard W.M. Jones <rjones>
Component: nbdkitAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: mxie <mxie>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: eblake, lersek, mxie, rjones, tyan, tzheng, virt-maint, vwu, xiaodwan, ymankad
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nbdkit-1.34.2-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:28:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Richard W.M. Jones 2023-08-01 13:36:34 UTC 
Description of problem:

Upstream bug: https://github.com/kubevirt/containerized-data-importer/issues/2809

I rewrote nbdkit-curl-plugin to use the curl "multi" interface:

https://gitlab.com/nbdkit/nbdkit/-/commit/a74b289ee15a7c75dceb8a96403f1fa8ebd72e88

In some situations this causes considerable performance improvements,
see the results in the commit message above and here:

https://github.com/kubevirt/containerized-data-importer/issues/2809#issuecomment-1660169454

We should backport this work to RHEL 9.3 so that Kubevirt can
consume it.

Version-Release number of selected component (if applicable):

nbdkit-1.34.1-1.el9.x86_64

How reproducible:

100%

Steps to Reproduce:

Example commands which run slowly (best to try these in a Beaker
instance if possible):

$ time nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img --run 'nbdcopy -p $uri null:'

$ time ./nbdkit -r -U - curl https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 --run 'nbdcopy -p $uri null:'

You can also compare the timings against using simple "wget" on the URL.
Comment 1 Richard W.M. Jones 2023-08-01 14:19:41 UTC 
Scratch build:
https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=2601711
Comment 2 Richard W.M. Jones 2023-08-01 14:50:28 UTC 
https://gitlab.com/redhat/centos-stream/rpms/nbdkit/-/merge_requests/47
Comment 3 Richard W.M. Jones 2023-08-02 15:55:30 UTC 
Results from benchmarking in this comment:
https://bugzilla.redhat.com/show_bug.cgi?id=2135765#c16
Comment 4 mxie@redhat.com 2023-08-03 07:14:20 UTC 
Reproduce the bug with nbdkit-server-1.34.1-1.el9.x86_64

1.Use nbdkit to downlad image from external website

1.1 # time nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	10m5.180s
user	0m19.987s
sys	0m10.309s


1.2 # time nbdkit -r -U - curl https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	5m49.831s
user	0m4.429s
sys	0m4.819s


1.3 # time nbdkit -r -U - curl  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	8m38.746s
user	0m4.781s
sys	0m4.762s

1.4  #  time nbdkit -r -U - curl  https://www.itechtics.com/?dl_id=173 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	119m38.789s
user	0m41.858s
sys	0m41.941s

1.5   # time nbdkit -r -U - curl   https://laotzu.ftp.acc.umu.se/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2  --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	5m12.692s
user	0m4.224s
sys	0m4.747s

2. Use wget to downlad same images from external website

2.1 # time wget -O lunar-server-cloudimg-amd64.img https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img 
--2023-08-03 02:33:41--  https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img
Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 2620:2d:4000:1::17, 2620:2d:4000:1::1a, 185.125.190.40, ...
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|2620:2d:4000:1::17|:443... failed: Network is unreachable.
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|2620:2d:4000:1::1a|:443... failed: Network is unreachable.
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|185.125.190.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 807141376 (770M) [application/octet-stream]
Saving to: ‘lunar-server-cloudimg-amd64.img’

lunar-server-cloudimg-amd64.img         100%[==============================================================================>] 769.75M  7.14MB/s    in 99s     

2023-08-03 02:35:22 (7.75 MB/s) - ‘lunar-server-cloudimg-amd64.img’ saved [807141376/807141376]


real	1m40.823s
user	0m3.257s
sys	0m5.702s


2.2 #  time  wget -O debian-12-generic-amd64-daily.qcow2 https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
--2023-08-03 02:37:41--  https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
Resolving gemmei.ftp.acc.umu.se (gemmei.ftp.acc.umu.se)... 2001:6b0:19::137, 194.71.11.137
Connecting to gemmei.ftp.acc.umu.se (gemmei.ftp.acc.umu.se)|2001:6b0:19::137|:443... failed: Network is unreachable.
Connecting to gemmei.ftp.acc.umu.se (gemmei.ftp.acc.umu.se)|194.71.11.137|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 379029504 (361M)
Saving to: ‘debian-12-generic-amd64-daily.qcow2’

debian-12-generic-amd64-daily.qcow2     100%[==============================================================================>] 361.47M  7.72MB/s    in 44s     

2023-08-03 02:38:26 (8.24 MB/s) - ‘debian-12-generic-amd64-daily.qcow2’ saved [379029504/379029504]


real	0m45.308s
user	0m1.827s
sys	0m2.971s

2.3  #  time wget -O debian-12-generic-amd64-daily.qcow2 https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
--2023-08-03 02:40:09--  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
Resolving cloud.debian.org (cloud.debian.org)... 2001:6b0:19::165, 2001:6b0:19::173, 2001:6b0:19::163, ...
Connecting to cloud.debian.org (cloud.debian.org)|2001:6b0:19::165|:443... failed: Network is unreachable.
Connecting to cloud.debian.org (cloud.debian.org)|2001:6b0:19::173|:443... failed: Network is unreachable.
Connecting to cloud.debian.org (cloud.debian.org)|2001:6b0:19::163|:443... failed: Network is unreachable.
Connecting to cloud.debian.org (cloud.debian.org)|194.71.11.173|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://saimei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 [following]
--2023-08-03 02:40:11--  https://saimei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
Resolving saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)... 2001:6b0:19::138, 194.71.11.138
Connecting to saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)|2001:6b0:19::138|:443... failed: Network is unreachable.
Connecting to saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)|194.71.11.138|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 379029504 (361M)
Saving to: ‘debian-12-generic-amd64-daily.qcow2’

debian-12-generic-amd64-daily.qcow2     100%[==============================================================================>] 361.47M  8.90MB/s    in 46s     

2023-08-03 02:40:58 (7.87 MB/s) - ‘debian-12-generic-amd64-daily.qcow2’ saved [379029504/379029504]


real	0m49.342s
user	0m1.495s
sys	0m2.592s


2.4   #  time wget -O win10  https://www.itechtics.com/?dl_id=173
--2023-08-03 02:42:39--  https://www.itechtics.com/?dl_id=173
Resolving www.itechtics.com (www.itechtics.com)... 2600:1f13:393:63f:e2fe:15a2:b5f1:fde0, 2600:1f13:393:63d:2ad4:3898:9a7f:827c, 2600:1f13:393:63e:1004:3e15:f1a:6bb, ...
Connecting to www.itechtics.com (www.itechtics.com)|2600:1f13:393:63f:e2fe:15a2:b5f1:fde0|:443... failed: Network is unreachable.
Connecting to www.itechtics.com (www.itechtics.com)|2600:1f13:393:63d:2ad4:3898:9a7f:827c|:443... failed: Network is unreachable.
Connecting to www.itechtics.com (www.itechtics.com)|2600:1f13:393:63e:1004:3e15:f1a:6bb|:443... failed: Network is unreachable.
Connecting to www.itechtics.com (www.itechtics.com)|44.231.217.234|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://windowstan.com/download/windows-10-22h2-x64/?wpdmdl=10968&refresh=641a734c7dbe21679455052 [following]
--2023-08-03 02:42:40--  https://windowstan.com/download/windows-10-22h2-x64/?wpdmdl=10968&refresh=641a734c7dbe21679455052
Resolving windowstan.com (windowstan.com)... 95.217.81.177
Connecting to windowstan.com (windowstan.com)|95.217.81.177|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://software.download.prss.microsoft.com/dbazure/Win10_22H2_EnglishInternational_x64v1.iso?t=ffb95491-dfd5-4445-bfb1-4411a9a01012&e=1691123017&h=6319822a786942430948d33cebd8457029d3acae844af967c34fc1c304b4afa4 [following]
--2023-08-03 02:42:43--  https://software.download.prss.microsoft.com/dbazure/Win10_22H2_EnglishInternational_x64v1.iso?t=ffb95491-dfd5-4445-bfb1-4411a9a01012&e=1691123017&h=6319822a786942430948d33cebd8457029d3acae844af967c34fc1c304b4afa4
Resolving software.download.prss.microsoft.com (software.download.prss.microsoft.com)... 43.141.53.90, 182.254.60.13, 43.141.49.146, ...
Connecting to software.download.prss.microsoft.com (software.download.prss.microsoft.com)|43.141.53.90|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6135633920 (5.7G) [application/octet-stream]
Saving to: ‘win10’

win10                                   100%[==============================================================================>]   5.71G  7.35MB/s    in 14m 18s 

2023-08-03 02:57:02 (6.82 MB/s) - ‘win10’ saved [6135633920/6135633920]


real	14m22.853s
user	0m23.859s
sys	0m43.303s


2.5   # time wget -O debian-12-generic-amd64.qcow2    https://laotzu.ftp.acc.umu.se/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2 
--2023-08-03 02:59:26--  https://laotzu.ftp.acc.umu.se/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2
Resolving laotzu.ftp.acc.umu.se (laotzu.ftp.acc.umu.se)... 2001:6b0:19::166, 194.71.11.166
Connecting to laotzu.ftp.acc.umu.se (laotzu.ftp.acc.umu.se)|2001:6b0:19::166|:443... failed: Network is unreachable.
Connecting to laotzu.ftp.acc.umu.se (laotzu.ftp.acc.umu.se)|194.71.11.166|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 378963968 (361M)
Saving to: ‘debian-12-generic-amd64.qcow2’

debian-12-generic-amd64.qcow2           100%[==============================================================================>] 361.41M  9.29MB/s    in 44s     

2023-08-03 03:00:12 (8.19 MB/s) - ‘debian-12-generic-amd64.qcow2’ saved [378963968/378963968]


real	0m45.575s
user	0m1.792s
sys	0m2.962s


3. Convert a guest from VMware without vddk by v2v
#  virt-v2v -ic vpx://root.212.129/data/10.73.212.36/?no_verify=1 esx8.0-rhel9.2-x86_64 -ip /home/passwd -o null
[   0.2] Setting up the source: -i libvirt -ic vpx://root.212.129/data/10.73.212.36/?no_verify=1 esx8.0-rhel9.2-x86_64
[   2.8] Opening the source
[  38.9] Inspecting the source
[ 961.9] Checking for sufficient free disk space in the guest
[ 961.9] Converting Red Hat Enterprise Linux 9.2 (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[2923.8] Mapping filesystem data to avoid copying unused and blank areas
[3440.4] Closing the overlay
[3440.6] Assigning disks to buses
[3440.6] Checking if the guest needs BIOS or UEFI to boot
[3440.6] Setting up the destination: -o null
[3441.7] Copying disk 1/1
█ 100% [****************************************]
[6105.2] Creating output metadata
[6105.2] Finishing off



Test the bug with nbdkit-server-1.34.2-1.el9.x86_64

Steps:
1. Use nbdkit to downlad image from external websites

1.1 #  time nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	2m6.068s
user	0m24.036s
sys	0m11.669s

1.2 #   time nbdkit -r -U - curl https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	1m11.273s
user	0m6.539s
sys	0m6.020s


1.3 #  time nbdkit -r -U - curl  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	5m13.000s
user	1m53.332s
sys	0m8.924s

1.4 # time nbdkit -r -U - curl  https://www.itechtics.com/?dl_id=173 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	29m47.808s
user	1m56.952s
sys	1m42.689s


1.5 # time nbdkit -r -U - curl   https://laotzu.ftp.acc.umu.se/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2  --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	0m59.037s
user	0m6.658s
sys	0m5.982s


2.  Convert a guest from VMware without vddk by v2v

# virt-v2v -ic vpx://root.212.129/data/10.73.212.36/?no_verify=1 esx8.0-rhel9.2-x86_64 -ip /home/passwd -o null
[   0.2] Setting up the source: -i libvirt -ic vpx://root.212.129/data/10.73.212.36/?no_verify=1 esx8.0-rhel9.2-x86_64
[   2.7] Opening the source
[  36.6] Inspecting the source
[ 965.9] Checking for sufficient free disk space in the guest
[ 965.9] Converting Red Hat Enterprise Linux 9.2 (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[2945.1] Mapping filesystem data to avoid copying unused and blank areas
[3490.4] Closing the overlay
[3490.7] Assigning disks to buses
[3490.7] Checking if the guest needs BIOS or UEFI to boot
[3490.7] Setting up the destination: -o null
[3492.2] Copying disk 1/1
█ 100% [****************************************]
[6155.4] Creating output metadata
[6155.4] Finishing off



Summary of the timings:

                                     nbdkit-1.34.1           wget                nbdkit-1.34.2

cloud-images.ubuntu.com              10m5.180s               1m40.823s           2m6.068s

gemmei.ftp.acc.umu.se                5m49.831s               0m43.591s           1m11.273s

cloud.debian.org                     8m38.746s               0m49.342s           5m13.000s

www.itechtics.com                    119m38.789s             14m22.853s          29m47.808s

laotzu.ftp.acc.umu.se                5m12.692s               0m45.575s           0m59.037s

VMware with v2v                      6105.2                                      6155.4



Hi Richard,
        
   Please check above summary of the timings, although the performance of nbdkit-1.34.2 is much better than that of nbdkit-1.34.1, nbdkit-1.34.2 still performs worse than wget(maybe today's internet speed is different from yesterday's), especially when downloading image from www.itechtics.com and cloud.debian.org, is it expected? By the way, I have attached the v2v debug log which is generated when convert guest from VMware without vddk for your reference, thanks!
Comment 6 Richard W.M. Jones 2023-08-03 09:06:58 UTC 
A few notes:

(1) We don't expect URLs using cloud.debian.org to ever perform well.
They have to be "pre-resolved" first using:

$ curl -I https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
HTTP/1.1 302 Found
Date: Thu, 03 Aug 2023 09:00:25 GMT
Server: Apache/2.4.55 (Unix)
Location: https://saimei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2
Cache-Control: max-age=300
Expires: Thu, 03 Aug 2023 09:05:25 GMT
Content-Type: text/html; charset=iso-8859-1

The reason for this is that curl keeps going back to cloud.debian.org and following
the redirect for every block (because curl doesn't cache redirects).

(2) These are HTTP/1.1 URLs.  To improve performance you might try adjusting
the nbdkit connections=... parameter upwards.  For the ubuntu URL I got:

      multi, connections=64           17.2s
      multi, connections=32           21.7s
      wget                            28.4s
      multi, connections=16           41.3s
      before this commit             180s

(3) The www.itechtics.com URL is weird.  It redirects to a cloudflare site which
redirects again to a Microsoft site.  It's just not a good URL to test at all.
It might be better to use a Microsoft URL directly.
https://www.microsoft.com/en-gb/software-download/windows10iso

Microsoft also use HTTP/2 which is much more efficient than HTTP/1.1.
Comment 7 Richard W.M. Jones 2023-08-03 09:21:58 UTC 
For the Microsoft ISO URI I got these times on Fedora:

$ time nbdkit -r -U - curl 'https://software.download.prss.microsoft.com/dbazure/Win10_22H2_EnglishInternational_x64v1.iso?t=1b2546c2-906a-4728-89e3-74929cec6099&e=1691139928&h=9eb3286c4f81a024ed30c72cc47438f1d595ebfeb402e536a85d967acc002a82' --run 'nbdcopy -p $uri null:'

nbdkit 1.35.7: 3m21 (before patch)

nbdkit 1.35.8: 3m17 (after patch)

         wget: 1m48

Basically not much difference with this patch, & wget is a bit faster.
Comment 8 mxie@redhat.com 2023-08-03 10:18:23 UTC 

(In reply to Richard W.M. Jones from comment #6)
> A few notes:
> 
> (1) We don't expect URLs using cloud.debian.org to ever perform well.
> They have to be "pre-resolved" first using:
> 
> $ curl -I
> https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-
> generic-amd64-daily.qcow2
> HTTP/1.1 302 Found
> Date: Thu, 03 Aug 2023 09:00:25 GMT
> Server: Apache/2.4.55 (Unix)
> Location:
> https://saimei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-
> generic-amd64-daily.qcow2
> Cache-Control: max-age=300
> Expires: Thu, 03 Aug 2023 09:05:25 GMT
> Content-Type: text/html; charset=iso-8859-1
> 
> The reason for this is that curl keeps going back to cloud.debian.org and
> following
> the redirect for every block (because curl doesn't cache redirects).
> 
> (2) These are HTTP/1.1 URLs.  To improve performance you might try adjusting
> the nbdkit connections=... parameter upwards.  For the ubuntu URL I got:
> 
>       multi, connections=64           17.2s
>       multi, connections=32           21.7s
>       wget                            28.4s
>       multi, connections=16           41.3s
>       before this commit             180s

1. Download image from cloud.debian.org with nbdkit-1.34.2-1, the performance can be improved with increasing the value of connections option

# time nbdkit -r -U - curl  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 connections=32 --run 'nbdcopy -p $uri null:' 
█ 100% [****************************************]

real	3m26.945s
user	1m44.090s
sys	0m7.545s

# time nbdkit -r -U - curl  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 connections=64 --run 'nbdcopy -p $uri null:' 
█ 100% [****************************************]

real	1m14.439s
user	0m22.020s
sys	0m6.095s

# time nbdkit -r -U - curl  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 connections=128 --run 'nbdcopy -p $uri null:' 
█ 100% [****************************************]

real	0m50.230s
user	0m9.892s
sys	0m6.887s

# time nbdkit -r -U - curl  https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 connections=256 --run 'nbdcopy -p $uri null:' 
█ 100% [****************************************]

real	0m46.594s
user	0m9.572s
sys	0m7.317s


2. Download image from cloud-images.ubuntu.com with nbdkit-1.34.2-1, the performance can be improved with increasing the value of connections option

#  time nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img connections=32 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	1m12.125s
user	0m21.254s
sys	0m11.009s

#  time nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img connections=64 --run 'nbdcopy -p $uri null:'
█ 100% [****************************************]

real	1m0.593s
user	0m19.532s
sys	0m10.207s



Hi Richard,

   I saw the default connections number is 16 for nbdkit curl plugin, why not make the curl plugin's default connection number larger if its performance can be improved by increasing the value of the connections option ?
Comment 9 Richard W.M. Jones 2023-08-03 10:33:17 UTC 
Unfortunately increasing the connections limit actually causes us to make
lots more TCP connections to the remote server, which is not very nice.  This
is particularly bad for HTTP/1.1 where each "stream" of data corresponds to
a new TCP connection.

HTTP/2 allows a single TCP connection to multiplex up to about 100 streams.

For example:

$ time ./nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img --run 'nbdcopy -p $uri null:' ipresolve=v4 connections=32

While this is running do:

$ sudo netstat -anp | grep 185.125.190.40:443 | wc -l
32

You can see it is actually opening 32 TCP connections.

For HTTP/2 this is much better.  For example:

$ time ./nbdkit -r -U - curl 'https://software.download.prss.microsoft.com/dbazure/Win10_22H2_EnglishInternational_x64v1.iso?t=1b2546c2-906a-4728-89e3-74929cec6099&e=1691139928&h=9eb3286c4f81a024ed30c72cc47438f1d595ebfeb402e536a85d967acc002a82' --run 'nbdcopy -p $uri null:' connections=256

$ sudo netstat -anp | grep :443
tcp        0      0 10.19.152.9:39124       152.195.19.97:443       ESTABLISHED 414492/nbdkit       

Even though I asked for 256 connections, curl is able to multiplex many
streams over a single HTTP/2 connection, so only one TCP connection is opened.

The updated curl plugin is much more efficient for HTTP/2, even if it doesn't
seem to be very much faster.
Comment 10 Richard W.M. Jones 2023-08-03 10:38:23 UTC 
Another thing you can try is to increase the nbdcopy --connections, eg:

$ nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img --run 'nbdcopy -p --connections=16 $uri null:' ipresolve=v4 

  nbdcopy <--------->  nbdkit curl plugin  <--------->  remote web server
              NBD                             HTTPS

Normally nbdcopy defaults to 4 NBD connections, but using nbdcopy --connections increases
that.

With old nbdkit this caused the curl plugin to "fight" over a limited pool of
curl handles.  The size of the pool was controlled by the (nbdkit) connections=XX
parameter.

With new nbdkit there is no internal limit on curl handles, but the (nbdkit) connections=XX
parameter limits how many HTTPS connections are made.  This seems to me to be much
more intuitive than the old behaviour.
Comment 13 mxie@redhat.com 2023-08-04 14:11:23 UTC 
Since new nbdkit has much better performance than old nbdkit according to the result of comment4, the bug has been fixed, so move the bug to VERIFIED status
Comment 15 errata-xmlrpc 2023-11-07 08:28:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nbdkit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6374