Patches posted upstream.

  https://listman.redhat.com/archives/libvir-list/2021-October/msg00459.html

v2 patches posted upstream.

  https://listman.redhat.com/archives/libvir-list/2021-October/msg00784.html

The patches have been ACKed upstream, and I've already pushed the
first few ones - those that were just setting the stage for
implementing the feature and made sense even as standalone cleanups.

I'll push the remaining ones once the QEMU part is merged upstream.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Reopening, this is planned for 9.1.

v3 patches, mistakenly marked as v2, posted upstream.

  https://listman.redhat.com/archives/libvir-list/2022-March/229397.html

(In reply to Andrea Bolognani from comment #16)
> v3 patches, mistakenly marked as v2, posted upstream.
> 
>   https://listman.redhat.com/archives/libvir-list/2022-March/229397.html

Eric, can you give these a try? The switch to JSON syntax for
-device, which is the only significant user-visible difference
compared to the previous version, should not cause any issues, but
it'd still be nice to get confirmation of that. I can pick up your
Tested-by tags if you provide them. Thanks!

(In reply to Andrea Bolognani from comment #17)
> (In reply to Andrea Bolognani from comment #16)
> > v3 patches, mistakenly marked as v2, posted upstream.
> > 
> >   https://listman.redhat.com/archives/libvir-list/2022-March/229397.html
> 
> Eric, can you give these a try? The switch to JSON syntax for
> -device, which is the only significant user-visible difference
> compared to the previous version, should not cause any issues, but
> it'd still be nice to get confirmation of that. I can pick up your
> Tested-by tags if you provide them. Thanks!

Hi Andrea, yes sure I will be happy test. Can you provide me with a brew build or guide me through a libvirt compilation from src (I saw your provided your branch link). What is the simplest?

Thanks

Eric

(In reply to Eric Auger from comment #18)
> (In reply to Andrea Bolognani from comment #17)
> > Eric, can you give these a try? The switch to JSON syntax for
> > -device, which is the only significant user-visible difference
> > compared to the previous version, should not cause any issues, but
> > it'd still be nice to get confirmation of that. I can pick up your
> > Tested-by tags if you provide them. Thanks!
> 
> Hi Andrea, yes sure I will be happy test. Can you provide me with a brew
> build or guide me through a libvirt compilation from src (I saw your
> provided your branch link). What is the simplest?

I actually realized that you don't need any specific hardware to test
these, which for some reason I was convinced was the case. So I just
went ahead and tested the changes myself - everything seems to be
fine.

In case you still want to play around with the patches, and for
future reference, the easiest way to test a random libvirt branch is
to build RPMs from it, which roughly looks like

  $ mkdir build && cd build
  $ meson && meson dist --no-tests
  $ rpmbuild -ta meson-dist/libvirt-*.tar.xz

To figure out the build dependencies, you can look into

  ci/containers/centos-stream-*.Dockerfile

Note that a few packages listed in the file are not available in
RHEL, but none of them is a hard requirement, so you can safely skip
them. You should also be able to avoid enabling the extra
repositories.

If you have installed your custom build of QEMU under /usr/local,
then you'll need to run something like

  $ for w in own mod con; do
      sudo ch$w --reference /usr/libexec/qemu-kvm \
                /usr/local/bin/qemu-system-aarch64;
    done

or libvirt will be unable to use it. You might be aware of this
already, but I've included it here for completeness' sake :)

v4 patches posted upstream.

  https://listman.redhat.com/archives/libvir-list/2022-April/229810.html

Merged upstream.

  commit 19734c3050c70dc3452c48af70558f5a06152031
  Author: Andrea Bolognani <abologna>
  Date:   Fri Sep 24 19:29:37 2021 +0200

    qemu: Generate command line for virtio-iommu
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1653327
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Ján Tomko <jtomko>

  v8.2.0-51-g19734c3050

It will be in libvirt 8.3.0, releasing early next month.

Hi Andrea and Eric. 

I used the upstream qemu and libvirt to test in advance.

libvirt: e53c02ea20 (HEAD -> master, origin/master, origin/HEAD) virportallocator: Use automatic mutex management
qemu: f53faa70bb (HEAD -> master, origin/staging, origin/master, origin/HEAD) Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging

Parts of guest xml:
```
    <iommu model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </iommu>

```

Then start guest.
a. Check qemu cmdline
cmdline has virtio-iommu element
```
-device {"driver":"virtio-iommu","bus":"pcie.0","addr":"0x3"}
```

b. Check guest dmesg
```
# dmesg | grep iommu
[    2.613310] iommu: Default domain type: Translated
[    2.614874] iommu: DMA domain TLB invalidation policy: lazy mode
[    2.949607] virtio_iommu virtio0: input address: 64 bits
[    2.951688] virtio_iommu virtio0: page mask: 0xfffffffffffff000
[    2.985120] xhci_hcd 0000:02:00.0: Adding to iommu group 0
[    2.986845] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA                                                                 
[    4.035370] pcieport 0000:00:01.0: Adding to iommu group 1
[    4.037157] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA                                                                 
[    4.056668] pcieport 0000:00:01.1: Adding to iommu group 1
[    4.070602] pcieport 0000:00:01.2: Adding to iommu group 1
[    4.087266] pcieport 0000:00:01.3: Adding to iommu group 1
[    4.103366] pcieport 0000:00:01.4: Adding to iommu group 1
[    4.119086] pcieport 0000:00:01.5: Adding to iommu group 1
[    4.135109] pcieport 0000:00:01.6: Adding to iommu group 1
[    4.151071] pcieport 0000:00:01.7: Adding to iommu group 1
[    4.167282] pcieport 0000:00:02.0: Adding to iommu group 2
[    4.169051] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA                                                                 
[    4.190429] pcieport 0000:00:02.1: Adding to iommu group 2
[    4.206620] pcieport 0000:00:02.2: Adding to iommu group 2
[    4.222490] virtio-pci 0000:01:00.0: Adding to iommu group 1
[    4.232386] virtio-pci 0000:03:00.0: Adding to iommu group 1
[    4.242856] virtio-pci 0000:04:00.0: Adding to iommu group 1
[    4.253576] virtio-pci 0000:05:00.0: Adding to iommu group 1
[    4.259425] virtio-pci 0000:06:00.0: Adding to iommu group 1
[    4.268598] virtio-pci 0000:07:00.0: Adding to iommu group 1
```

I just did a most basic test, if there is anything else I need to test please tell me.

(In reply to Yiding Liu (Fujitsu) from comment #23)
> I just did a most basic test, if there is anything else I need to test
> please tell me.

Hi,

apologies for the long delay in replying. Your message fell through
the cracks somehow.

What you did for testing looks reasonable enough for me - in fact,
it's pretty much exactly what I did during development :)

Eric, any additional thoughts?

the downstream qemu-kvm shall be usable to test the feature as the following patches are now there:

d9c96f2425  virtio-iommu: Support bypass domain (8 weeks ago) <Jean-Philippe Brucker>
448179e33e  virtio-iommu: Default to bypass during boot (8 weeks ago) <Jean-Philippe Brucker>

What did you use as a guest? boot bypass feature is not yet supported downstream: I guess you use upstream kernel?

I think the test should feature both a virtio block pci device and virtio net pci device protected by the virtio-iommu. A virtio-gpu may be a plus as well.

Check the iommu groups on the guest
find /sys/kernel/iommu_groups/

You can also stimulate the protected NIC using dnf install <pkg> or any script like:

#!/bin/bash

for i in `seq 1 30`
do
    echo "iteration $i"
    if [ -d netperf ]; then
        rm -rf netperf
    fi
    git clone https://github.com/HewlettPackard/netperf.git netperf
    if [ -d iperf ]; then
        rm -rf iperf
    fi
    git clone https://github.com/esnet/iperf.git iperf
done

Doing a whole VM install with the virtio-iommu would be perfect as well.

Thanks

Eric

(In reply to Eric Auger from comment #25)
> the downstream qemu-kvm shall be usable to test the feature as the following
> patches are now there:
> 
> d9c96f2425  virtio-iommu: Support bypass domain (8 weeks ago) <Jean-Philippe
> Brucker>
> 448179e33e  virtio-iommu: Default to bypass during boot (8 weeks ago)
> <Jean-Philippe Brucker>
> 
> What did you use as a guest? boot bypass feature is not yet supported
> downstream: I guess you use upstream kernel?

No, I use a RHEL9 guest to test. 

> 
> I think the test should feature both a virtio block pci device and virtio
> net pci device protected by the virtio-iommu. A virtio-gpu may be a plus as
> well.
> 
> Check the iommu groups on the guest
> find /sys/kernel/iommu_groups/
> 
> You can also stimulate the protected NIC using dnf install <pkg> or any
> script like:
> 
> #!/bin/bash
> 
> for i in `seq 1 30`
> do
>     echo "iteration $i"
>     if [ -d netperf ]; then
>         rm -rf netperf
>     fi
>     git clone https://github.com/HewlettPackard/netperf.git netperf
>     if [ -d iperf ]; then
>         rm -rf iperf
>     fi
>     git clone https://github.com/esnet/iperf.git iperf
> done
> 
> Doing a whole VM install with the virtio-iommu would be perfect as well.

Ok, got it. Thanks for your advice.

> 
> Thanks
> 
> Eric

Pre Verify Env:
Host:
 kernel-5.14.0-92.el9.aarch64
 libvirt-8.3.0-1.el9.aarch64
 qemu-kvm-7.0.0-3.el9.aarch64
Guest:
 kernel-5.14.0-92.el9.aarch64

Pre Verify Result:
a. Guest can successfully start with the virtio-iommu
Parts of guest xml:
```
    <iommu model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </iommu>
```
b. dmesg | grep iommu
```
# dmesg | grep iommu
[    3.316754] iommu: Default domain type: Translated 
[    3.318171] iommu: DMA domain TLB invalidation policy: lazy mode 
[    3.740987] virtio_iommu virtio0: input address: 64 bits
[    3.742534] virtio_iommu virtio0: page mask: 0xfffffffffffff000
[    3.799342] xhci_hcd 0000:02:00.0: Adding to iommu group 0
[    3.800934] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    6.045464] pcieport 0000:00:01.0: Adding to iommu group 1
[    6.047995] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    6.098540] pcieport 0000:00:01.1: Adding to iommu group 1
[    6.117988] pcieport 0000:00:01.2: Adding to iommu group 1
[    6.143471] pcieport 0000:00:01.3: Adding to iommu group 1
[    6.165886] pcieport 0000:00:01.4: Adding to iommu group 1
[    6.187972] pcieport 0000:00:01.5: Adding to iommu group 1
[    6.220452] pcieport 0000:00:01.6: Adding to iommu group 1
[    6.254336] pcieport 0000:00:01.7: Adding to iommu group 1
[    6.272365] pcieport 0000:00:02.0: Adding to iommu group 2
[    6.273923] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    6.299766] pcieport 0000:00:02.1: Adding to iommu group 2
[    6.316418] pcieport 0000:00:02.2: Adding to iommu group 2
[    6.333190] virtio-pci 0000:01:00.0: Adding to iommu group 1
[    6.344131] virtio-pci 0000:03:00.0: Adding to iommu group 1
[    6.355628] virtio-pci 0000:04:00.0: Adding to iommu group 1
[    6.367419] virtio-pci 0000:05:00.0: Adding to iommu group 1
[    6.373608] virtio-pci 0000:06:00.0: Adding to iommu group 1
[    6.383215] virtio-pci 0000:07:00.0: Adding to iommu group 1
```
c. the iommu groups on the guest
```
# ll /sys/kernel/iommu_groups/
total 0
drwxr-xr-x. 3 root root 0 May 18 14:12 0
drwxr-xr-x. 3 root root 0 May 18 14:12 1
drwxr-xr-x. 3 root root 0 May 18 14:12 2
```

2 things:

I see this xml snippet, suggesting that the virtio-iommu-pci is potentially pluggable at any PCIe address. An issue was reported if this latter is instantiated elsewhere than on the root bus (pcie root prot for instance) using the qemu interface (BZ2087155). However maybe and most probably given Andrea's info such setting is forbidden by libvirt. However it would be nice to double check if this is handled: ie. try on instantiate it on a pcie root port instead.

    <iommu model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </iommu>


> # ll /sys/kernel/iommu_groups/
> total 0
> drwxr-xr-x. 3 root root 0 May 18 14:12 0
> drwxr-xr-x. 3 root root 0 May 18 14:12 1
> drwxr-xr-x. 3 root root 0 May 18 14:12 2
> ```
I am surprised to see no group here.

(In reply to Eric Auger from comment #28)

> > # ll /sys/kernel/iommu_groups/
> > total 0
> > drwxr-xr-x. 3 root root 0 May 18 14:12 0
> > drwxr-xr-x. 3 root root 0 May 18 14:12 1
> > drwxr-xr-x. 3 root root 0 May 18 14:12 2
> > ```
> I am surprised to see no group here.
Need another coffee. Forget that one ;-)

(In reply to Eric Auger from comment #28)
> 2 things:
> 
> I see this xml snippet, suggesting that the virtio-iommu-pci is potentially
> pluggable at any PCIe address. An issue was reported if this latter is
> instantiated elsewhere than on the root bus (pcie root prot for instance)
> using the qemu interface (BZ2087155). However maybe and most probably given
> Andrea's info such setting is forbidden by libvirt. However it would be nice
> to double check if this is handled: ie. try on instantiate it on a pcie root
> port instead.
> 
>     <iommu model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </iommu>


Hi, Eric

I did some change for the xml：
```
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>

    <iommu model='virtio'>
      <address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
    </iommu>
```

Then it failed to define:
```
# virsh define /tmp/fj-km-vm.xml 
error: Failed to define domain from /tmp/fj-km-vm.xml
error: XML error: The device at PCI address 0000:0c:00.0 needs to be an integrated device (bus=0)
```

Is it appropriate to test this way, does it fail as expected?

Thanks!
Hu Shuai

> Is it appropriate to test this way, does it fail as expected?

Yes that looks the right way to test and it confirms the expectation. Thank you!

Eric

Env:
Host kernel: 5.14.0-96.el9.aarch64
Qemu: qemu-kvm-7.0.0-4.el9.aarch64
Libivrt: libvirt-8.3.0-1.el9.aarch64
Guest kernel: 

Based on comment 25.
Verify this BZ from install stage.

Step1.
Install a guest with virtio-iommu and virtio block pci device, virtio net pci device, virtio-gpu protected by the virtio-iommu.
# virt-install --name test-iommu --machine virt --memory 8192 --vcpu 8 --iommu virtio --import --disk /var/lib/libvirt/images/RHEL-9.0-aarch64-latest.qcow2,driver
.iommu=on --network network=default,model=virtio,driver.iommu=on --video model=virtio,driver.iommu=on


Step2.
Guest start successfully, check qemu-cmdline. Devices were created as expected.
```
/usr/libexec/qemu-kvm -name guest=test-iommu,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-6-test-iommu/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/test-iommu_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine virt-rhel9.0.0,usb=off,dump-guest-core=off,gic-version=3,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=mach-virt.ram -accel kvm -cpu host -m 8192 -object {"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":8589934592} -overcommit mem-lock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 17ed9a0f-75a4-4638-aeb5-7b3112337f84 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=32,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device {"driver":"virtio-iommu","bus":"pcie.0","addr":"0x2"} -device {"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"} -device {"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"} -device {"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"} -device {"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"} -device {"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"} -device {"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"} -device {"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/RHEL-9.0-aarch64-latest.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device {"driver":"virtio-blk-pci","iommu_platform":true,"bus":"pci.4","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1} -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=35 -device {"driver":"virtio-net-pci","iommu_platform":true,"netdev":"hostnet0","id":"net0","mac":"52:54:00:40:b1:82","bus":"pci.1","addr":"0x0"} -chardev pty,id=charserial0 -serial chardev:charserial0 -chardev socket,id=charchannel0,fd=30,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -audiodev {"id":"audio1","driver":"none"} -device {"driver":"virtio-gpu-pci","iommu_platform":true,"id":"video0","max_outputs":1,"bus":"pci.5","addr":"0x0"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
```

Step3. Login guest(via console) and check device info, dmesg and iommu_groups
```
[root@localhost devices]# lspci             
00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge
00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port               
00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port      
00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port             
00:01.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port        
00:01.4 PCI bridge: Red Hat, Inc. QEMU PCIe Root port                                                                                                                                         
00:01.5 PCI bridge: Red Hat, Inc. QEMU PCIe Root port        
00:02.0 Unclassified device [00ff]: Red Hat, Inc. Device 1057 (rev 01)                                                                                                                        
01:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
02:00.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
03:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01)
04:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
05:00.0 Display controller: Red Hat, Inc. Virtio GPU (rev 01)
[root@localhost devices]# dmesg | grep iommu                                                   
[    1.262372] iommu: Default domain type: Translated          
[    1.264475] iommu: DMA domain TLB invalidation policy: lazy mode 
[    1.523717] virtio_iommu virtio0: input address: 64 bits    
[    1.526277] virtio_iommu virtio0: page mask: 0xfffffffffffff000
[    1.552675] xhci_hcd 0000:02:00.0: Adding to iommu group 0                                                                                                                                 
[    1.554958] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA                                                                  
[    2.552210] pcieport 0000:00:01.0: Adding to iommu group 1                                                                                                                                 
[    2.554602] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA                                                                  
[    2.577648] pcieport 0000:00:01.1: Adding to iommu group 1                                                                                                                                 
[    2.594593] pcieport 0000:00:01.2: Adding to iommu group 1                                                                                                                                 
[    2.614362] pcieport 0000:00:01.3: Adding to iommu group 1                                                                                                                                 
[    2.633246] pcieport 0000:00:01.4: Adding to iommu group 1                                                                                                                                 
[    2.651717] pcieport 0000:00:01.5: Adding to iommu group 1                                                                                                                                 
[    2.670112] virtio-pci 0000:01:00.0: Adding to iommu group 1                                                                                                                               
[    2.680815] virtio-pci 0000:03:00.0: Adding to iommu group 1                                                                                                                               
[    2.691762] virtio-pci 0000:04:00.0: Adding to iommu group 1                                                                                                                               
[    2.697294] virtio-pci 0000:05:00.0: Adding to iommu group 1  
[root@localhost devices]# ls /sys/kernel/iommu_groups/1/devices/
0000:00:01.0  0000:00:01.2  0000:00:01.4  0000:01:00.0  0000:04:00.0
0000:00:01.1  0000:00:01.3  0000:00:01.5  0000:03:00.0  0000:05:00.0 
```
Devices were available in the guest and iommu_group info matched dmesg iommu.

Step4. Reboot guest && Login guest(via ssh) && Stimulate the protected NIC
```
[root@localhost ~]# ./test.sh                                                                                                                                                                 
iteration 1                                                                                                                                                                                   
Cloning into 'netperf'...                                                                                                                                                                     
remote: Enumerating objects: 5252, done.                                                                                                                                                      
remote: Counting objects: 100% (324/324), done.                                                                                                                                               
remote: Compressing objects: 100% (94/94), done.                                                                                                                                              
remote: Total 5252 (delta 234), reused 305 (delta 229), pack-reused 4928                                                                                                                      
Receiving objects: 100% (5252/5252), 16.75 MiB | 13.71 MiB/s, done.                                                                                                                           
Resolving deltas: 100% (3930/3930), done.                                                                                                                                                     
Cloning into 'iperf'...
[snip]
iteration 30
Cloning into 'netperf'...
remote: Enumerating objects: 5252, done.
remote: Counting objects: 100% (324/324), done.
remote: Compressing objects: 100% (94/94), done.
remote: Total 5252 (delta 234), reused 305 (delta 229), pack-reused 4928
Receiving objects: 100% (5252/5252), 16.75 MiB | 12.39 MiB/s, done.
Resolving deltas: 100% (3930/3930), done.
Cloning into 'iperf'...
remote: Enumerating objects: 8821, done.
remote: Counting objects: 100% (687/687), done.
remote: Compressing objects: 100% (300/300), done.
remote: Total 8821 (delta 458), reused 560 (delta 387), pack-reused 8134
Receiving objects: 100% (8821/8821), 12.55 MiB | 13.84 MiB/s, done.
Resolving deltas: 100% (6237/6237), done.                       
```

Verified.
@yalzhang. Please help set the BZ as Verified. Thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8003