*** Bug 2057748 has been marked as a duplicate of this bug. ***

The additional support for mtu setting (as described in bug 2057748) can be done as part of this requirement or can be done in a separate bug.

Hi Jonathon, Please help to check bug 2068999 - Guest doesn't show correct queues number after Hot plug vdap with multiple queues. I can also reproduce the bug. Thank you!

Start vm with vdpa interface set with multiqueue
# rpm -q libvirt qemu-kvm kernel iproute
libvirt-8.0.0-7.el9_0.x86_64
qemu-kvm-6.2.0-11.el9_0.1.x86_64
kernel-5.14.0-70.4.1.el9_0.x86_64
iproute-5.15.0-2.2.el9_0.x86_64

guest kernel:
5.14.0-70.2.1.el9_0.x86_64

1. Initialize the vdpa device
# ./ovs_init.sh  0000:5e:00.0  4 

2. Prepare vm with device as below:
<vcpu placement='static'>10</vcpu>
...
<interface type='vdpa'>
      <mac address='00:11:22:33:44:00'/>
      <source dev='/dev/vhost-vdpa-0'/>
      <model type='virtio'/>
      <driver queues='8'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

3. Start the vm:
# virsh start rhel
# virsh domiflist rhel 
 Interface   Type   Source              Model    MAC
--------------------------------------------------------------------
 -           vdpa   /dev/vhost-vdpa-0   virtio   00:11:22:33:44:00

login the vm, and check the queues:
[root@localhost ~]# ethtool -l enp1s0
Channel parameters for enp1s0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	8
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	8

4. Configure the ip address of the enp94s0f0np0_br port on the host, and the ip address of the vm interface, test the connectivity:
on guest:
[root@localhost ~]# ping 100.100.100.100 -c 2
PING 100.100.100.100 (100.100.100.100) 56(84) bytes of data.
64 bytes from 100.100.100.100: icmp_seq=1 ttl=64 time=0.219 ms
64 bytes from 100.100.100.100: icmp_seq=2 ttl=64 time=0.314 ms

--- 100.100.100.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.219/0.266/0.314/0.047 ms

5. Reboot the vm, after reboot, guest works well except there is an error log in /var/log/libvirt/qemu/rhel.log, it is a qemu bug tracked bug 2055955
# cat /var/log/libvirt/qemu/rhel.log
......
2022-03-28T07:43:26.349926Z qemu-kvm: vhost VQ 16 ring restore failed: -22: Invalid argument (22)

6. suspend and resume the vm, will also got the error messages as above(should be the same qemu bug).

hotplug and unplug vdpa type interface with multiqueue

1.Start 1 vm with 1 interface:
# virsh start rhel 
viDomain 'rhel' started

# virsh domiflist rhel 
 Interface   Type      Source    Model    MAC
-------------------------------------------------------------
 vnet3       network   default   virtio   52:54:00:9b:b1:85

2. prepare a vdpa interface xml with multiqueue:
# cat interface.xml 
<interface type='vdpa'>
      <mac address='00:11:22:33:44:11'/>
      <source dev='/dev/vhost-vdpa-1'/>
      <model type='virtio'/>
      <driver queues='8'/>
    </interface>

# virsh attach-device rhel interface.xml  
Device attached successfully

3. check on vm:
[root@localhost ~]# ethtool -l enp4s0
Channel parameters for enp4s0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	1
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	1

The queue size is 1, which is not expected. This issue is reported as bug 2068999
Catch the related libvirtd.log as below:
2022-03-28 08:07:05.035+0000: 2230: info : qemuMonitorSend:914 : QEMU_MONITOR_SEND_MSG: mon=0x7f4fe0086b90 msg={"execute":"device_add","arguments":{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"00:11:22:33:44:11","bus":"pci.4","addr":"0x0"},"id":"libvirt-416"}
 fd=-1
2022-03-28 08:07:05.035+0000: 4613: info : virObjectRef:402 : OBJECT_REF: obj=0x7f4fe0086b90
2022-03-28 08:07:05.035+0000: 4613: info : qemuMonitorIOWrite:402 : QEMU_MONITOR_IO_WRITE: mon=0x7f4fe0086b90 buf={"execute":"device_add","arguments":{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"00:11:22:33:44:11","bus":"pci.4","addr":"0x0"},"id":"libvirt-416"}
 len=170 ret=170 errno=0

There should be elements like this in the command: 
"mq":true,"vectors":18 

This is reported on libvirt bug 2068999

4. hotunplug works well.
# virsh detach-device rhel interface.xml  
Device detached successfully

5. hotplug and hotunplug again, works well except the issue in step 3.

As the mac address and queue size set by libvirt will not take effects, it is suggested to use the same mac address and queue size set by vdpa tool. Should we mentions it somewhere in the document? Please help to confirm, Thank you!

Unfortunately it seems that we missed something in libvirt. I will propose a patch upstream very soon.

Test scenarios in comment 33 and comment 34, PASS.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390