Bug 2068999

Description of problem:
Boot VM, then hot plug vdpa with multiple queues. Check queues with "ethtool -l $nic" in VM, it still shows 1 queue.

Version-Release number of selected component (if applicable):
5.14.0-70.5.1.el9_0.x86_64
qemu-kvm-6.2.0-11.el9_0.1.x86_64
libvirt-8.0.0-7.el9_0.x86_64
iproute-5.15.0-2.2.el9_0.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Setup vdpa with max_vqp 2

# modprobe vhost_vdpa
# modprobe mlx5_vdpa

# echo 0 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
# readlink /sys/bus/pci/devices/0000:3b:00.0/virtfn*
# devlink dev eswitch set pci/0000:3b:00.0 mode switchdev
# echo 1 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
# readlink /sys/bus/pci/devices/0000:3b:00.0/virtfn*
# echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind
# devlink dev eswitch set pci/0000:3b:00.0 mode switchdev
# echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/bind

# vdpa mgmtdev show | grep pci
pci/0000:3b:00.2: 
# vdpa dev add name vdpa0 mgmtdev pci/0000:3b:00.2 mac 00:11:22:33:44:03 max_vqp 2


2. Boot VM without any vdpa ports

3. Hot plug vdpa with 2 queues

# cat rhel9.0_vdpa_nic.xml
<interface type="vdpa">
  <mac address="00:11:22:33:44:03" />
  <source dev="/dev/vhost-vdpa-0" />
  <model type="virtio" />
  <driver queues='2' />
  <address type="pci" domain="0x0000" bus="0x6" slot="0x00" function="0x0" />
</interface>


# virsh attach-device rhel9.0 rhel9.0_vdpa_nic.xml 
Device attached successfully


4. Check queues in VM, it still shows 1, however we expect 2.

# ethtool -l enp6s0
Channel parameters for enp6s0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	1
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	1

Actual results:
Queues in VM is wrong.

Expected results:
Queues in VM should show expected value.

Additional info:
1. This issue cannot be reproduced with qemu.

(1) Boot qemu
# ulimit -l unlimited
# /usr/libexec/qemu-kvm \
-name guest=rhel9.0,debug-threads=on \
-machine pc-q35-rhel9.0.0,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=split \
-accel kvm \
-cpu Skylake-Server-IBRS,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rsba=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,tsc-deadline=on,pmu=off \
-m 8192 \
-overcommit mem-lock=on \
-smp 6,sockets=3,dies=1,cores=1,threads=2 \
-uuid 91e86dae-adf5-11ec-b911-20040fec000c \
-no-user-config \
-nodefaults \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-blockdev '{"driver":"file","filename":"/home/images_nfv-virt-rt-kvm/rhel9.0.qcow2","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.2,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=88:66:da:5f:dd:11,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-device VGA,bus=pci.4 \
-vnc :0  \

(2) Hot plug vdpa 2 queues
{"execute": "qmp_capabilities"}
{"return": {}}

{"execute":"netdev_add","arguments":{"type":"vhost-vdpa","id":"hostnet1","vhostdev":"/dev/vhost-vdpa-0","queues": 2}}
{"return": {}}

{"execute":"device_add","arguments":{"driver":"virtio-net-pci","netdev":"hostnet1","mac":"00:11:22:33:44:03","id": "net1","bus":"pci.6","addr":"0x0","mq": true,"vectors": 6}}
{"return": {}}

(3) Check queues in VM, it's expected queue value.
# ethtool -l enp6s0
Channel parameters for enp6s0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	2
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	2



2. With libvirt, boot VM with vdpa 2 queues directly, everything works expected.

<domain type="kvm">
  <name>rhel9.0</name>
....
  <devices>
....
    <interface type="vdpa">
      <mac address="00:11:22:33:44:03" />
      <source dev="/dev/vhost-vdpa-0" />
      <model type="virtio" />
      <driver queues='2' />
      <address type="pci" domain="0x0000" bus="0x6" slot="0x00" function="0x0" />
    </interface>
....
  </devices>
</domain>

# ethtool -l enp6s0
Channel parameters for enp6s0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	2
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	2