Description of problem: Hot plug vhost-user with 2 queues, fail with error: error: Failed to attach device from /tmp/rhel8.3_nic2.xml error: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter type for 'queues', expected: integer Version-Release number of selected component (if applicable): qemu-kvm-5.0.0-0.scrmod+el8.3.0+6495+1936fa11.wrb200506.x86_64 4.18.0-195.el8.x86_64 python3-libvirt-6.2.0-1.module+el8.3.0+6371+f67a7ce3.x86_64 openvswitch2.11-2.11.0-53.20200327gita4efc59.el8fdp.x86_64 tuned-2.13.0-6.el8.noarch How reproducible: 100% Steps to Reproduce: 1. Boot ovs with vhostuserclient ports, refer to [1] 2. Boot VM without any vhost-user device 3. Hot plug vhost-user with 2 queues. Fail # cat /tmp/rhel8.3_nic2.xml <interface type="vhostuser"> <mac address="18:66:da:5f:dd:23" /> <source mode="server" path="/tmp/vhostuser1.sock" type="unix" /> <model type="virtio" /> <driver ats="on" iommu="on" name="vhost" queues="2" rx_queue_size="1024" /> <address bus="0x7" domain="0x0000" function="0x0" slot="0x00" type="pci" /> </interface> # /bin/virsh attach-device rhel8.3 /tmp/rhel8.3_nic2.xml error: Failed to attach device from /tmp/rhel8.3_nic2.xml error: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter type for 'queues', expected: integer Actual results: Hot plug vhost-user with 2 queues fail. Expected results: Hot plug vhost-user with 2 queues should work. Additional info: 1. Hot plug vhost-user with single queue works well. # cat /tmp/rhel8.3_nic1.xml <interface type="vhostuser"> <mac address="18:66:da:5f:dd:22" /> <source mode="server" path="/tmp/vhostuser0.sock" type="unix" /> <model type="virtio" /> <driver ats="on" iommu="on" name="vhost" queues="2" rx_queue_size="1024" /> <address bus="0x6" domain="0x0000" function="0x0" slot="0x00" type="pci" /> </interface> # /bin/virsh attach-device rhel8.3 /tmp/rhel8.3_nic1.xml Device attached successfully Reference [1] # ovs-vsctl show c587e06a-2135-4480-b6fb-548fc3bf39bb Bridge "ovsbr1" Port "ovsbr1" Interface "ovsbr1" type: internal Port "vhost-user1" Interface "vhost-user1" type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser1.sock"} Port "dpdk1" Interface "dpdk1" type: dpdk options: {dpdk-devargs="0000:5e:00.1", n_rxq="2", n_txq="2"} Bridge "ovsbr0" Port "dpdk0" Interface "dpdk0" type: dpdk options: {dpdk-devargs="0000:5e:00.0", n_rxq="2", n_txq="2"} Port "vhost-user0" Interface "vhost-user0" type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser0.sock"} Port "ovsbr0" Interface "ovsbr0" type: internal 2. This should be regression qemu bug. Below version works well. qemu-kvm-4.2.0-19.module+el8.3.0+6371+f67a7ce3.x86_64 python3-libvirt-6.2.0-1.module+el8.3.0+6371+f67a7ce3.x86_64
Created attachment 1686072 [details] VM XML without vhost-user NICs
The qemu change was intentional, see: commit db2a380c84574d8c76d7193b8af8535234fe5156 Author: Eric Blake <eblake> Date: Tue Mar 17 15:17:10 2020 -0500 net: Complete qapi-fication of netdev_add We've had all the required pieces for doing a type-safe representation of netdev_add as a flat union for quite some time now (since 0e55c381f6 in v2.7.0, released in 2016), but did not make the final switch to using it because of concern about whether a command-line regression in accepting "1" in place of 1 for integer arguments would be problematic. ... But this means that libvirt was mistakenly relying on the non-strict string passing instead of type-safe passing. I think the correct fix is for libvirt to scrub all netdev QMP calls to be type-correct, rather than adding back-compat parser hacks back into qemu.
We couldn't really have known what the correct format is, but we definitely can fix it rather easily.
*** Bug 1835312 has been marked as a duplicate of this bug. ***
Fixed upstream by the following patchset: 576746d412 qemuxml2argvtest: Add QAPI/QMP schema validation for -blockdev and -netdev b6246d9320 testutilsqemuschema: Allow loading non-latest schema 18e9806c64 testQEMUSchemaLoad: Rename to testQEMUSchemaLoadLatest 78d30aa0bf qemu: Prepare for testing of 'netdev_add' props via qemuxml2argvtest 529ad62c0d qemuMonitorAddNetdev: Convert to the native JSON props object 11758f9245 qemuBuildChannelGuestfwdNetdevProps: Convert to generating JSON props da820e1c22 virQEMUBuildNetdevCommandlineFromJSON: Prepare for quirky 'guestfwd' b6738ffc9f qemu: command: Generate -netdev command line via JSON->cmdline conversion 486bfba8b2 util: virqemu: Introduce virQEMUBuildNetdevCommandlineFromJSON eb56a98672 qemuMonitorJSON(Add|Remove)Netdev: Refactor cleanup 973e2d9b9f testCompareXMLToArgv: Split out preparation and command formatting 9c9732d03a util: json: Introduce virJSONValueObjectAppendStringPrintf 003f063dd6 virCommand: Introduce virCommandGetArgList 3cc35bf88e virQEMUBuildCommandLineJSON: Add possibility for using 'on/off' instead of 'yes/no' 220751091f virQEMUBuildCommandLineJSON: Allow skipping certain keys 54e4328295 qemuBuildHostNetStr: Stop using 'ipv6-net' convenience argument 202bb53cc2 qemuBuildChannelChrDeviceStr: Remove formatting of properties for -netdev 63e5124ac1 qemuBuildChannelsCommandLine: Extract common formatting of 'chardev' 63fc99402e qemuBuildChannelsCommandLine: Use typecasted switch for channel type 8f2d30e9bc qemuMonitorJSONParseKeywords: remove constant argument 26c8b4f1fc qemu: domain: Forbid unsupported 'tftp' protocol and handle tests de8216af13 testQEMUSchemaValidate*: Reject usage of fields with 'deprecated' set add3607ea8 qemumonitorjsontest: Mark recently deprecated migration command in our tests 6c93020fcb qemumonitorjsontest: Allow use of deprecated 'cpu-add' and 'change' command 4bad0d913a qemumonitorjsontest: Allow use of deprecated 'query-cpus' 5703b48103 testQemuMonitorJSONqemuMonitorJSONQueryCPUs: Split off test for query-cpus-fast 199cd7a37e qemumonitorjsontest: Add infrastructure for generated tests of deprecated commands 2700779f6f testQemuHotplugCpuPrepare: Allow deprecated commands for non-modern cpu hotplug test 4fcea23e6c testQEMUSchemaValidate(Command): Allow skipping validation of deprecated fields 2c44ea900e testutilsqemuschema: Pass in 'schema' and 'debug' variables to workers in a struct 47bc49879f testutilsqemuschema: Use automatic variable clearing where possible 1ef3d0fc97 qemumonitortestutils: Introduce qemuMonitorTestSkipDeprecatedValidation 7014d2ef14 qemuMonitorTestProcessCommandDefaultValidate: Clean up return value use 7f350f5260 qemuMonitorTestProcessCommandDefaultValidate: Use testQEMUSchemaValidateCommand 23bd18dc75 testutilsqemuschema: Introduce testQEMUSchemaValidateCommand 3b78d28974 testQemuHotplugCpuPrepare: Allow unused monitor commands only on failure
Update: In my testing, this issue is gone with libvirt-6.4.0-1.scrmod+el8.3.0+6893+614302c0.x86_64. Hot plug/unplug vhost-user 2 queues works very well, no any error. Test result: Testcase: vhostuser_hotplug_nonrt_server_iommu Packets_loss Frame_Size Run_No Throughput Avg_Throughput 0 64 0 21.307348 21.307348 0 64 0 21.307348 21.307348 Test versions: 4.18.0-211.el8.x86_64 qemu-kvm-5.0.0-0.scrmod+el8.3.0+6893+614302c0.wrb200603.x86_64 tuned-2.13.0-6.el8.noarch libvirt-6.4.0-1.scrmod+el8.3.0+6893+614302c0.x86_64 python3-libvirt-6.3.0-1.module+el8.3.0+6485+695fc960.x86_64 openvswitch2.13-2.13.0-35.el8fdp.x86_64 dpdk-19.11.2-1.el8.x86_64
Reproduce the bug on: libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64 qemu-kvm-5.0.0-0.module+el8.3.0+6612+6b86f0c9.x86_64 # cat interface.xml <interface type="vhostuser"> <mac address="18:66:da:5f:dd:23" /> <source mode="client" path="/var/run/openvswitch/vhost-user2" type="unix" /> <model type="virtio" /> <driver name="vhost" queues="4" rx_queue_size="1024" /> </interface> # virsh attach-device rhel interface.xml error: Failed to attach device from interface.xml error: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter type for 'queues', expected: integer # virsh attach-interface rhel network default --model virtio error: Failed to attach interface error: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter type for 'vhost', expected: boolean Update libvirt and qemu to latest version and re-test again, the result is as expected. libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64 qemu-kvm-5.1.0-2.module+el8.3.0+7652+b30e6901.x86_64 # virsh attach-interface rhel network default --model virtio Interface attached successfully # virsh attach-device rhel interface.xml Device attached successfully check the live xml, the interfaces are attached successfully, also check the result of auto jobs, the issue is fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137