Bug 1541960
Summary: | set rx_queue_size and tx_queue_size in qemu.conf | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Michal Privoznik <mprivozn> | ||||
Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | ||||
Status: | CLOSED WONTFIX | QA Contact: | yalzhang <yalzhang> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 7.4 | CC: | atelang, berrange, chhu, dyuan, fbaudin, jdenemar, jishao, jkurik, jsuchane, ksundara, mburns, mprivozn, mtessun, rbalakri, rhel-osp-director-maint, skramaja, stephenfin, supadhya, vchundur, xuzhang, yalzhang, yrachman, ysubrama | ||||
Target Milestone: | rc | Keywords: | FutureFeature, Upstream | ||||
Target Release: | --- | ||||||
Hardware: | All | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | 1540525 | Environment: | |||||
Last Closed: | 2018-02-13 12:51:22 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1540525 | ||||||
Attachments: |
|
Description
Michal Privoznik
2018-02-05 10:17:52 UTC
This is already committed upstream: commit 038eb472a0d970a17ccf4343ead0666df5c92f9d Author: Michal Privoznik <mprivozn> AuthorDate: Fri Jan 19 11:34:54 2018 +0100 Commit: Michal Privoznik <mprivozn> CommitDate: Fri Feb 2 07:09:22 2018 +0100 qemu: Expose rx/tx_queue_size in qemu.conf too In 2074ef6cd4a2 and c56cdf259 (and friends) we've added two attributes to virtio NICs: rx_queue_size and tx_queue_size. However, sysadmins might want to set these on per-host basis but don't necessarily have an access to domain XML (e.g. because they are generated by some other app). So let's expose them under qemu.conf (the settings from domain XML still take precedence as they are more specific ones). Signed-off-by: Michal Privoznik <mprivozn> Reviewed-by: John Ferlan <jferlan> v4.0.0-126-g038eb472a Test on below packages, the result is as expected, set the bug to be verified libvirt-3.9.0-12.el7.x86_64 qemu-kvm-rhev-2.10.0-20.el7.x86_64 scenarios: 1. start guest with the setting in qemu.conf; 2. start guest with setting both in qemu.conf and xml; 3. hotplug interface; 4. hotplug device with or without the setting in xml; 5. migration; 6. save -> restore, and managedsave -> start; 7. negative: set invalid value in qemu.conf; and cover interface type as vhostuser, direct and network, hostdev Details: S1: 1. set "rx_queue_size = 1024 tx_queue_size = 512" in qemu.conf, and restart libvirtd 2. set 'rx_queue_size' then start the guest, the qemu cmd and guest xml will auto added 'tx_queue_size' and keep the xml setting. # virsh dumpxml new | grep /interface -B6 <interface type='network'> <mac address='52:54:00:53:cc:3f'/> <source network='default'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> # virsh start new Domain new started # virsh dumpxml new | grep /interface -B7 <mac address='52:54:00:53:cc:3f'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512' tx_queue_size='512'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> qemu cmd: -netdev tap,fds=27:29:30:31:32,id=hostnet0,vhost=on,vhostfds=33:34:35:36:37 -device virtio-net-pci,mq=on,vectors=12,rx_queue_size=512,tx_queue_size=512,netdev=hostnet0,id=net0,mac=52:54:00:53:cc:3f,bus=pci.1,addr=0x0 3. restart libvirtd, then check on guest, the result is as expected # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 512 RX Mini: 0 RX Jumbo: 0 TX: 256 Current hardware settings: RX: 512 RX Mini: 0 RX Jumbo: 0 TX: 256 4. hotplug interface with virtio model type, the 'rx_queue_size' or 'tx_queue_size' will be added # virsh attach-interface new direct enp4s0f0 --model virtio # virsh dumpxml new | grep /interface -B9 ... <interface type='direct'> <mac address='52:54:00:49:9d:65'/> <source dev='enp4s0f0' mode='vepa'/> <target dev='macvtap0'/> <model type='virtio'/> <driver rx_queue_size='1024' tx_queue_size='512'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </interface> 5. for other model types, no 'rx_queue_size' or 'tx_queue_size' added. 6. attach device with 'rx_queue_size' setting # cat interface.xml <interface type='direct'> <mac address='52:54:00:47:11:4d'/> <source network='direct-macvtap' dev='enp4s0f0' mode='bridge'/> <target dev='macvtap2'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512'> </driver> </interface> # virsh attach-device new interface.xml Device attached successfully # virsh dumpxml new | grep /interface -B9 ... <interface type='direct'> <mac address='52:54:00:47:11:4d'/> <source dev='enp4s0f0' mode='bridge'/> <target dev='macvtap1'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512' tx_queue_size='512'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </interface> with macvtap passthrough mode and source dev is vf # cat interface.xml <interface type='direct'> <source dev='p6p1_3' mode='passthrough'/> <model type='virtio'/> </interface> # virsh attach-device rhel interface.xml Device attached successfully after attach: <interface type='direct'> <mac address='52:54:00:cf:33:8d'/> <source dev='p6p1_3' mode='passthrough'/> <target dev='macvtap0'/> <model type='virtio'/> <driver rx_queue_size='1024' tx_queue_size='1024'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </interface> for hostdev interface type # cat interface.xml <interface type='hostdev' managed='yes'> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x04' slot='0x10' function='0x4'/> </source> <model type='virtio'/> ===> this is invalid setting </interface> # virsh attach-device rhel interface.xml Device attached successfully after attach, no rx* or tx* is added, this is expected <interface type='hostdev' managed='yes'> <mac address='52:54:00:df:a1:f9'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x04' slot='0x10' function='0x4'/> </source> <model type='virtio'/> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </interface> S2: Test vhostuser interface type, and do migration, all things work as expected S3: negative test: set invalid value in qemu.conf like "rx_queue_size = 100000 tx_queue_size = 8" # systemctl restart libvirtd start a guest with virtio interface # virsh start new error: Failed to start domain new error: internal error: qemu unexpectedly closed the monitor: 2018-02-08T11:25:56.289230Z qemu-kvm: -device virtio-net-pci,mq=on,vectors=12,rx_queue_size=100000,tx_queue_size=8,netdev=hostnet0,id=net0,mac=52:54:00:53:cc:3f,bus=pci.1,addr=0x0: Parameter 'rx_queue_size' expects uint16_t set as "rx_queue_size = 100 tx_queue_size = 8" # virsh start new error: Failed to start domain new error: internal error: qemu unexpectedly closed the monitor: 2018-02-08T11:27:06.575614Z qemu-kvm: -device virtio-net-pci,mq=on,vectors=12,rx_queue_size=100,tx_queue_size=8,netdev=hostnet0,id=net0,mac=52:54:00:53:cc:3f,bus=pci.1,addr=0x0: Invalid rx_queue_size (= 100), must be a power of 2 between 256 and 1024. Hi Michal, could you please help to check if the error message in step 3 for vhostuser type interface is accepted? As it is not so detailed as other type interfaces. Thank you very much! Test with qemu-kvm-rhev which do not support tx_queue_size # rpm -q libvirt qemu-kvm-rhev libvirt-3.9.0-12.el7.x86_64 qemu-kvm-rhev-2.9.0-16.el7_4.1.x86_64 1. set 'tx_queue_size = 1024 rx_queue_size = 1024' in qemu.conf 2. restart libvirtd 3. start guest # virsh dumpxml new | grep /interface -B6 <interface type='network'> <mac address='52:54:00:53:cc:3f'/> <source network='default'/> <model type='virtio'/> <driver name='vhost' queues='5'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> # virsh start new error: Failed to start domain new error: unsupported configuration: virtio tx_queue_size option is not supported with this QEMU binary for vhostuser type interface: # virsh dumpxml test | grep /interface -B5 <interface type='vhostuser'> <mac address='52:54:00:93:51:dd'/> <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # virsh start test error: Failed to start domain test error: internal error: Error generating NIC -device string (In reply to yalzhang from comment #9) > Hi Michal, could you please help to check if the error message in step 3 for > vhostuser type interface is accepted? As it is not so detailed as other type > interfaces. Thank you very much! > > > Test with qemu-kvm-rhev which do not support tx_queue_size > > # rpm -q libvirt qemu-kvm-rhev > libvirt-3.9.0-12.el7.x86_64 > qemu-kvm-rhev-2.9.0-16.el7_4.1.x86_64 > > 1. set 'tx_queue_size = 1024 rx_queue_size = 1024' in qemu.conf > 2. restart libvirtd > 3. start guest > # virsh dumpxml new | grep /interface -B6 > <interface type='network'> > <mac address='52:54:00:53:cc:3f'/> > <source network='default'/> > <model type='virtio'/> > <driver name='vhost' queues='5'/> > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' > function='0x0'/> > </interface> > > # virsh start new > error: Failed to start domain new > error: unsupported configuration: virtio tx_queue_size option is not > supported with this QEMU binary This is expected. You requested configuration that qemu doesn't support. Instead of ignoring the requested setting silently libvirt errors out loudly. > > for vhostuser type interface: > # virsh dumpxml test | grep /interface -B5 > <interface type='vhostuser'> > <mac address='52:54:00:93:51:dd'/> > <source type='unix' path='/var/run/openvswitch/vhost-user2' > mode='client'/> > <model type='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > > # virsh start test > error: Failed to start domain test > error: internal error: Error generating NIC -device string This is interesting - I'm unable to reproduce. Can you please attach full debug logs? Thanks. Created attachment 1393490 [details] log for comment 10 start vm with vhostuser interface on unsupported qemu Test comment 10 on packages: libvirt-3.9.0-12.el7.x86_64 qemu-kvm-rhev-2.9.0-16.el7_4.1.x86_64 The log shows: # grep error libvirtd.log 2018-02-09 01:52:49.053+0000: 13705: error : qemuBuildNicDevStr:3877 : unsupported configuration: virtio tx_queue_size option is not supported with this QEMU binary 2018-02-09 01:52:49.053+0000: 13705: error : qemuBuildVhostuserCommandLine:8540 : internal error: Error generating NIC -device string one more scenario with unsupported qemu: 1. no tx_queue_size or rx_queue_size setting in qemu.conf 2. start vm with vhostuser interface 3. set tx_queue_size & rx_queue_size in qemu.conf and restart libvirtd 4. attach-device to hotplug vhostuser interface, it will fail with # virsh attach-device test interface.xml error: Failed to attach device from interface.xml error: unsupported configuration: virtio tx_queue_size option is not supported with this QEMU binary It is expected. On further review, the consensus opinion is that this patch is flawed and should never have been added to libvirt. It is going to be reverted in RHEL builds and upstream. Putting back to assigned until the patch is reverted. (In reply to Daniel Berrange from comment #12) > On further review, the consensus opinion is that this patch is flawed and > should never have been added to libvirt. It is going to be reverted in RHEL > builds and upstream. Putting back to assigned until the patch is reverted. Do we have a consensus on the "proper" solution? (In reply to Franck Baudin from comment #13) > Do we have a consensus on the "proper" solution? If a mgmt app wants to have specific values for these settings, it should provide a way to set them itself. Libvirt isn't going to add config options to qemu.conf for arbirary guest XML elements that an mgmt app hasn't got around to supporting itself yet. IOW this is nova's job to solve. Reverted in libvirt-3.9.0-13.el7. |