Hide Forgot
Description of problem: We will backport rx batching for tun to 7.4. Which can batch several packets before submitting them to kernel. This could be configured through ethtool -C $tap rx-frames N while N is the maximum numbers of packets that could be batched by tun. Default value is 0, maximum value is 64. Configurations that has long xmit path may benefit from this because of the improvement of cache utilization. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
V2 posted upstream: https://www.redhat.com/archives/libvir-list/2017-April/msg00939.html
Fixed upstream with commits v3.2.0-206-g652ef9bc8c72..v3.2.0-208-gfcef44728dff: commit fcef44728dff9cb708d00d17f5e0b44aa513f27b Author: Martin Kletzander <mkletzan> Date: 2017-04-07 17:54:12 +0200 Set coalesce settings for domain interfaces commit 523c9960621eaf307ae8d4ae2735fb66f89d5634 Author: Martin Kletzander <mkletzan> Date: 2017-04-07 17:46:32 +0200 conf, docs: Add support for coalesce setting(s) commit 652ef9bc8c72a7118436a8c95c5742ec6a6d12a3 Author: Martin Kletzander <mkletzan> Date: 2017-04-07 17:38:06 +0200 util: Add virNetDevSetCoalesce function
I would probably say "Added support for rx batching on tun/tap devices" for the first line of the Doc Text, so I fixed that. But other than that it looks good to me. Thanks.
I have test it on a host with below packages: libvirt-3.2.0-7.el7.x86_64 kernel-3.10.0-671.el7.x86_64 To check if I understand the bug, I have done the performance test with pktgen by below steps, please help to check if I got the point, thank you in advance. 1. edit 2 guests without Coalesce settings, then start the 2 guests. <interface type='network'> <source network='default'/> <model type='virtio'/> </interface> 2. on vm1 and vm2, load the pktgen module # modprobe pktgen # modinfo pktgen filename: /lib/modules/3.10.0-675.el7.x86_64/kernel/net/core/pktgen.ko.xz version: 2.74 license: GPL description: Packet Generator tool author: Robert Olsson <robert.olsson.se> rhelversion: 7.4 srcversion: BF9522B0DBF61EAB7EF44C7 depends: intree: Y vermagic: 3.10.0-675.el7.x86_64 SMP mod_unload modversions signer: Red Hat Enterprise Linux kernel signing key sig_key: 84:DD:2F:C3:34:C0:13:EA:D7:FA:0F:4E:EF:0B:BB:A3:A4:96:46:26 sig_hashalgo: sha256 parm: pg_count_d:Default number of packets to inject (int) parm: pg_delay_d:Default delay between packets (nanoseconds) (int) parm: pg_clone_skb_d:Default number of copies of the same packet (int) parm: debug:Enable debugging of pktgen module (int) 3. on vm1, execute the test script, after about 1 minute, stop the script, at the same time, monitor on host # cat /proc/net/pktgen/eth0 Params: count 0 min_pkt_size: 60 max_pkt_size: 60 frags: 0 delay: 0 clone_skb: 1000 ifname: eth0 flows: 0 flowlen: 0 queue_map_min: 0 queue_map_max: 0 dst_min: 192.168.122.48 dst_max: src_min: src_max: src_mac: 52:54:00:8e:31:1e dst_mac: 52:54:00:6e:f4:f1 udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9 src_mac_count: 0 dst_mac_count: 0 Flags: Current: pkts-sofar: 58636535 errors: 0 started: 4589110046us stopped: 4796286358us idle: 77600us seq_num: 58636536 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 cur_saddr: 192.168.122.77 cur_daddr: 192.168.122.48 cur_udp_dst: 9 cur_udp_src: 9 cur_queue_map: 0 flows: 0 Result: OK: 207176311(c207098711+d77600) usec, 58636535 (60byte,0frags) 283027pps 135Mb/sec (135852960bps) errors: 0 # sar -n DEV 4 Linux 3.10.0-671.el7.x86_64 (server74) 06/06/2017 _x86_64_ (4 CPU) 06:52:50 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 06:52:55 PM enp0s25 1.00 0.40 0.12 0.20 0.00 0.00 0.20 06:52:55 PM virbr3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:52:55 PM vnet1 1.00 296917.80 0.09 17397.52 0.00 0.00 0.00 06:52:55 PM virbr3-nic 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:52:55 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:52:55 PM virbr0-nic 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:52:55 PM virbr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:52:55 PM vnet0 298327.80 1.40 17480.14 0.11 0.00 0.00 0.00 4. destroy the 2 vm2, and add below content in the interface xml, then start rhe 2 guests. <coalesce> <rx> <frames max='128'/> </rx> </coalesce> # ethtool -c vnet0 | grep rx-frames: rx-frames: 64 # ethtool -c vnet1 | grep rx-frames: rx-frames: 64 5. on vm1, execute the test script then check the result, and monitor on host # cat /proc/net/pktgen/eth0 Params: count 0 min_pkt_size: 60 max_pkt_size: 60 frags: 0 delay: 0 clone_skb: 1000 ifname: eth0 flows: 0 flowlen: 0 queue_map_min: 0 queue_map_max: 0 dst_min: 192.168.122.48 dst_max: src_min: src_max: src_mac: 52:54:00:8e:31:1e dst_mac: 52:54:00:6e:f4:f1 udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9 src_mac_count: 0 dst_mac_count: 0 Flags: Current: pkts-sofar: 36929996 errors: 0 started: 129442039us stopped: 243491182us idle: 106549us seq_num: 36929997 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 cur_saddr: 192.168.122.77 cur_daddr: 192.168.122.48 cur_udp_dst: 9 cur_udp_src: 9 cur_queue_map: 0 flows: 0 Result: OK: 114049143(c113942593+d106549) usec, 36929996 (60byte,0frags) 323807pps 155Mb/sec (155427360bps) errors: 0 # sar -n DEV 4 Linux 3.10.0-671.el7.x86_64 (server74) 06/06/2017 _x86_64_ (4 CPU) 07:04:02 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 07:04:06 PM enp0s25 0.75 0.50 0.05 0.25 0.00 0.00 0.25 07:04:06 PM virbr3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:04:06 PM vnet1 1.00 332754.50 0.09 19497.33 0.00 0.00 0.00 07:04:06 PM virbr3-nic 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:04:06 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:04:06 PM virbr0-nic 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:04:06 PM virbr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:04:06 PM vnet0 333396.50 1.50 19534.95 0.11 0.00 0.00 0.00 After this test, got the result: rx-frames pkts/s -----------+--------------+ 0 283027pps -----------+--------------+ 1 296906pps -----------+--------------+ 4 313498pps -----------+--------------+ 16 322192pps -----------+--------------+ 64 323807pps -----------+--------------+ 128 331654pps -> the rx-frames of vnet0 and vnet1 is 64 -----------+--------------+ 256 323908pps -> the rx-frames of vnet0 and vnet1 is 64 -----------+--------------+ once the "frames max" > 64 setting in the guest's xml, the rx_frames of the tap device will keep as 64. If this is expected? # virsh dumpxml rhel7 | grep /coalesce -B4 <coalesce> <rx> <frames max='65'/> </rx> </coalesce> # ethtool -c vnet0 | grep rx-frames: rx-frames: 64
(In reply to yalzhang from comment #14) I can't reply for kernel, but from libvirt's POV it's expected because we set the number and then retrieve it back so that we report the right information in the XML (when you do dumpxml on a running domain). So the limitation comes from the kernel. The same happens if you set higher number using 'ethtool -C'.
(In reply to Martin Kletzander from comment #15) > (In reply to yalzhang from comment #14) > I can't reply for kernel, but from libvirt's POV it's expected because we > set the number and then retrieve it back so that we report the right > information in the XML (when you do dumpxml on a running domain). So the > limitation comes from the kernel. The same happens if you set higher number > using 'ethtool -C'. Yes, we limit it to 64 for preventing bh from being disabled too long. So it's expected. Thanks
Test on below packages: libvirt-3.2.0-7.el7.x86_64 qemu-kvm-rhev-2.9.0-8.el7.x86_64 kernel-3.10.0-671.el7.x86_64 scenarios: 1. set coalesce in interface type='network' Q1: for hostdev and macvtap network, coalesce settings exists in the guest xml, but makes no sense, if libvirt plan do the check and reject such setting? 2. set coalesce in interface type='bridge' ---> PASS 3. hotplug an interface device with coalesce setting ---> PASS 4. boundary test and illegal value test,the legal value is 0～4294967295 -->PASS 5. update the coalesce setting on the fly by update-device ---> FAIL 6. model type other than virtio, such as rtl8139, e1000, e1000e, it is ok to set coalesce, but make no sense. Q2: if libvirt plan to do the check and reject such setting? 7. virtio with driver name=qemu, it is ok to set coalesce, but make no sense as it needs the vhost as backend. Q3: if libvirt plan to do the check and reject such setting? Scenario 1: 1. set coalesce in interface=network with macvtap network # virsh net-dumpxml direct-macvtap <network> <name>direct-macvtap</name> <uuid>06848478-25db-4edc-8a85-27a0778b3f50</uuid> <forward dev='enp0s25' mode='bridge'> <interface dev='enp0s25'/> </forward> </network> # virsh start rhel7.4 Domain rhel7.4 started # virsh dumpxml rhel7.4 | grep /interface -B12 <interface type='direct'> <mac address='52:54:00:f1:e7:e1'/> <source network='direct-macvtap' dev='enp0s25' mode='bridge'/> <target dev='macvtap0'/> <model type='virtio'/> <coalesce> <rx> <frames max='64'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # ethtool -c macvtap0 Coalesce parameters for macvtap0: Cannot get device coalesce settings: Operation not supported Scenario 5: 1. start a guest with coalesce setting # virsh dumpxml rhel7.4 | grep /interface -B12 <interface type='network'> <mac address='52:54:00:2c:24:22'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='64'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # cat interface1.xml <interface type='network'> <mac address='52:54:00:2c:24:22'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='32'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> 2. update the coalesce setting by update-device, the command return succeed, but no change. # virsh update-device rhel7.4 interface1.xml Device updated successfully # virsh dumpxml rhel7.4 | grep /interface -B12 <interface type='network'> <mac address='52:54:00:2c:24:22'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='64'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # ethtool -c vnet0 | grep rx-frames: rx-frames: 64
one more scenario: 8. migration from rhel7.4 -> rhel7.4 ---> PASS migration from rhel7.4 -> rhel7.3 ---> PASS, the guest's xml will ignore the coalesce setting after migrated to a libvirt version which do not support it.
A1: No, we did not reject it by decision and now we cannot reject this anymore since there were two releases already and that would break compatibility. A2: Same here. It would be logic-duplicating from somewhere else. Does the live XML contain the data as well? This could be the only bug to be fixed. A3: We cannot change it when parsing, but we can check for it when starting the domain. We can do that actually for the previous cases as well, but please, if that is an issue for you, create another bug for it (and right away assign it to me) so that we don't postpone this fixed one due to some error messages raised when starting a domain. Feel free to mention all three issues there in that case.
Hi Martin, thank you for your confirmation. I have submitted Bug 1460862 for above 3 scenarios. For the scenario 5 in comment #17, update-device take no effect but return success. I think it is an issue need to be fixed, please check, thank you~
I will send additional patch for that and I'll see if the additional checks will be possible to put in as well.
Fixed upstream with: commit 307a205e25ad7db7c895c42ab2e8f59f3839c058 (origin/master, origin/HEAD) Author: Martin Kletzander <mkletzan> Date: 2017-06-15 14:22:26 +0200 qemu: Allow live-updates of coalesce settings
Test on libvirt-3.2.0-11.el7.x86_64, still some issues, see step 2. 1. # virsh dumpxml rhel7.4 | grep /interface -B12 <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='32'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # cat net.xml <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='64'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> 2. try to update rx_frames from current 32 to 64, failed. # virsh update-device rhel7.4 net.xml Device updated successfully # virsh dumpxml rhel7.4 | grep /interface -B12 <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='32'/> ==============> no changes </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # ethtool -c vnet0 | grep rx-frames: rx-frames: 32 =================> no changes 3. try to update rx_frames from current 32 to unconfigured 0, succeed # cat net0.xml <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # virsh update-device rhel7.4 net0.xml Device updated successfully # virsh dumpxml rhel7.4 | grep /interface -B7 <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> ======> no coalesce settings <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # ethtool -c vnet0 | grep rx-frames: rx-frames: 0 4. try to update the rx-frames from unconfigured 0 to 16, succeed # cat net1.xml <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <coalesce> <rx> <frames max='16'/> </rx> </coalesce> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # virsh update-device rhel7.4 net1.xml Device updated successfully # virsh dumpxml rhel7.4 | grep /interface -B12 <interface type='network'> <mac address='52:54:00:04:b2:d8'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <coalesce> <rx> <frames max='16'/> </rx> </coalesce> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # ethtool -c vnet0 | grep rx-frames: rx-frames: 16
Would you like to fix it in the coming build for rhel7.4 or just verify this bug then track the tiny issue in a new bug? Sorry for the late reply, I would test the patch few days ago...
No problem, thanks for that, that's definitely a bug and my fault. Yet another fix is this one: commit ff7bae6e4fb74a52239d53af3672900c69801508 Author: Martin Kletzander <mkletzan> Date: 2017-06-21 09:00:58 +0200 qemu: Change coalesce settings on hotplug when they are different
Test on libvirt-3.2.0-14.el7.x86_64 with the scenarios in comment 24, comment 17, the result is as expected, verify this bug.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846