Description of problem: When virtio_net driver is enabled the KVM guest is dropping RX packets at a high rate. When disabling the virtio_net by addig in the Guest vm xml the stanza <drive name = 'qemu'/> there is few dropped RX packet at start, but no increase Version-Release number of selected component (if applicable): host side: qemu-kvm-2.4.1-5.fc23.x86_64 libvirt-daemon-kvm-1.2.18.2-1.fc23.x86_64 qemu-common-2.4.1-5.fc23.x86_64 qemu-kvm-2.4.1-5.fc23.x86_64 qemu-img-2.4.1-5.fc23.x86_64 ipxe-roms-qemu-20150407-3.gitdc795b9f.fc23.noarch libvirt-daemon-driver-qemu-1.2.18.2-1.fc23.x86_64 qemu-system-x86-2.4.1-5.fc23.x86_64 libvirt-daemon-driver-nodedev-1.2.18.2-1.fc23.x86_64 libvirt-daemon-driver-storage-1.2.18.2-1.fc23.x86_64 libvirt-daemon-config-network-1.2.18.2-1.fc23.x86_64 libvirt-daemon-1.2.18.2-1.fc23.x86_64 libvirt-daemon-driver-secret-1.2.18.2-1.fc23.x86_64 libvirt-daemon-driver-network-1.2.18.2-1.fc23.x86_64 libvirt-daemon-driver-nwfilter-1.2.18.2-1.fc23.x86_64 libvirt-daemon-driver-qemu-1.2.18.2-1.fc23.x86_64 libvirt-daemon-kvm-1.2.18.2-1.fc23.x86_64 libvirt-client-1.2.18.2-1.fc23.x86_64 libvirt-daemon-driver-interface-1.2.18.2-1.fc23.x86_64 on Guest side: qemu-guest-agent-2.3.1-10.fc22.x86_64 How reproducible: - Setup KVM/QEMU/LIBVIRT - Create a Guest VM usig the Nat subnet - Start the VM guest - and observe that the VM guest ifconfig eth0 reports RX packet dropped Steps to Reproduce: 1.create the nat-subnet with libvirt <network connections='5'> <name>nat-internet</name> <uuid>4cff86b1-8e63-40be-ac9c-d3dcd405a9d3</uuid> <forward dev='eth0' mode='nat'> <nat> <port start='1024' end='65535'/> </nat> <interface dev='eth0'/> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:e4:ec:1b'/> <domain name='nat-internet'/> <ip address='192.168.100.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.100.128' end='192.168.100.254'/> </dhcp> </ip> </network> 2.Create a VM guest , here after is the dumpxml <domain type='kvm' id='4'> <name>Network</name> <uuid>006ec4e9-028c-4fef-94ec-4e9efbab61ff</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type> <kernel>/var/lib/libvirt/boot/vmlinuz</kernel> <initrd>/var/lib/libvirt/boot/initramfs.img</initrd> <cmdline>root=/dev/vda selinux=0 audit=0 console=ttyS0 nosplash quiet</cmdline> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/vault-storage/network-root'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/vault-storage/network-bootswap'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:36:ac:80'/> <source network='nat-internet' bridge='virbr1'/> <target dev='vnet9'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/4'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/4'> <source path='/dev/pts/4'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Network.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' port='5903' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </memballoon> </devices> </domain> 3. Start the Guest VM Actual results: ifconfig; sleep 5 ; ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.100.15 netmask 255.255.255.0 broadcast 192.168.100.255 inet6 fe80::5054:ff:fe36:ac80 prefixlen 64 scopeid 0x20<link> ether 52:54:00:36:ac:80 txqueuelen 1000 (Ethernet) RX packets 137073 bytes 137962206 (131.5 MiB) RX errors 0 dropped 82696 overruns 0 frame 0 TX packets 47716 bytes 3261557 (3.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.100.15 netmask 255.255.255.0 broadcast 192.168.100.255 inet6 fe80::5054:ff:fe36:ac80 prefixlen 64 scopeid 0x20<link> ether 52:54:00:36:ac:80 txqueuelen 1000 (Ethernet) RX packets 137101 bytes 137963956 (131.5 MiB) RX errors 0 dropped 82715 overruns 0 frame 0 TX packets 47722 bytes 3263161 (3.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Expected results: ifconfig should not report any RX dropped ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.100.15 netmask 255.255.255.0 broadcast 192.168.100.255 inet6 fe80::5054:ff:fe36:ac80 prefixlen 64 scopeid 0x20<link> ether 52:54:00:36:ac:80 txqueuelen 1000 (Ethernet) RX packets 137073 bytes 137962206 (131.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 47716 bytes 3261557 (3.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Additional info:
Additionally increasing the ring buffer is not possible # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 256 RX Mini: 0 RX Jumbo: 0 TX: 256 Current hardware settings: RX: 256 RX Mini: 0 RX Jumbo: 0 TX: 256 ethtool -G eth0 rx 512 Cannot set device ring parameters: Operation not supported
I also found a similar one which has been closed #709551 (https://bugzilla.redhat.com/show_bug.cgi?id=709551 )
Thanks for the report. Please provide /var/log/libvirt/qemu/YOUR-VM-NAME.log as well What host kernel version are you using? Try grabbing the latest with sudo dnf --enablerepo=updates-testing update kernel reboot into that version and see if it still reproduces.
Created attachment 1118824 [details] /var/log/libvirt/qemu/guest-vm.log As requested find here attached the Guest-vm.log file from/var/log/libvirt/qemu/
Kernel is kernel-4.3.3-301.fc23 on both VM and Host I however since I have log the case, did further investigation, and I have been able to identified what those dropped packets (on the virtio-net) are. With tcpdump, These are the every 2 seconds STP from the virtual switch for the nat-network. What you have here after is on the Guest VM , an ifconfig done twice with 5 seconds in between, and then an tcpdump on the eth0 interface. In order to avoid creating noise, I have removed most of the network daemons ... and I got connected to the VMv from the Host with the virsh console command Interestingly,I have done the same on a VM where I have disabled the virtio-net driver by adding the stanza <driver name= 'qemu'/> in the VMguest, and while I do observe not increase in the RX dropped packet, the tcpdump command still see the STP packets. It looks like in one case the dropped packets get increment, while in the other nop ! [root@network ~]# ifconfig ; sleep 5 ; ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.100.15 netmask 255.255.255.0 broadcast 192.168.100.255 inet6 fe80::5054:ff:fe36:ac80 prefixlen 64 scopeid 0x20<link> ether 52:54:00:36:ac:80 txqueuelen 1000 (Ethernet) RX packets 2113 bytes 235690 (230.1 KiB) RX errors 0 dropped 560 overruns 0 frame 0 TX packets 1205 bytes 153488 (149.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 4 bytes 340 (340.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4 bytes 340 (340.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.100.15 netmask 255.255.255.0 broadcast 192.168.100.255 inet6 fe80::5054:ff:fe36:ac80 prefixlen 64 scopeid 0x20<link> ether 52:54:00:36:ac:80 txqueuelen 1000 (Ethernet) RX packets 2116 bytes 235846 (230.3 KiB) RX errors 0 dropped 563 overruns 0 frame 0 TX packets 1205 bytes 153488 (149.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 4 bytes 340 (340.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4 bytes 340 (340.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tcpdump -i eth0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:51:49.960417 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:51:51.960470 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:51:53.960455 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:51:55.960465 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:51:57.960476 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:51:59.960465 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:52:01.960497 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35 22:52:03.960502 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e4:ec:1b.8003, length 35
<driver name='qemu'/> will turn off vhost-net, so that's the differentiating bit. mst, do you know how vhost vs non-vhost handles stp packets? is dropping them a problem?
@Cole for your question to mst I understand, but does it make sense that teh STP packet are going to the guest , as they should be only visible on the virtual switch, no ?
(In reply to Patrick Pichon from comment #7) > @Cole for your question to mst I understand, but does it make sense that teh > STP packet are going to the guest , as they should be only visible on the > virtual switch, no ? honestly not sure if the STP behavior is correct or not...
Don't see why won't STP packets go to guests. I didn't understand the comment above. I don't expect vhost versus non vhost to differ in handling stp.
I don't to whom the commennt #9 is for, but for me as the originator of the issue, I don't expect to see in the iconfig and netstat information about dropped packet due to the STP packets reaching the vhost. In comparaison when I have a physical host connected to a physical switch, I do not see the dropped packet of the physical host impacted by STP packet
In addition to comment #10, why do we have a different behaviour when we turn of vhost-net ?
This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Patrick, do you see this on fedora 24 or fedora 25?
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.