Hide Forgot
Description of problem: there is a problem with RHEL6 as a virtualization guest under Microsoft Hyper-V which is 100% reproducible for them: when irqbalance is enabled in the guest, its networking stops shortly after the network service is started. Version-Release number of selected component (if applicable): Guest: Red Hat Enterprise Linux 6 Host: Windows Server 2008 R2 How reproducible: 100% reproducible Steps to Reproduce: Install RHEL6 in Hyper-V with 2 or more vCPUs and start the irqbalance daemon. Actual results: I did the tests using network legacy drivers and I have a reproducer behavior in my LAB. I got following errors after start the irqbalance: --snip-- ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available --/snip-- Probably when the irqbalance started the network interruptions doesn't reach correctly the Hypervisor, then the buffer fills. ==> irqbalance stopped: [asilva@localhost ~]$ ping www.redhat.com PING e86.b.akamaiedge.net (96.6.144.112) 56(84) bytes of data. 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=1 ttl=55 time=124 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=2 ttl=55 time=123 ms ------ 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=3 ttl=55 time=126 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=4 ttl=55 time=131 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=5 ttl=55 time=125 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=6 ttl=55 time=125 ms ^C --- e86.b.akamaiedge.net ping statistics --- 96 packets transmitted, 96 received, 0% packet loss, time 95725ms ==> Stating irqbalance: [root@localhost ~]# service irqbalance start Starting irqbalance: [ OK ] [asilva@localhost ~]$ ping www.redhat.com PING e86.b.akamaiedge.net (96.6.144.112) 56(84) bytes of data. 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=1 ttl=55 time=123 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=2 ttl=55 time=260 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=3 ttl=55 time=137 ms 64 bytes from a96-6-144-112.deploy.akamaitechnologies.com (96.6.144.112): icmp_seq=4 ttl=55 time=126 ms ping: sendmsg: No buffer space available ^C --- e86.b.akamaiedge.net ping statistics --- 22 packets transmitted, 8 received, 63% packet loss, time 40614ms rtt min/avg/max/mdev = 123.494/172.594/310.905/67.992 ms ==> Monitoring interruptions from eth0 to vCPU0: [root@localhost ~]# while [ 1 ] ; do grep eth0 /proc/interrupts >> test; sleep 3; done ^C [root@localhost ~]# cat test 9: 1631 0 IO-APIC-fasteoi acpi, eth0 9: 1641 0 IO-APIC-fasteoi acpi, eth0 9: 1651 0 IO-APIC-fasteoi acpi, eth0 9: 1663 0 IO-APIC-fasteoi acpi, eth0 9: 1694 0 IO-APIC-fasteoi acpi, eth0 9: 1697 0 IO-APIC-fasteoi acpi, eth0 9: 1705 0 IO-APIC-fasteoi acpi, eth0 9: 1711 0 IO-APIC-fasteoi acpi, eth0 9: 1720 0 IO-APIC-fasteoi acpi, eth0 9: 1726 0 IO-APIC-fasteoi acpi, eth0 9: 1784 0 IO-APIC-fasteoi acpi, eth0 9: 1791 0 IO-APIC-fasteoi acpi, eth0 9: 1799 0 IO-APIC-fasteoi acpi, eth0 9: 1808 0 IO-APIC-fasteoi acpi, eth0 9: 1814 0 IO-APIC-fasteoi acpi, eth0 9: 1821 0 IO-APIC-fasteoi acpi, eth0 9: 1827 0 IO-APIC-fasteoi acpi, eth0 9: 1834 0 IO-APIC-fasteoi acpi, eth0 9: 1843 0 IO-APIC-fasteoi acpi, eth0 9: 1849 0 IO-APIC-fasteoi acpi, eth0 9: 1855 0 IO-APIC-fasteoi acpi, eth0 9: 1862 0 IO-APIC-fasteoi acpi, eth0 9: 1869 0 IO-APIC-fasteoi acpi, eth0 9: 1877 0 IO-APIC-fasteoi acpi, eth0 9: 1883 0 IO-APIC-fasteoi acpi, eth0 9: 1889 0 IO-APIC-fasteoi acpi, eth0 9: 1909 0 IO-APIC-fasteoi acpi, eth0 9: 1916 0 IO-APIC-fasteoi acpi, eth0 9: 1920 0 IO-APIC-fasteoi acpi, eth0 9: 1922 0 IO-APIC-fasteoi acpi, eth0 9: 1924 0 IO-APIC-fasteoi acpi, eth0 9: 1925 0 IO-APIC-fasteoi acpi, eth0 9: 1935 0 IO-APIC-fasteoi acpi, eth0 9: 1943 0 IO-APIC-fasteoi acpi, eth0 ==> from here (irqbalance started) the processor no longer responds to interrupts 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 9: 1948 0 IO-APIC-fasteoi acpi, eth0 Expected results: The irqbalance work correctly. Additional info: I also tested RHEL 6 with irqbalance on RHEV 2.2 and I didn't get issues with the network.
Neil, any guess here?
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. This request has been proposed for the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative.
Anton, sorry its taken me so long to see this. In regards to your question, I'm not entirely sure whats going on here. What I can say definitively is that its not (strictly speaking) an irqbalance problem. The irqbalance daemon moves processor affinity around by writing to /proc/irq/<irqn>/smp_affinity. So even if you turned off irqbalance, you could (or should), if such affinity movements are the root of this problem, be able to trigger the issue with a manual echo of an affinity mask to that same file. As such, it would seem to me to be the kernels job to prevent such malicious behavior. Comparing hyper-v and xen, it would seem to me that the hyper-v kernel code needs to remap the irqs that get requested in the guest to a new irq_chip structure so that it has control over the set_affinity method when user space changes affinity (see bind_evtchn_to_irq for an example). Alternatively, hyper-v could add some arch specific code to flag all irq chips as not supporting affinity movement, so that userspace can't make any changes. Or, the customer could use kvm, which already has this hashed out in qemu.
Closing the bug as WONTFIX, that stands for unable to fix here. This is the arch-specific issue that must be addressed in hyper-v kernel.
Still same issue with 6.2.. Will this be fixed?
This issue is discussed in <https://access.redhat.com/kb/docs/DOC-49132>, "Network stops functioning on RHEL6 guest under Hyper-V when the irqbalance service is started". The Hyper-V hypervisor is a product of Microsoft. Fixing its limitations is outside Red Hat's scope. If you would like to see this limitation addressed, please contact your Microsoft support representative.
I did thought that virtualization works this way: 1) Guest OSs are "unaware" of being virtualized. 2) Hypervisor is called only when needed - - -> facilitate simultaneous operation of OSs and protect access to SHARED system resources.. The link above seems useless, how can you adress hyper-v as being the root cause here? I reproduced the same issue with red hat el 6.1 / 6.2 with only 1vcpu..
From my point of view, the /etc/rc.d/init.d/irqbalance should have a section like this in the beginning: if [ -x /usr/sbin/virt-what ]; then if [ "$(/usr/sbin/virt-what)" = "hyperv" ]; then exit 1 fi fi
I have opened case 00598829 in the Red Hat customer portal to request this workaround or similar or better.
*** Bug 788607 has been marked as a duplicate of this bug. ***
Robert, you have the right idea, but look at the description of the bug - RHEL is the Guest OS here, when we alter irq affinity, we do so without any knoweldge of the fact that we are running under a hyperv hypervisor. It is the responsibility of the hypervisor to trap such affinity changes and prevent them from occuring, if doing so will stop delivery of needed interrupts to the guest. The Hyper-V hypervisor is a microsoft product, hence this is their problem to fix