Created attachment 470943 [details] kernel error message dump before system resets Description of problem: A kernel panic is experienced in the Dom0 when it is under high I/O load such as during system startup when multiple DomU's are starting up at the same time. The issue is seen when running the DomU's on top of DRBD and LVM. The kernel crash is dependent on network driver and the specific details of the cause are related to the scatter/gather features of the NIC are best explained here; http://lists.linbit.com/pipermail/drbd-user/2009-March/011652.html For the purpose of searching this bug a snippet of the text in the above link is given. --- The problem is that when you have scatter/gather enabled, DRBD uses the zero-copy interface to the network stack -- the network stack has the annoying tendency to keep a reference to the pages of requests even after the TCP connection they were sent on is terminated _but_ DRBD makes the assumption that once the TCP connection to the peer is terminated, the pages of all pending requests are no longer referenced by the network stack and it can safely complete any pending I/O requests upwards. --- When turning off the NIC's scatter/gather feature the Dom0 no longer crashes. Further discussion about this issue is on the xen-devel mailing list, with link to thread here; http://xen.1045712.n5.nabble.com/Dom0-panic-on-xen-3-4-3-td2642628.html Version-Release number of selected component (if applicable): We have two server environments running the exact same domU's. The first environment running CentOS 5.3 does NOT crash and has a different network chipset and driver as follows; --- $ uname -a Linux ldx12020 2.6.18-128.4.1.el5xen #1 SMP Tue Aug 4 20:51:12 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux $ rpm -q -a | grep drbd kmod-drbd82-xen-8.2.6-2 drbd82-8.2.6-1.el5.centos $ rpm -q -a | grep xen kmod-drbd82-xen-8.2.6-2 kernel-xen-2.6.18-128.4.1.el5 xen-libs-3.4.0-2.el5 xen-3.4.0-2.el5 $ rpm -q -a | grep lvm system-config-lvm-1.1.5-1.0.el5 lvm2-2.02.40-6.el5 $ rpm -q -a | grep qemu qemu-0.9.0-4 $ cat /etc/redhat-release CentOS release 5.3 (Final) $ lspci 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express $ ethtool -i eth0 driver: tg3 version: 3.93 firmware-version: 5722-v3.07, ASFIPMI v6.02 bus-info: 0000:03:00.0 --- The server configuration running CentOS 5.5 that DOES crash is as follows; --- $ uname -a Linux tnx176 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linux $ rpm -q -a | grep drbd kmod-drbd82-xen-8.2.6-2 drbd82-8.2.6-1.el5.centos $ rpm -q -a | grep xen xen-libs-3.4.3-3.el5 xen-3.4.3-3.el5 kmod-drbd82-xen-8.2.6-2 kernel-xen-2.6.18-194.26.1.el5 $ rpm -q -a | grep lvm lvm2-2.02.56-8.el5_5.6 $ rpm -q -a | grep qemu qemu-img-0.10.5-1.el5 $ cat /etc/redhat-release CentOS release 5.5 (Final) $ lspci 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) dmesg: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.2 (Aug 21, 2009) --- How reproducible: Under high I/O it is always reproducible. Steps to Reproduce: 1. With the above server config (Dell 710) setup Xen running across dual nodes sharing LVM volumes via DRBD. 2. Create multiple domU's (in our case 9 domU's) and add symlink to /etc/xen/auto/ so domU's start on bootup. 3. Boot server from cold and during domU start up dom0 crashes with paging request dump. Actual results: dom0 outputs a kernel paging error then immediate resets. Please note that the attached image was captured from a remote KVM session so only a partial kernel error is provided. For a full kernel error message refer to the xen-devel thread link posted above. Expected results: Dom0 should not crash under high I/O. Additional info:
My understanding is this: - DRBD (block device driver) causes the network stack to add references to pages that were temporarily granted to dom0 (blkback) - DRBD reports "I'm done" to blkback, violating the blkdev interface contract (because "background references" remain) - blkback waives its grants on the pages (correctly), the pages are unmapped - the network stack blows up Blkback works on any block device, and so it can rely on nothing else than the block device interface contract. Even if blkback knew about DRBD specific "background references", it could only deal with those by deferring unmapping. Then it would have to poll to see if the references are gone. Or the network stack would have to notify blkback. Alternatively, the network stack would have to unmap the pages itself and clean up the grants. These are unreasonable mixups of responsibilities. Additionally, if blkback waited for background references to vanish, those could stall blkback indefinitely (by holding back pages / ring entries), without actual block device activity. This problem should be fixed in DRBD. At least DRBD should add an option to disable the use of the zero-copy network interface. Ultimately, dom0 crashes due to the non-conforming dom0 block driver. For the time being, RHEL5 dom0 should protect itself with the "ethtool -K xenbr0 sg off" workaround, as verified in http://lists.linbit.com/pipermail/drbd-user/2009-March/011652.html. The command could be added to the xen script that brings up xenbr0. The performance effects of disabling scatter/gather on xenbr0 should be measured, and based on that we should decide whether the ethtool command should run unconditionally or be configurable.
(In reply to comment #5) > This problem should be fixed in DRBD. At least DRBD should add an option to > disable the use of the zero-copy network interface. Ultimately, dom0 crashes > due to the non-conforming dom0 block driver. I meant: This problem should be fixed in DRBD. At least DRBD should add an option to disable the use of the zero-copy network interface. Ultimately, dom0 crashes due to *this* non-conforming dom0 block driver.
As there's no DRBD package in RHEL that could fix this problem and easy workaround is available, we closed this bz as WONTFIX.