Bug 666005

Summary:

Dom0 crashes on high I/O when using DRBD from suspected network driver bug

Product:

Red Hat Enterprise Linux 5

Reporter:

prickett233

Component:

xen

Assignee:

Xen Maintainance List <xen-maint>

Status:

CLOSED DUPLICATE

QA Contact:

Virtualization Bugs <virt-bugs>

Severity:

high

Docs Contact:

Priority:

low

Version:

5.5

CC:

drjones, leiwang, lersek, minovotn, mrezanin, mshao, qwan, xen-maint, yuzhang

Target Milestone:

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2011-03-02 15:33:11 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

514499, 570796

Attachments:

Description	Flags
kernel error message dump before system resets	none

Description prickett233 2010-12-28 11:53:38 UTC

Created attachment 470943 [details]
kernel error message dump before system resets

Description of problem:

A kernel panic is experienced in the Dom0 when it is under high I/O load such as during system startup when multiple DomU's are starting up at the same time.  The issue is seen when running the DomU's on top of DRBD and LVM.

The kernel crash is dependent on network driver and the specific details of the cause are related to the scatter/gather features of the NIC are best explained here;

http://lists.linbit.com/pipermail/drbd-user/2009-March/011652.html 

For the purpose of searching this bug a snippet of the text in the above link is given.

---
 The problem is that when you have scatter/gather enabled, DRBD uses the
 zero-copy interface to the network stack -- the network stack has the
 annoying tendency to keep a reference to the pages of requests even
 after the TCP connection they were sent on is terminated _but_ DRBD
 makes the assumption that once the TCP connection to the peer is
 terminated, the pages of all pending requests are no longer referenced
 by the network stack and it can safely complete any pending I/O requests
 upwards.
---

When turning off the NIC's scatter/gather feature the Dom0 no longer crashes.

Further discussion about this issue is on the xen-devel mailing list, with link to thread here;

http://xen.1045712.n5.nabble.com/Dom0-panic-on-xen-3-4-3-td2642628.html




Version-Release number of selected component (if applicable):

We have two server environments running the exact same domU's.  The first environment running CentOS 5.3 does NOT crash and has a different network chipset and driver as follows;

---
$ uname -a
Linux ldx12020 2.6.18-128.4.1.el5xen #1 SMP Tue Aug 4 20:51:12 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -q -a | grep drbd
kmod-drbd82-xen-8.2.6-2
drbd82-8.2.6-1.el5.centos


$ rpm -q -a | grep xen
kmod-drbd82-xen-8.2.6-2
kernel-xen-2.6.18-128.4.1.el5
xen-libs-3.4.0-2.el5
xen-3.4.0-2.el5

$ rpm -q -a | grep lvm
system-config-lvm-1.1.5-1.0.el5
lvm2-2.02.40-6.el5

$ rpm -q -a | grep qemu
qemu-0.9.0-4

$ cat /etc/redhat-release 
CentOS release 5.3 (Final)

$ lspci
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express
04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express


$ ethtool -i eth0
driver: tg3
version: 3.93
firmware-version: 5722-v3.07, ASFIPMI v6.02
bus-info: 0000:03:00.0
---


The server configuration running CentOS 5.5 that DOES crash is as follows;

---
$ uname -a
Linux tnx176 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -q -a | grep drbd
kmod-drbd82-xen-8.2.6-2
drbd82-8.2.6-1.el5.centos

$ rpm -q -a | grep xen
xen-libs-3.4.3-3.el5
xen-3.4.3-3.el5
kmod-drbd82-xen-8.2.6-2
kernel-xen-2.6.18-194.26.1.el5

$ rpm -q -a | grep lvm
lvm2-2.02.56-8.el5_5.6

$ rpm -q -a | grep qemu
qemu-img-0.10.5-1.el5

$ cat /etc/redhat-release 
CentOS release 5.5 (Final)

$ lspci
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

dmesg:
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.2 (Aug 21, 2009)
---



How reproducible:

Under high I/O it is always reproducible.



Steps to Reproduce:
1. With the above server config (Dell 710) setup Xen running across dual nodes sharing LVM volumes via DRBD.
2. Create multiple domU's  (in our case 9 domU's) and add symlink to /etc/xen/auto/ so domU's start on bootup.
3. Boot server from cold and during domU start up dom0 crashes with paging request dump.
  


Actual results:

dom0 outputs a kernel paging error then immediate resets.  Please note that the attached image was captured from a remote KVM session so only a partial kernel error is provided.  

For a full kernel error message refer to the xen-devel thread link posted above. 




Expected results:

Dom0 should not crash under high I/O.



Additional info:

Comment 5 Laszlo Ersek 2011-01-04 12:51:49 UTC

My understanding is this:
- DRBD (block device driver) causes the network stack to add references to
pages that were temporarily granted to dom0 (blkback)
- DRBD reports "I'm done" to blkback, violating the blkdev interface contract
(because "background references" remain)
- blkback waives its grants on the pages (correctly), the pages are unmapped
- the network stack blows up

Blkback works on any block device, and so it can rely on nothing else than the block device interface contract. Even if blkback knew about DRBD specific "background references", it could only deal with those by deferring unmapping. Then it would have to poll to see if the references are gone. Or the network stack would have to notify blkback. Alternatively, the network stack would have to unmap the pages itself and clean up the grants. These are unreasonable mixups of responsibilities.

Additionally, if blkback waited for background references to vanish, those could stall blkback indefinitely (by holding back pages / ring entries), without actual block device activity.

This problem should be fixed in DRBD. At least DRBD should add an option to disable the use of the zero-copy network interface. Ultimately, dom0 crashes due to the non-conforming dom0 block driver.

For the time being, RHEL5 dom0 should protect itself with the "ethtool -K xenbr0 sg off" workaround, as verified in http://lists.linbit.com/pipermail/drbd-user/2009-March/011652.html. The command could be added to the xen script that brings up xenbr0. The performance effects of disabling scatter/gather on xenbr0 should be measured, and based on that we should decide whether the ethtool command should run unconditionally or be configurable.

Comment 6 Laszlo Ersek 2011-01-04 12:57:19 UTC

(In reply to comment #5)

> This problem should be fixed in DRBD. At least DRBD should add an option to
> disable the use of the zero-copy network interface. Ultimately, dom0 crashes
> due to the non-conforming dom0 block driver.

I meant:

This problem should be fixed in DRBD. At least DRBD should add an option to
disable the use of the zero-copy network interface. Ultimately, dom0 crashes
due to *this* non-conforming dom0 block driver.

Comment 15 Miroslav Rezanina 2011-03-02 15:33:11 UTC

As there's no DRBD package in RHEL that could fix this problem and easy workaround is available, we closed this bz as WONTFIX.