Bug 198498 - 2.6.17-1.2365.fc6 and Calgary-X watchdog hangs (DMA error) on 2-node x460.
2.6.17-1.2365.fc6 and Calgary-X watchdog hangs (DMA error) on 2-node x460.
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
6
All Linux
medium Severity medium
: ---
: ---
Assigned To: Konrad Rzeszutek
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-07-11 10:40 EDT by Konrad Rzeszutek
Modified: 2007-11-30 17:11 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-07-31 12:28:39 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Serial output showing the problem. (55.97 KB, text/plain)
2006-07-11 10:40 EDT, Konrad Rzeszutek
no flags Details
Config file used to build 2.6.17-1.2365.fc6 kernel. (60.40 KB, text/plain)
2006-07-12 10:48 EDT, Konrad Rzeszutek
no flags Details
Sysreport (248.24 KB, application/octet-stream)
2006-07-12 13:02 EDT, Konrad Rzeszutek
no flags Details
Mainline patch to fix this bug (6.33 KB, patch)
2006-07-14 19:33 EDT, Jon Mason
no flags Details | Diff
Patch against the fc6 kernel (6.49 KB, patch)
2006-07-17 14:29 EDT, Jon Mason
no flags Details | Diff

  None (edit)
Description Konrad Rzeszutek 2006-07-11 10:40:28 EDT
Description of problem:

2.6.17-1.2365.fc6 hangs under a two node x460. There is a couple of show tracks
and lastly this message shows up:

calgary_watchdog: DMA error on bus 0, CSR = 0x2010000


Version-Release number of selected component (if applicable):
2.6.17-1.2365.fc6

How reproducible:
Install 2.6.17-1.2365.fc6 kernel

Actual results:

Expected results:


Additional info:
Comment 1 Konrad Rzeszutek 2006-07-11 10:40:29 EDT
Created attachment 132239 [details]
Serial output showing the problem.
Comment 2 Muli Ben-Yehuda 2006-07-12 00:22:07 EDT
Looks like a Calgary bug indeed...

<Jul/11 10:28 am>Starting udev: Unable to handle kernel NULL pointer dereference
at 0000000000000030 RIP: 
<Jul/11 10:28 am> [<ffffffff80283c2f>] calgary_alloc_coherent+0x3c/0xfb

Can you please attach your .config?

As a workaround, you can boot with command line arguments "iommu=off" until we
fix this one. Thanks for reporting!
Comment 3 Konrad Rzeszutek 2006-07-12 10:48:01 EDT
Created attachment 132310 [details]
Config file used to build 2.6.17-1.2365.fc6 kernel.

If you want to test this kernel (please keep in mind that the FC/RHEL5 kernel
has also some extra patches on top of the 2.6.17+2.6.18-rc1-git3 patch set),
you can download the source rpm from:

ftp://ftp.tu-chemnitz.de/pub/linux/fedora-core/development/source/SRPMS/

There are also other mirror sites are much faster. Pls take a look
in: http://fedora.redhat.com/Download/mirrors.html.

If the problem is due to some patches being screwed up, please please report
this here with an explanation of what needs to be done to correct the problem.

Thanks!
Comment 4 Muli Ben-Yehuda 2006-07-12 12:19:09 EDT
Tnanks Konrad, I was looking for the SRPM earlier today. We'll give it a spin on
our test machine, if it's reproducible there, so much the better. We're also
trying to get some time on a dual node 460, will keep you updated.
Comment 5 Konrad Rzeszutek 2006-07-12 13:02:50 EDT
Created attachment 132316 [details]
Sysreport

I am also attaching a sysreport (a tarball of /proc, /etc, dmidecode, etc) so
that you can find out if your machine mirrors yours in case this is a config
issue.

Please note that I have HotPlug Memory on in the BIOS.
Comment 6 Muli Ben-Yehuda 2006-07-12 14:00:13 EDT
Ok, I think I know what's going on. This machine violates a rule we counted on
relating the PCI bus number to the Calgary PHB (which was a total hack, but
worked on every machine we tried it on so far). You have PCI busses 0xe and 0xf,
for which bus_to_phb() returns the same result (it should return different
results, as they're different PHBs), which leads to one of the busses not
getting initialized (in init_one()/init_one_nontranslated()), which leads to
dev->bus->self not being set, which leads to the NULL pointer deref.

Fixing this is a good chance to redo this logic to stop relying on any
relationship between PCI bus numbers and Calgary/PHB-per-Calgary. We're working
on it.
Comment 7 Jon Mason 2006-07-14 19:33:22 EDT
Created attachment 132472 [details]
Mainline patch to fix this bug

I was able to get access to a x460 and reproduce this problem with a mainline
kernel.  Attached is a patch against mainline that fixes the issue on my
system.  I am in the process of verifing the fix against the fc6 kernel you
linked above.  

There are many issues that had to be resolved to fix this problem.  Firstly,
when I originally wrote the code to handle NUMA systems, I had a large
misunderstand that was not corrected until now.  That was that I thought the
"number of nodes online" referred to number of physical systems connected.  So
that if NUMA was disabled, there would only be 1 node and it would only show
that node's PCI bus.  In reality if NUMA is disabled, the system displays all
of the connected system but is only ignorant of the delays in accessing main
memory.  Therefore, references to num_online_nodes() and MAX_NUMNODES are
incorrect and need to be set to the maximum number of nodes that can be
accessed (which are 4).  I created a variable, CAL_MAX_NODES, and set it to 4
to fix this.

Secondly, when walking the PCI in detect_calgary, the code only checked the
first "slot" when looking to see if a device is present.  This will work for
most cases, but unfortunately it isn't always the case.  In the NUMA MXE
drawers, there are USB devices present on the 3rd slot (with slot 1 being
empty).  So, to work around this, all slots (up to 8) are scanned to see if
there are any devices present.

Lastly, the bus is being enumerated on large systems in a different way the we
originally thought.  This throws the ugly logic we had out the window.	To more
elegantly handle this, I reorganized the kva array to be sparse (which removed
the need to have any bus number to kva slot logic in tce.c) and created a
secondary space array to contain the bus number to phb mapping.  This will
enable the remove the translation_disabled bitmap in the future.

With these changes Calgary boots on an x460 with 4 nodes with and without NUMA
enabled.
Comment 8 Jon Mason 2006-07-14 19:54:35 EDT
The patch above applies with minimal fuzz (-2 lines) to the fc6.  Testing on
this tree shortly.

Also, a better workaround for this problem would be "iommu=soft" as "iommu=off"
will cause problems in systems with mroe than 4G RAM.
Comment 9 Konrad Rzeszutek 2006-07-17 10:41:49 EDT
Jon,

The x460 can be an 8-node machine. Perhaps the CAL_MAX_NODES should be set to 8
instead of 4?
Comment 10 Jon Mason 2006-07-17 11:52:39 EDT
I verified that there are a maximum of 8 chassis for the x460 on the IBM
website.  I guess that is what happens when you ask someone instead of checking
the web.  Thanks Konrad, I'll fix that in the next version of the patch (to be
posted shortly).
Comment 11 Jon Mason 2006-07-17 14:29:49 EDT
Created attachment 132566 [details]
Patch against the fc6 kernel

This patch should fix the issue.  Please test and verify it works on your
system.
Comment 12 Konrad Rzeszutek 2006-07-18 16:28:54 EDT
Patch works splendidly.
Comment 13 Konrad Rzeszutek 2006-07-18 16:40:00 EDT
Patch posted on the internal reflector for inclusion in RHEL5.
Comment 14 Muli Ben-Yehuda 2006-07-30 05:38:36 EDT
Patch included into mainline as of 2.6.18-rc3. Bug can probably be closed now?
Comment 15 Konrad Rzeszutek 2006-07-31 12:17:38 EDT
Muli,

Not yet. Dave Jones/Don Zickus need to close this bug when he puts the patch in
the FC/RHEL kernel.
Comment 16 Don Zickus 2006-07-31 12:28:39 EDT
Sorry, patch was picked up and then dropped b/c rc3 has it.

-Don

Note You need to log in before you can comment on or make changes to this bug.