Bug 10509

Summary:	floating point calculations yield NaN although they shouldn't
Product:	[Retired] Red Hat Linux	Reporter:	Cord Kielhorn <kielhorn>
Component:	kernel	Assignee:	Michael K. Johnson <johnsonm>
Status:	CLOSED ERRATA	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	6.2	CC:	lewilson
Target Milestone:	---
Target Release:	---
Hardware:	i386
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2000-04-22 07:19:06 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Cord Kielhorn 2000-04-01 18:34:29 UTC

This happens on at least PIII (Katmai) SMP machines with kernel-2.2.14-5.0
(the pre 6.2 one from rawhide, but the 6.2 version is not different with
respect to that (btw: why didn't you increment the release number between
rawhide's 2.2.14-5.0 and zoot's 2.2.14-5.0?):
Under seldom circumstances floating point calculations yield NaN although
they haven't before in the same run. For example the same calculation
yields the correct result for several million times when it suddenly
"Nans". Maybe sometimes floating point calculations even yield a wrong but
non-Nan result, we haven't checked that extensively. We have observed this
behaviour on several machines, so it shouldn't be a hardware problem. The
processors are 500 MHz Katmai with CPU id 0672 and microcode patches 000B
and 0010. It seems to be a problem with the fxsave and fxrestore fast
extended FPU save and restore feature used in kernel-2.2.14-5.0's
linux-2.2.12-PIII.patch: if one rebuilds the kernel with CONFIG_X86_FX
unconfigured the problems disappear.

Cheers
  Cord

Comment 1 Doug Ledford 2000-04-22 07:19:59 UTC

The errata kernel doesn't include the PIII patches and as such solves this
problem.  It's also beleived to be fixed internally in the latest version of the
PIII patches.