Bug 121801

I'm assigning this to Ingo to validate/certify that the suggestion
works and can be supported.

But Duncan, if you are truly trying to use an Opteron-based system,
then please install the relevant x86_64 RPM instead (for best
performance/scalability).

Thanks.  -ernie

Enabling HIGHMEM64G would break all 3rd party kernel modules. Unless
there is a very good reason why the 64 bit kernel cannot be used,
compatibility should be preserved.

I agree the easiest answer is to use the x86_64 version,
but we have some customers who are not ready to make
that change yet.

Breaking 3rd party modules would be bad, so in that
case is there any reason why an i686-smp kernel would
not be an acceptable (or supported) alternative?

Duncan, yes, installing an i686-smp kernel is a reasonable alternative,
although most preferable is installing the x86_64-smp kernel.

One reason why someone (myself included) would not use an i686-smp kernel is if 
you were going to run Oracle on it. They currently support Opteron only with 32 
bit kernel. In fact they suggest getting Opterons for future upgrades. So this 
leads us back to how do we get to see > 4GB of RAM with a 32-bit kernel on 
Opteron CPU's.

We just ran into this problem of RHEL 3.0 32-bit not recognizing more than 4GB
of memory on Opteron systems.

If I read this bug right, it looks like the following is the case:

1. RHEL documentation specifically states that the smp kernel supports 16GB of
memory: 

https://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-kernel.html

2. The 32-bit athlon kernel, which gets installed automatically on Opteron
systems, only supports 4GB of memory.

3. Red Hat isn't going to fix this.

Surely that can't be true?!?!? Can somebody please clarify?

Your Athlon system should be able to run the i686 SMP kernel and see all memory,
right?

And yes, the i686 kernel is 32 bit.

Yes. The i686 smp kernel can be installed, and yes, when it's installed the
system does see all available memory.

However, that's not a completely trivial process (reboot to uniprocessor kernel;
rpm -e the athlon smp kernel, install the i686 kernel, manually update
grub.conf, reboot), and simply shouldn't have to be done. We kickstart systems
-- it takes less than 60 seconds of an admin's time to initiate a kickstart, and
then you just let things run; when the system comes up it's immediately
available in production. With this, not only can't we do that, but it multiplies
the sysadmin time requirements to put a system in production more than tenfold.

If I install RHEL on a supported hardware configuration, it should work
correctly "out of the box" with the default install.

OK, so we need a way by which you can specify, in your kickstart config file,
that the i686 smp kernel should be installed instead of the athlon-smp kernel.

Jeremy, any idea how to do this?

The athlon-smp kernel performs better than the i686-smp kernel *because* the
i686-smp has support for >4GB enabled.  This means using 3 level page tables
instead of 2 level page tables, and being able to cache fewer page table entries
in the TLB.

If the athlon-smp kernel were modified to support >4GB of RAM, a similar kind of
performance degradation would happen to it.  However, changing the athlon-smp
kernel to support >4GB would involve switching to another page table format
(PAE), which would totally break compatibility with binary 3rd party modules. 
This is unacceptable in the middle of a RHEL release, so the solution to your
problem would be using the i686-smp kernel instead (and making sure that is the
kernel which is installed).