In order to support between 4GB and 16GB of RAM in 32-bit mode on an opteron system the kernel needs to be compiled with the CONFIG_HIGHMEM64G option enabled. From what I can gather, the -smp kernels should be compiled with PAE + highpte support by enabling the CONFIG_HIGHMEM64G option, which will also enable CONFIG_X86_PAE. Then the -hugemem kernel should support PAE + highpte + 4/4 split by enabling the CONFIG_HIGHMEM64G and the CONFIG_X86_4G options. (see the posting by Rik van Riel about this: http://www.redhat. com/archives/taroon-beta-list/2003-August/msg00270.html) Currently the i686-smp kernel has the HIGHMEM64G option enabled while the athlon-smp kernel only has HIGHMEM4G enabled. This means it is possible to use the i686-smp kernel to run opteron system with between 4GB and 16GB in 32-bit mode and use the hugemem kernel on systems with >16GB. However it seems that the athlon-smp kernel should also be compiled with the CONFIG_HIGHMEM64G option so it looks and acts the same as the i686-smp variant. Steps to Reproduce: 1. Install >4GB RAM in an opteron system 2. Install and boot kernel-smp-2.4.21-9.0.3.EL.athlon.rpm Actual Results: recognized RAM is limited to 4GB Expected Results: all available RAM is recognized (up to 16GB)
I'm assigning this to Ingo to validate/certify that the suggestion works and can be supported. But Duncan, if you are truly trying to use an Opteron-based system, then please install the relevant x86_64 RPM instead (for best performance/scalability). Thanks. -ernie
Enabling HIGHMEM64G would break all 3rd party kernel modules. Unless there is a very good reason why the 64 bit kernel cannot be used, compatibility should be preserved.
I agree the easiest answer is to use the x86_64 version, but we have some customers who are not ready to make that change yet. Breaking 3rd party modules would be bad, so in that case is there any reason why an i686-smp kernel would not be an acceptable (or supported) alternative?
Duncan, yes, installing an i686-smp kernel is a reasonable alternative, although most preferable is installing the x86_64-smp kernel.
One reason why someone (myself included) would not use an i686-smp kernel is if you were going to run Oracle on it. They currently support Opteron only with 32 bit kernel. In fact they suggest getting Opterons for future upgrades. So this leads us back to how do we get to see > 4GB of RAM with a 32-bit kernel on Opteron CPU's.
We just ran into this problem of RHEL 3.0 32-bit not recognizing more than 4GB of memory on Opteron systems. If I read this bug right, it looks like the following is the case: 1. RHEL documentation specifically states that the smp kernel supports 16GB of memory: https://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-kernel.html 2. The 32-bit athlon kernel, which gets installed automatically on Opteron systems, only supports 4GB of memory. 3. Red Hat isn't going to fix this. Surely that can't be true?!?!? Can somebody please clarify?
Your Athlon system should be able to run the i686 SMP kernel and see all memory, right? And yes, the i686 kernel is 32 bit.
Yes. The i686 smp kernel can be installed, and yes, when it's installed the system does see all available memory. However, that's not a completely trivial process (reboot to uniprocessor kernel; rpm -e the athlon smp kernel, install the i686 kernel, manually update grub.conf, reboot), and simply shouldn't have to be done. We kickstart systems -- it takes less than 60 seconds of an admin's time to initiate a kickstart, and then you just let things run; when the system comes up it's immediately available in production. With this, not only can't we do that, but it multiplies the sysadmin time requirements to put a system in production more than tenfold. If I install RHEL on a supported hardware configuration, it should work correctly "out of the box" with the default install.
OK, so we need a way by which you can specify, in your kickstart config file, that the i686 smp kernel should be installed instead of the athlon-smp kernel. Jeremy, any idea how to do this?
The athlon-smp kernel performs better than the i686-smp kernel *because* the i686-smp has support for >4GB enabled. This means using 3 level page tables instead of 2 level page tables, and being able to cache fewer page table entries in the TLB. If the athlon-smp kernel were modified to support >4GB of RAM, a similar kind of performance degradation would happen to it. However, changing the athlon-smp kernel to support >4GB would involve switching to another page table format (PAE), which would totally break compatibility with binary 3rd party modules. This is unacceptable in the middle of a RHEL release, so the solution to your problem would be using the i686-smp kernel instead (and making sure that is the kernel which is installed).