From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040524

Description of problem:
Running a third party 32bit application on a x86_64 install using the 2.6.9-22.0.1.EL (SMP) x86_64 kernel crashes the machine with:

Kernel BUG at prio_tree:528
invalid operand: 0000 [1] SMP 
CPU 0 
Modules linked in: nfsd exportfs md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core nfs lockd sunrpc dm_mod button battery ac ohci_hcd ehci_hcd tg3 floppy ext3 jbd sata_nv libata sd_mod scsi_mod
Pid: 3173, comm: app.bin Not tainted 2.6.9-22.0.1.ELsmp
RIP: 0010:[<ffffffff8015e780>] <ffffffff8015e780>{vma_prio_tree_add+70}
RSP: 0018:000001007d4ade80  EFLAGS: 00010216
RAX: 0000000000000016 RBX: 00000100349be4b8 RCX: 0000000000000000
RDX: 0000000000000434 RSI: 000001003c6d11f8 RDI: 00000100349be4b8
RBP: 000001007efb2040 R08: 0000000000000016 R09: 0000000000000000
R10: ffffffff803ef900 R11: 0000010001003950 R12: 0000010034ab8298
R13: 000001013f667ba0 R14: 0000010034ab82a8 R15: 0000010034ab8268
FS:  0000002a959b6de0(0000) GS:ffffffff804d3080(005b) knlGS:00000000f1a316c0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000e6209000 CR3: 0000000000101000 CR4: 00000000000006e0
Process app.bin (pid: 3173, threadinfo 000001007d4ac000, task 000001007fdd67f0)
Stack: ffffffff80169022 0000000000000000 00000100349be4b8 0000000000000000 
       0000000000100077 000001013d9bc7c0 00000000e620a000 0000000000435000 
       ffffffff8016a9a7 0000000000000435 
Call Trace:<ffffffff80169022>{vma_link+204} <ffffffff8016a9a7>{do_mmap_pgoff+1444} 
       <ffffffff80126f00>{sys32_mmap2+253} <ffffffff8012500f>{cstar_do_call+27} 
       

Code: 0f 0b 4e eb 31 80 ff ff ff ff 10 02 48 c7 47 60 00 00 00 00 
RIP <ffffffff8015e780>{vma_prio_tree_add+70} RSP <000001007d4ade80>
 <0>Kernel panic - not syncing: Oops


Version-Release number of selected component (if applicable):
kernel-smp-2.6.9-22.0.1.EL

How reproducible:
Always

Steps to Reproduce:
1. Run application

  

Actual Results:  Oops as above

Expected Results:  Doesn't oops ...

Additional info:

I think it's related to to the linux-2.6.9-x86_64-flexmmap.patch (and linux-2.6.9-x86_64-task_size-32bit.patch) - backing out both these patches and rebuilding a kernel and the application doesn't oops.

However, with the default kernel, 32 bit applications can grab up to 4Gb of memory - with these patches removed, they can only get about 2.7Gb

Also, using a FC4 kernel (2.6.11-1.1369_FC4smp) kernel on the same machine and it doesn't oops ...