Description of problem: When calling mmap64, a subroutine call to do_mmap_pgoff() appears to fail. A semaphore is also held and never released because of this failure, causing the application to hang. Engineering suspects a failed call to make_pages_present() on line 1157 of mm/mmap.c may be a cause, but this requires more investigation. The customer is running application with realtime priority 1 (lowest of the realtime priorities) using the FIFO scheduler. 32 cores on the host, 30 cores are used for the application. They use the kernel parameter isolcpus=1-7,9-31 which is uncommon. Version-Release number of selected component (if applicable): RHEL5.5 kernel 2.6.18-238.21.1.el5. How reproducible: Four instances of this are believed to have happened to the customer. Steps to Reproduce: Unknown. 1. 2. 3. Actual results: Expected results: do_mmap_pgoff() should be successful. Additional info: Crash dump vmcore.00914556 uploading to dropbox.
Correction: the line is really isolcpus=1,3-31 .
Making bug public.
Since we've identified this as an application / library interaction problem, I'm closing this as NOTABUG. We can re-open if the customer finds a real RH bug in their further investigation.