Red Hat Bugzilla – Bug 497653
get "bad pmd" when forking process with hugepage shared memeory segments
Last modified: 2014-01-12 19:00:08 EST
Description of problem:
We have a daemon that fork/execs scripts periodically.
Presently switching to hugepage allocations for shared
Works, but now the dmesg log is flooded with messages like
mm/memory.c:118: bad pmd ffff8102d3ab8fb0(80000003974000e7).
mm/memory.c:118: bad pmd ffff8102d3ab8fb8(80000003976000e7).
mm/memory.c:118: bad pmd ffff8102d3ab8fc0(80000003970000e7).
mm/memory.c:118: bad pmd ffff8102d3ab8fc8(80000003972000e7).
whenever fork() is executed by the daemon. Would use
posix_spawn() but apparently in Linux this is just a
library function that performs a fork/exec.
Appears on both an Opteron and a Core 2 server.
Version-Release number of selected component (if applicable):
Allocate hugepage shared memory segments and attache them.
Then issue fork().
"bad pmd" messages from kernel
No error messages under RHEL4 kernel 2.6.9-78.0.13.ELsmp.
Tried vfork() and this works-around the issue. Not a great
solution since the effective user ID for the child cannot be
changed to non-root when using vfork() rather than fork().
Also it's bad that the parent daemon (multi-threaded) is
blocked until the exec() call issued.
Best solution would be for Linux to implement a proper
posix_spawn() system call.
Created attachment 344083 [details]
At least on the DL160 this testcase wrecks kernel
2.6.18-128.1.6.el5 100% every time. hugepages=2048
should be set.
Upstream reports and patches:
[current activity and patches all here]
The attached patch that was posted to rhkernel-list fixes this problem:
--- linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c.orig 2010-06-09 10:01:41.000000000 -0400
+++ linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c 2010-06-09 10:02:27.000000000 -0400
@@ -26,12 +26,15 @@
unsigned long sbase = saddr & PUD_MASK;
unsigned long s_end = sbase + PUD_SIZE;
+ /* allow segments to share if only one is marked locked */
+ unsigned long vm_flags = vma->vm_flags & ~VM_LOCKED;
+ unsigned long svm_flags = svma->vm_flags & ~VM_LOCKED;
* match the virtual addresses, permission and the alignment of the
* page table page.
if (pmd_index(addr) != pmd_index(saddr) ||
- vma->vm_flags != svma->vm_flags ||
+ vm_flags != svm_flags ||
sbase < svma->vm_start || svma->vm_end < s_end)
Created attachment 516255 [details]
the reproducer program
I updated the reproducer.
Before running the reproduce case, you may need to
set overcommit_memory and hugepages, do like this:
# echo 2048 > /proc/sys/vm/nr_hugepages
# echo 1 > /proc/sys/vm/overcommit_memory
and I reproduced it on kernel-2.6.18-128.1.6.el5 and kernel-2.6.18-164.el5
Development Management has reviewed and declined this request. You may appeal
this decision by reopening this request.
This is a serious system corruption bug with an upstream fix.
This request was evaluated by Red Hat Product Management for inclusion in Red Hat Enterprise Linux 5.8 and Red Hat does not plan to fix this issue the currently developed update.
Contact your manager or support representative in case you need to escalate this bug.
(In reply to comment #4)
> The attached patch that was posted to rhkernel-list fixes this problem:
> --- linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c.orig 2010-06-09
> 10:01:41.000000000 -0400
> +++ linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c 2010-06-09
> 10:02:27.000000000 -0400
> @@ -26,12 +26,15 @@
> unsigned long sbase = saddr & PUD_MASK;
> unsigned long s_end = sbase + PUD_SIZE;
> + /* allow segments to share if only one is marked locked */
> + unsigned long vm_flags = vma->vm_flags & ~VM_LOCKED;
> + unsigned long svm_flags = svma->vm_flags & ~VM_LOCKED;
> * match the virtual addresses, permission and the alignment of the
> * page table page.
> if (pmd_index(addr) != pmd_index(saddr) ||
> - vma->vm_flags != svma->vm_flags ||
> + vm_flags != svm_flags ||
> sbase < svma->vm_start || svma->vm_end < s_end)
> return 0;
> Larry Woodman
Larry, I found this patch is already included in RHEL5.6(2.6.18-238) and later.
But Changelog doesn't include this BZ#, and this case is "Assigned" status.
Would you mind to double check if this symptom is fixed?
Masaki, yes this problem is fixed by the patch in Comment #4 and it is in RHEL5.6.
Do you know if anyone has seen this "bad pmd" message while running RHEL5.6 or later???
Can re-test if it would be helpful.
Currently running 2.6.18-308.el5.
Larry, Thanks for the confirmation.
One of my customer encountered this symptom on RHEL5.3.
When I was looking for a solution, I found this BZ and get confused.
I am the original reporter. Eventually
figured out that the Linux implementation
of vfork() only blocks the calling thread
and allows modification of user id, group
id and other process attributes (unlike
traditional UNIX vfork) so we never went
back and tested fork()--especially as
RH management declared that it was not
and would not be fixed.
However a simple environment variable
tweak will put fork() back so I am
willing to re-test it if it makes
a difference to anyone.
After thorough deliberation, this bugzilla is not planned on being addressed in the Red Hat Enterprise Linux 5 time frame. Current efforts are focused on Red Hat Enterprise Linux 6, and future major releases.