Bug 497653 - get "bad pmd" when forking process with hugepage shared memeory segments [NEEDINFO]
get "bad pmd" when forking process with hugepage shared memeory segments
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.3
All Linux
medium Severity medium
: rc
: ---
Assigned To: Larry Woodman
Zhouping Liu
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-25 15:20 EDT by starlight
Modified: 2014-01-12 19:00 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-11 15:03:52 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
mfuruta: needinfo? (lwoodman)


Attachments (Terms of Use)
testcase (3.78 KB, text/plain)
2009-05-15 01:50 EDT, starlight
no flags Details
the reproducer program (3.65 KB, text/plain)
2011-08-02 02:26 EDT, Zhouping Liu
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Linux Kernel 13302 None None None Never

  None (edit)
Description starlight 2009-04-25 15:20:37 EDT
Description of problem:
=======================

We have a daemon that fork/execs scripts periodically.
Presently switching to hugepage allocations for shared
memory segments.

Works, but now the dmesg log is flooded with messages like

   mm/memory.c:118: bad pmd ffff8102d3ab8fb0(80000003974000e7).
   mm/memory.c:118: bad pmd ffff8102d3ab8fb8(80000003976000e7).
   mm/memory.c:118: bad pmd ffff8102d3ab8fc0(80000003970000e7).
   mm/memory.c:118: bad pmd ffff8102d3ab8fc8(80000003972000e7).

whenever fork() is executed by the daemon.  Would use
posix_spawn() but apparently in Linux this is just a
library function that performs a fork/exec.

Appears on both an Opteron and a Core 2 server.


Version-Release number of selected component (if applicable):
=============================================================

kernel 2.6.18-128.1.6.el5

How reproducible:
=================

Allocate hugepage shared memory segments and attache them.
Then issue fork().
  
Actual results:

"bad pmd" messages from kernel

Expected results:

No errors.

Additional info:

No error messages under RHEL4 kernel 2.6.9-78.0.13.ELsmp.
Comment 1 starlight 2009-04-27 16:40:49 EDT
Tried vfork() and this works-around the issue.  Not a great 
solution since the effective user ID for the child cannot be 
changed to non-root when using vfork() rather than fork().

Also it's bad that the parent daemon (multi-threaded) is
blocked until the exec() call issued.

Best solution would be for Linux to implement a proper 
posix_spawn() system call.
Comment 2 starlight 2009-05-15 01:50:23 EDT
Created attachment 344083 [details]
testcase

At least on the DL160 this testcase wrecks kernel
2.6.18-128.1.6.el5 100% every time.  hugepages=2048
should be set.
Comment 3 starlight 2009-06-01 23:29:48 EDT
Upstream reports and patches:

http://bugzilla.kernel.org/show_bug.cgi?id=13302
[current activity and patches all here]

http://bugzilla.kernel.org/show_bug.cgi?id=13192

http://bugzilla.kernel.org/show_bug.cgi?id=12134
Comment 4 Larry Woodman 2010-06-09 14:50:15 EDT
The attached patch that was posted to rhkernel-list fixes this problem:

--- linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c.orig 2010-06-09 10:01:41.000000000 -0400
+++ linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c      2010-06-09 10:02:27.000000000 -0400
@@ -26,12 +26,15 @@
        unsigned long sbase = saddr & PUD_MASK;
        unsigned long s_end = sbase + PUD_SIZE;

+       /* allow segments to share if only one is marked locked */
+       unsigned long vm_flags = vma->vm_flags & ~VM_LOCKED;
+       unsigned long svm_flags = svma->vm_flags & ~VM_LOCKED;
        /*
         * match the virtual addresses, permission and the alignment of the
         * page table page.
         */
        if (pmd_index(addr) != pmd_index(saddr) ||
-           vma->vm_flags != svma->vm_flags ||
+           vm_flags != svm_flags ||
            sbase < svma->vm_start || svma->vm_end < s_end)
                return 0;



Larry Woodman
Comment 5 Zhouping Liu 2011-08-02 02:26:42 EDT
Created attachment 516255 [details]
the reproducer program

I updated the reproducer.
 
Before running the reproduce case, you may need to
set overcommit_memory and hugepages, do like this:
# echo 2048 > /proc/sys/vm/nr_hugepages
# echo 1 > /proc/sys/vm/overcommit_memory
and I reproduced it on kernel-2.6.18-128.1.6.el5 and kernel-2.6.18-164.el5

thanks.
Comment 6 RHEL Product and Program Management 2011-08-11 21:25:06 EDT
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.
Comment 7 starlight 2011-08-12 00:39:32 EDT
This is a serious system corruption bug with an upstream fix.
Comment 8 RHEL Product and Program Management 2012-01-09 08:55:11 EST
This request was evaluated by Red Hat Product Management for inclusion in Red Hat Enterprise Linux 5.8 and Red Hat does not plan to fix this issue the currently developed update.

Contact your manager or support representative in case you need to escalate this bug.
Comment 11 masanari iida 2012-11-28 03:30:03 EST
(In reply to comment #4)
> The attached patch that was posted to rhkernel-list fixes this problem:
> 
> --- linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c.orig 2010-06-09
> 10:01:41.000000000 -0400
> +++ linux-2.6.18.x86_64/arch/i386/mm/hugetlbpage.c      2010-06-09
> 10:02:27.000000000 -0400
> @@ -26,12 +26,15 @@
>         unsigned long sbase = saddr & PUD_MASK;
>         unsigned long s_end = sbase + PUD_SIZE;
> 
> +       /* allow segments to share if only one is marked locked */
> +       unsigned long vm_flags = vma->vm_flags & ~VM_LOCKED;
> +       unsigned long svm_flags = svma->vm_flags & ~VM_LOCKED;
>         /*
>          * match the virtual addresses, permission and the alignment of the
>          * page table page.
>          */
>         if (pmd_index(addr) != pmd_index(saddr) ||
> -           vma->vm_flags != svma->vm_flags ||
> +           vm_flags != svm_flags ||
>             sbase < svma->vm_start || svma->vm_end < s_end)
>                 return 0;
> 
> 
> 
> Larry Woodman

Larry, I found this patch is already included in RHEL5.6(2.6.18-238) and later.
But Changelog doesn't include this BZ#, and this case is "Assigned" status.
Would you mind to double check if this symptom is fixed?
Comment 14 Larry Woodman 2012-11-29 10:31:24 EST
Masaki, yes this problem is fixed by the patch in Comment #4 and it is in RHEL5.6.
Do you know if anyone has seen this "bad pmd" message while running RHEL5.6 or later???

Larry
Comment 15 starlight 2012-11-29 10:44:04 EST
Can re-test if it would be helpful.
Currently running 2.6.18-308.el5.
Comment 16 masanari iida 2012-11-29 19:59:24 EST
Larry, Thanks for the confirmation.
One of my customer encountered this symptom on RHEL5.3.
When I was looking for a solution, I found this BZ and get confused.
Comment 19 starlight 2012-11-29 21:08:06 EST
I am the original reporter.  Eventually
figured out that the Linux implementation
of vfork() only blocks the calling thread
and allows modification of user id, group
id and other process attributes (unlike
traditional UNIX vfork) so we never went
back and tested fork()--especially as
RH management declared that it was not
and would not be fixed.

However a simple environment variable
tweak will put fork() back so I am
willing to re-test it if it makes
a difference to anyone.
Comment 20 Andrius Benokraitis 2013-11-11 15:03:52 EST
After thorough deliberation, this bugzilla is not planned on being addressed in the Red Hat Enterprise Linux 5 time frame. Current efforts are focused on Red Hat Enterprise Linux 6, and future major releases.

Note You need to log in before you can comment on or make changes to this bug.