> It has been fixed in recent (last 2 months) FC6/RawHide kernels, I did not track
> down the specific change.

Hi Jan,
 
Can you point me to a specific Red Hat kernel where you see something
other than "---p" for the "maps" line for the gate virtual memory area?

I ask because I cannot find any Red Hat kernels that have a "gate_vma" which
have its "vm_flags" field set to anything other than zero -- which causes
what you're seeing.  Here's the relevant kernel function:

static int show_map(struct seq_file *m, void *v)
{
        struct vm_area_struct *map = v;
        struct file *file = map->vm_file;
        int flags = map->vm_flags;
        unsigned long ino = 0;
        dev_t dev = 0;
        int len;

        if (file) {
                struct inode *inode = map->vm_file->f_dentry->d_inode;
                dev = inode->i_sb->s_dev;
                ino = inode->i_ino;
        }

        seq_printf(m, "%08lx-%08lx %c%c%c%c %08lx %02x:%02x %lu %n",
                        map->vm_start,
                        map->vm_end,
                        flags & VM_READ ? 'r' : '-',
                        flags & VM_WRITE ? 'w' : '-',
                        (flags & VM_EXEC
#ifdef __i386__
                                || (!nx_enabled && map->vm_mm &&
                                (map->vm_start < map->vm_mm->context.exec_limit))
#endif
                        )
                                ? 'x' : '-',
                        flags & VM_MAYSHARE ? 's' : 'p',
                        map->vm_pgoff << PAGE_SHIFT,
                        MAJOR(dev), MINOR(dev), ino, &len);
...

Assigning this to myself.

FC6-GA: kernel-2.6.18-1.2798.fc6.x86_64
$ set +H
$ echo 'int main(void) { for (;;); }' >deadlock.c;gcc -o deadlock32 deadlock.c
-ggdb3 -Wall -m32;./deadlock32 &cat /proc/$!/maps|grep ^ffff
ffffe000-fffff000 r-xp ffffe000 00:00 0 
$ echo 'int main(void) { for (;;); }' >deadlock.c;gcc -o deadlock64 deadlock.c
-ggdb3 -Wall -m64;./deadlock64 &cat /proc/$!/maps|grep ^ffff
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]

RHEL4U4: kernel-smp-2.6.9-42.0.3.EL.x86_64
$ set +H
$ echo 'int main(void) { for (;;); }' >deadlock.c;gcc -o deadlock32 deadlock.c
-ggdb3 -Wall -m32;./deadlock32 &cat /proc/$!/maps|grep ^ffff
ffffc000-ffffe000 rw-p ffffc000 00:00 0   # ignore this line...
ffffe000-fffff000 ---p 00000000 00:00 0 
$ echo 'int main(void) { for (;;); }' >deadlock.c;gcc -o deadlock64 deadlock.c
-ggdb3 -Wall -m64;./deadlock64 &cat /proc/$!/maps|grep ^ffff
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 

You can see there the (32-bit only) "r-xp" (FC6-GA) vs. "---p" (RHEL4.4)
difference for the range ffffe000-fffff000 where the code hangs during backtrace
of a process in system call.

It should be fixed to "r-xp" even for the 64-bit mode but I am not aware of any
application using it, gdb(1) does not need it for the backtraces of 64-bit
processes.

The fix for 32-bit processes running on x86_64 has already been committed
to the RHEL4 U5 patch pool on 11-Oct-2006 (in kernel version 2.6.9-42.18.EL)
as part of Eric Sandeen's fix for bug 178981.

On i386, both upstream 2.6.18 and Fedora FC-6 kernels do not show a
vm region from 0xffffe000 to 0xfffff000 in /proc/<pid>/maps, and RHEL4
shows an unaccessible region (---p) which is truly unaccessible, since
it is mapped for kernel read-only access.

Thus, on i386 (i.e., x86), I don't believe any changes are appropriate.

I'm marking this BZ as specific to x86_64, and I'll address the remaining
case (native 64-bit processes on x86_64) tomorrow.

A patch to fix the native 64-bit case on x86_64 has been posted for
internal review on 9-Nov-2006.  I've also sent a fix upstream.

committed in stream U5 build 42.25. A test kernel with this patch is available
from http://people.redhat.com/~jbaron/rhel4/

QE ack for RHEL4.5.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0304.html