Bug 12230 - mmap64: incomplete check of offset
mmap64: incomplete check of offset
Product: Red Hat Linux
Classification: Retired
Component: glibc (Show other bugs)
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Jakub Jelinek
Depends On:
  Show dependency treegraph
Reported: 2000-06-13 19:33 EDT by John Reiser
Modified: 2008-05-01 11:37 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2000-06-13 22:44:40 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description John Reiser 2000-06-13 19:33:48 EDT
Bits 10 and 11 of the offset are not checked for being zero
when the offset is converted from a 64-bit byte count to a
32-bit page count.  The low 12 bits are discarded, but only
the low 10 bits (and the top 20 bits) are checked for being zero.

Here is version info and demonstration of the bug:
$ gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/specs
gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)
$ uname -a
Linux buffalo 2.2.14-5.0 #1 Tue Mar 7 21:07:39 EST 2000 i686 unknown
$ gdb /lib/libc.so.6
GNU gdb 19991004
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
welcome to change it and/or distribute copies of it under certain
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux"...
(gdb) x/20i mmap64
0xafc20 <__mmap64>:     push   %ebp
0xafc21 <__mmap64+1>:   push   %ebx
0xafc22 <__mmap64+2>:   push   %esi
0xafc23 <__mmap64+3>:   push   %edi
0xafc24 <__mmap64+4>:   mov    0x28(%esp,1),%edx
0xafc28 <__mmap64+8>:   mov    0x2c(%esp,1),%ecx
0xafc2c <__mmap64+12>:  test   $0x3ff,%edx         # test bottom 10 bits
0xafc32 <__mmap64+18>:  jne    0xafc6b <__mmap64+75>
0xafc34 <__mmap64+20>:  shrd   $0xc,%ecx,%edx      \  discard bottom 12
0xafc38 <__mmap64+24>:  shr    $0xc,%ecx           /
0xafc3b <__mmap64+27>:  jne    0xafc6b <__mmap64+75>    # test top 20 bits
0xafc3d <__mmap64+29>:  mov    %edx,%ebp
0xafc3f <__mmap64+31>:  mov    0x14(%esp,1),%ebx
0xafc43 <__mmap64+35>:  mov    0x18(%esp,1),%ecx
0xafc47 <__mmap64+39>:  mov    0x1c(%esp,1),%edx
0xafc4b <__mmap64+43>:  mov    0x20(%esp,1),%esi
0xafc4f <__mmap64+47>:  mov    0x24(%esp,1),%edi
0xafc53 <__mmap64+51>:  mov    $0xc0,%eax
0xafc58 <__mmap64+56>:  int    $0x80
0xafc5a <__mmap64+58>:  pop    %edi
Comment 1 John Reiser 2000-06-13 22:44:39 EDT
$  rpm -q -f /lib/libc.so.6
Comment 2 Jakub Jelinek 2000-09-03 10:03:48 EDT
Fixed in glibc 2.1.92-14

Note You need to log in before you can comment on or make changes to this bug.