Bug 91280

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.3)
Gecko/20030312

Description of problem:
I have a multiplatform package that compiles and runs fine on many platforms,
including previous Redhat Linux distributions (up to 8). Compiling it on Redhat
9 causes a specific segmentation fault which I traced to the particular libc and
libm shared objects in /lib/tls. If I create links to the corresponding shared
objects in /lib, my package runs, but when I log out, the login screen never
comes up. Also on reboot, the X-server refuses to come up. The conclusion is
that something is wrong with /lib/tls/libc-2.3.2.so and/or /lib/tls/libm-2.3.2.so.

More detail: Here is what I get when I run gdb on a core file:
[bheymann@Callisto] 511 (~/projects/rubisco): gdb bhead core.1256
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Core was generated by `bhead -v7 t.map t2.map'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /home4/EMprogs/bsoft_linux/lib/libbsoft.so...done.
Loaded symbols for /home4/EMprogs/bsoft_linux/lib/libbsoft.so
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0  0x420744f5 in _int_malloc () from /lib/tls/libc.so.6
(gdb) where
#0  0x420744f5 in _int_malloc () from /lib/tls/libc.so.6
#1  0x4207360b in malloc () from /lib/tls/libc.so.6
#2  0x400bd068 in balloc(long) (memsize=1108550432) at utilities.c:165
#3  0x40056df5 in img_read_data(_IO_FILE*, Bimage*, int, int, int, int) (
    fimg=0x804b9a0, p=0x804b388, select=0, swap=0, vax=0, pad=0) at rwimg.c:223
#4  0x40052714 in readCCP4(Bimage*) (p=0x804b388) at rwCCP4.c:134
#5  0x400569a1 in read_img(char*, int, int) (filename=0xbffff5c8 "t.map",
    readdataflag=1, select=-1) at rwimg.c:114
#6  0x0804914b in main (argc=4, argv=0xbfffdcc4) at bhead.c:152
#7  0x420156a4 in __libc_start_main () from /lib/tls/libc.so.6
(gdb) up
#1  0x4207360b in malloc () from /lib/tls/libc.so.6
(gdb) up
#2  0x400bd068 in balloc(long) (memsize=1108550432) at utilities.c:165
165             if ( ( ptr = (char *) malloc(memsize*sizeof(char)) ) == NULL ) {
Current language:  auto; currently c++
(gdb) up
#3  0x40056df5 in img_read_data(_IO_FILE*, Bimage*, int, int, int, int) (
    fimg=0x804b9a0, p=0x804b388, select=0, swap=0, vax=0, pad=0) at rwimg.c:223
223             char*                   data = (char *) balloc(datasize*sizeof(c
har));
(gdb) p datasize
$1 = 40000
(gdb) quit

The key line is #2, where the argument (memsize=1108550432) passed into the
function balloc is incorrect, regardless of the actual value passed
(datasize=40000). It seems there is a problem with the argument passing.

Version-Release number of selected component (if applicable):
libc-2.3.2.so

How reproducible:
Always

Steps to Reproduce:
1. Install Bsoft: Unpack tgz file
2. Compile under Redhat 9: type "bmake" in the bsoft directory
3. Run a program, reading and writing an image file:
bimg -v7 img.pif out.pif

Actual Results:  Segmentation fault
Cause: Failure in memory allocation because the wrong argument value was passed
to the function calling malloc.

Expected Results:  Correct memory allocating, reading and writing of the data files.

Additional info: