Bug 158665 - ppmquant fails with "rc=11" because of segfault in pnmcolormap
ppmquant fails with "rc=11" because of segfault in pnmcolormap
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: netpbm (Show other bugs)
3
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Jindrich Novy
Ben Levenson
:
: 139111 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2005-05-24 13:56 EDT by Josh Parsons
Modified: 2013-07-02 19:07 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-06-01 07:53:27 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch to fix the pnmcolormap segfault (404 bytes, patch)
2005-05-31 09:06 EDT, Jindrich Novy
no flags Details | Diff

  None (edit)
Description Josh Parsons 2005-05-24 13:56:39 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Fedora/1.0.4-1.3.1 Firefox/1.0.4

Description of problem:
ppmquant fails with "rc=11" on every input I've been able to give it.

Version-Release number of selected component (if applicable):
netpbm-progs-10.27-3.FC3

How reproducible:
Always

Steps to Reproduce:
ppmpat -madras 16 16 |/usr/bin/ppmquant 2 >/dev/null

Actual Results:  pnmcolormap: making histogram...
pnmcolormap failed, rc=11


Expected Results:  pnmcolormap: making histogram...
pnmcolormap: 3 colors found
pnmcolormap: choosing 2 colors...
pnmremap: 2 colors found in colormap


Additional info:
Comment 1 Jindrich Novy 2005-05-25 04:55:52 EDT
This is weird. This just works for me with netppm-10.27-3. My output is:

[jnovy@obelix FC-3]$ ppmpat -madras 16 16 |/usr/bin/ppmquant 2 >/dev/null
pnmcolormap: making histogram...
pnmcolormap: 3 colors found
pnmcolormap: choosing 2 colors...
pnmremap: 2 colors found in colormap
[jnovy@obelix FC-3]$

It looks like it fails in computeHistogram() in pnmcolormap.c. From a brief look
into the code I see it might fail in seek/tell file operations called via the
libpam interface. However I cannot reproduce it. Have you checked free space on
your disc?
Comment 2 Ben Webb 2005-05-25 20:06:39 EDT
Same problem here on two FC3 boxes: with netpbm-10.25-2 it works:
[ben@synth doc_INT]$ ppmpat -madras 16 16 | /usr/bin/ppmquant 2 > /dev/null
pnmcolormap: making histogram...
pnmcolormap: 3 colors found
pnmcolormap: choosing 2 colors...
pnmremap: 2 colors found in colormap

while with netpbm-10.27-3.FC3 it fails:
[ben@synth doc_INT]$ ppmpat -madras 16 16 | /usr/bin/ppmquant 2 > /dev/null
pnmcolormap: making histogram...
pnmcolormap failed, rc=11

Simply downgrading to 10.25-2 again makes the problem go away, so it can't be a
disk space problem.
Comment 3 Josh Parsons 2005-05-26 18:48:46 EDT
I have plenty of free disk space - but the problem is getting worse.  Now
pnmcolormap is dumping core:

$ ppmpat -madras 16 16 |pnmcolormap all >/dev/null
pnmcolormap: making histogram...
Segmentation fault

$ ppmpat -madras 16 16 >test.ppm
$ gdb --args /usr/bin/pnmcolormap all test.ppm

...

Program received signal SIGSEGV, Segmentation fault.
0x414a4513 in pnm_freetupletable () from /usr/lib/libnetpbm.so.10
(gdb) bt
#0  0x414a4513 in pnm_freetupletable () from /usr/lib/libnetpbm.so.10
#1  0x414a46a0 in pnm_computetuplefreqtable2 () from /usr/lib/libnetpbm.so.10
#2  0x08048ecb in ?? ()
#3  0xbfffeb10 in ?? ()
#4  0x00000000 in ?? ()
Comment 4 Jindrich Novy 2005-05-30 10:32:30 EDT
Yes, this error is caused by the segfault in pnmcolormap call to tuplehashtotable():

Program received signal SIGSEGV, Segmentation fault.
0x0000002a9559e93e in tuplehashtotable (pamP=0x7fbffff690,
tuplehash=0x2a95559010, allocsize=32767) at libpammap.c:431

431                     tupletable[j]->value = p->tupleint.value;
(gdb) bt
#0  0x0000002a9559e93e in tuplehashtotable (pamP=0x7fbffff690,
tuplehash=0x2a95559010, allocsize=32767) at libpammap.c:431
#1  0x0000002a9559eafb in pnm_computetuplefreqtable2 (pamP=0x7fbffff690,
tupleArray=0x0, maxsize=32767, newMaxval=255, countP=0x7fbffff4e4)
    at libpammap.c:538
#2  0x000000000040254c in computeHistogram (pamP=0x7fbffff690,
colorfreqtableP=0x7fbffff4e8, colorsP=0x7fbffff4e4, freqtableMaxvalP=0x7fbffff4d8)
    at pnmcolormap.c:655
#3  0x00000000004025fb in computeColorMapFromInput (pamP=0x7fbffff690,
allColors=1, reqColors=0, methodForLargest=LARGE_NORM, methodForRep=REP_CENTER_BOX,
    colormapP=0x7fbffff538, colorsP=0x7fbffff544, colormapMaxvalP=0x7fbffff530)
at pnmcolormap.c:713
#4  0x0000000000402c78 in main (argc=3, argv=0x7fbffff8f8) at pnmcolormap.c:907

Comment 5 Jindrich Novy 2005-05-30 10:41:22 EDT
*** Bug 139111 has been marked as a duplicate of this bug. ***
Comment 6 Jacques Rodary 2005-05-30 20:06:12 EDT
sorry: bug 139111 is definetly not a duplicate of this one: I had no problem with
netpbm-10.26.1, which isn't available anymore.   
Comment 7 Jacques Rodary 2005-05-31 04:02:39 EDT
and downgrading to 10.25-2 doesn't work for my problem of image conversion in
latex2html: I had to downgrade to netpbm-9.24-10.90.1
Comment 8 Jindrich Novy 2005-05-31 09:06:28 EDT
Created attachment 114988 [details]
Patch to fix the pnmcolormap segfault

The segfault is caused by the fact that variable "retval" is declared twice in
pnm_alloctupletable(), first in the global function context and secondly in a
separate block, where the memory is allocated. The culprit is the global
"retval" which is returned uninitialized. This can be seen from here:

tupletable
pnm_alloctupletable(const struct pam * const pamP,
		    unsigned int       const size) {

    tupletable retval;

    if (UINT_MAX / sizeof(struct tupleint) < size)
	pm_error("size %u is too big for arithmetic", size);
    else {
	unsigned int const mainTableSize = size * sizeof(struct tupleint *);
	unsigned int const tupleIntSize =
	    sizeof(struct tupleint) - sizeof(sample)
	    + pamP->depth * sizeof(sample);

	/* To save the enormous amount of time it could take to allocate
	   each individual tuple, we do a trick here and allocate everything
	   as a single malloc block and suballocate internally.
	*/
	if ((UINT_MAX - mainTableSize) / tupleIntSize < size)
	    pm_error("size %u is too big for arithmetic", size);
	else {
	    void * pool;
	    tupletable retval;
	    unsigned int i;

	    pool = malloc(mainTableSize + size * tupleIntSize);

	    retval = (tupletable) pool;

	    for (i = 0; i < size; ++i)
		retval[i] = (struct tupleint *)
		    ((char*)pool + mainTableSize + i * tupleIntSize);
	}
    }
    return retval;
}

The attached one-liner fixes this. It seems this mistake was for ages in the
upstream netpbm as I can see from Jacques Rodary's comment.
Comment 9 Jindrich Novy 2005-05-31 09:08:10 EDT
This is now fixed in devel and FC3 branch.

I'll close this ERRATA when FC3 update is released.
Comment 10 Jacques Rodary 2005-05-31 10:56:53 EDT
It's OK now with netpbm-10.27-4.FC3 already available with FC3-updates. Many
thanks. 

Note You need to log in before you can comment on or make changes to this bug.