Bug 56609 - pnmdepth bogosity for PGM maxval=65535 -> 255 conversion
pnmdepth bogosity for PGM maxval=65535 -> 255 conversion
Product: Red Hat Linux
Classification: Retired
Component: netpbm (Show other bugs)
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Phil Knirsch
Brian Brock
Depends On:
  Show dependency treegraph
Reported: 2001-11-21 17:12 EST by Roland Roberts
Modified: 2015-03-04 20:09 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2001-11-25 23:02:52 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Roland Roberts 2001-11-21 17:12:46 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.5) Gecko/20011031

Description of problem:
If you create a PGM with maxval=65535 and then use pnmdepth to scale it,
the resulting PGM is not scaled properly due to integer overflow in the
scaling algorithm.  In particular, the following lines in pnndepth.c are
the culprit:

    for ( i = 0; i <= maxval; ++i )
       newvals[i] = ( i * newmaxval + maxval / 2 ) / maxval;

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Attached in the "additional information" section is a perl script which
creates a 16-bit PGM.  Run it via

    vignette.pl > foo16.pgm

Use something like xv to view the PGM (xv happily handles 16-bit PGM, the
GIMP does not).

Scale the PGM via

    pnmdepth 255 foo16.pgm > foo8.pgm

Now view the resulting PGM with xv.

Actual Results:  The original PGM is a smooth grayscale image running from
about 70% white at the edge to 100% white in the center.  The pnmdepth
scaled image looks like I was playing with some fractal generator.

Expected Results:  The pnmdepth scaled image should have looked
substantially the same as the original.

Additional info:

The "easy" fix is to promote the calculation to double instead of integer
but that is probably a bad thing on a real 386 with no floating-point
support.  Still, the mapping is calculated only once and them used in a
lookup table, so it isn't a total killer.

Here is the perl script (watch for spurious line wraps):

#! /usr/bin/perl

$opt{maxval} = 65535;
$opt{xsize}  = 160;
$opt{ysize}  = 120;
$opt{minval} = 0.7;
printf ("P5 %d %d %d\n", $opt{xsize}, $opt{ysize}, $opt{maxval});

$h = 0.5 * sqrt($opt{xsize} * $opt{xsize} + $opt{ysize} * $opt{ysize});
$d_square = $h * $h * sqrt($opt{minval}) / (1.0 - sqrt($opt{minval}));
$d = $h * sqrt(sqrt($opt{minval})) / sqrt(1.0 - sqrt($opt{minval}));

for ($j = 0, $y = 0.5*($opt{ysize}-1); $j < $opt{ysize}; $j++, $y -= 1.0) {
    for ($i = 0, $x = -0.5*($opt{xsize}-1); $i < $opt{xsize}; $i++, $x +=
1.0) {
	$h_square = $x*$x + $y*$y;
	$v = $h_square / ($d_square + $h_square);
	$v = 1 - $v;
	$v *= $v;
	$v *= $opt{maxval};
	print pack('S', int($v));
Comment 1 Roland Roberts 2001-11-21 17:26:06 EST
My analysis of the bug is wrong; the code in question can't be overflowing as
newval[] should be of type unsigned long is more than big enough to handle the
65535*255+65535/2 calculation for i=maxval.

I'll run it under the debugger and see what I can figure out....
Comment 2 Roland Roberts 2001-11-25 23:02:44 EST
It's an endian problem.  But there is no endian setting which will result in an
image which displays correctly via xv or ee as a 16-bit grayscale and still
display correctly as an 8-bit grayscale when converted via pnmdepth.  I *can*
get a correct image by forcing keeping everything as ascii format pgm files
(i.e., P2 instead of P5).
Comment 3 Roland Roberts 2001-11-25 23:49:14 EST
Apparently xv doesn't read 16-bit pgm format correctly.  both ee and display
(ImageMagick) agree on the image content and a careful reading of the netpbm
code indicates that 16-bit raw should be in network byte-order.  So the problem
is *not* in netpbm but rather in xv.

Note You need to log in before you can comment on or make changes to this bug.