Bug 844105 (CVE-2012-3438)

Tom Lane (tgl) found an issue in ImageMagick that is also present 
in GraphicsMagick. Basically CVE-2011-3026 deals with libpng memory 
allocation, limitations have been added so that a bad PNG can't cause the 
system to allocate a lot of memory causing a denial of service. However on 
further investigation of ImageMagick Tom Lane found that PNG malloc 
function (Magick_png_malloc) in turn calls AcquireMagickMemory with an 
improper size argument:

#ifdef PNG_USER_MEM_SUPPORTED
static png_voidp Magick_png_malloc(png_structp png_ptr,png_uint_32 size)
{
  (void) png_ptr;
  return((png_voidp) AcquireMagickMemory((size_t) size));
}

Similar code is present in GraphicsMagick:

#ifdef PNG_USER_MEM_SUPPORTED
static png_voidp png_IM_malloc(png_structp png_ptr,png_uint_32 size)
{
  (void) png_ptr;
  return MagickAllocateMemory(png_voidp,(size_t) size);
}

This is incorrect, the size argument should be declared png_alloc_size_t 
according to 1.5, or png_size_t according to 1.2.

"As this function stands, it invisibly does the wrong thing for any request 
over 4GB.  On big-endian architectures it very possibly will do the wrong 
thing even for requests less than that. So the reason why the hard-wired 4GB 
limit prevents a core dump is that it masks the ABI mismatch here."

So basically we have memory allocations problems that can probably lead to a 
denial of service.