Tom Lane (tgl) found an issue in ImageMagick that is also present in GraphicsMagick. Basically CVE-2011-3026 deals with libpng memory allocation, limitations have been added so that a bad PNG can't cause the system to allocate a lot of memory causing a denial of service. However on further investigation of ImageMagick Tom Lane found that PNG malloc function (Magick_png_malloc) in turn calls AcquireMagickMemory with an improper size argument: #ifdef PNG_USER_MEM_SUPPORTED static png_voidp Magick_png_malloc(png_structp png_ptr,png_uint_32 size) { (void) png_ptr; return((png_voidp) AcquireMagickMemory((size_t) size)); } Similar code is present in GraphicsMagick: #ifdef PNG_USER_MEM_SUPPORTED static png_voidp png_IM_malloc(png_structp png_ptr,png_uint_32 size) { (void) png_ptr; return MagickAllocateMemory(png_voidp,(size_t) size); } This is incorrect, the size argument should be declared png_alloc_size_t according to 1.5, or png_size_t according to 1.2. "As this function stands, it invisibly does the wrong thing for any request over 4GB. On big-endian architectures it very possibly will do the wrong thing even for requests less than that. So the reason why the hard-wired 4GB limit prevents a core dump is that it masks the ABI mismatch here." So basically we have memory allocations problems that can probably lead to a denial of service.
Created GraphicsMagick tracking bugs for this issue Affects: fedora-all [bug 844106]
Created GraphicsMagick tracking bugs for this issue Affects: epel-all [bug 844107]
I'm told upstream has already committed a fix for this, so you should be able to pull a patch out of their SCM.
Upstream patch: http://graphicsmagick.hg.sourceforge.net/hgweb/graphicsmagick/graphicsmagick/rev/d6e469d02cd2
Just to note, that CVE-2012-3437 is for this flaw in ImageMagick, while CVE-2012-3438 is for this flaw in GraphicsMagick. The initial description does not make it clear that separate CVEs were assigned for each.