Murray McAllister of the Red Hat Security Response Team reported the following potential vulnerability in potrace:


There is a possible issue in potrace-1.11-1.fc18.x86_64. The attached 
bmp file (1.bmp) triggers it. I suspect less memory is allocated than 
expected in bm_new() due to integer overflow. I have not investigated it 
closely or the rest of the application yet.

$ potrace 1.bmp
*** glibc detected *** potrace: free(): invalid next size (fast): 
0x0000000001263580 ***
======= Backtrace: =========
/usr/lib64/libc.so.6[0x3c6de7ca8e]
/usr/lib64/libpotrace.so.0[0x354ae0612f]
/usr/lib64/libpotrace.so.0(potrace_trace+0x106)[0x354ae06356]
potrace[0x40361b]
potrace[0x402c9f]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x3c6de21a05]
potrace[0x40303d]

..

==2042== Memcheck, a memory error detector
==2042== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==2042== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==2042== Command: potrace 1.bmp
==2042==
==2042== Invalid read of size 8
==2042==    at 0x405F2D: bm_read (bitmap_io.c:615)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==  Address 0x4c14680 is 0 bytes after a block of size 0 alloc'd
==2042==    at 0x4A0887C: malloc (vg_replace_malloc.c:270)
==2042==    by 0x405069: bm_read (bitmap.h:66)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==
==2042== Invalid write of size 8
==2042==    at 0x405F31: bm_read (bitmap_io.c:615)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==  Address 0x4c14680 is 0 bytes after a block of size 0 alloc'd
==2042==    at 0x4A0887C: malloc (vg_replace_malloc.c:270)
==2042==    by 0x405069: bm_read (bitmap.h:66)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==
==2042== Invalid read of size 8
==2042==    at 0x405585: bm_read (bitmap_io.c:615)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==  Address 0x4c14688 is 8 bytes after a block of size 0 alloc'd
==2042==    at 0x4A0887C: malloc (vg_replace_malloc.c:270)
==2042==    by 0x405069: bm_read (bitmap.h:66)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==
==2042== Invalid write of size 8
==2042==    at 0x405589: bm_read (bitmap_io.c:615)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==  Address 0x4c14688 is 8 bytes after a block of size 0 alloc'd
==2042==    at 0x4A0887C: malloc (vg_replace_malloc.c:270)
==2042==    by 0x405069: bm_read (bitmap.h:66)
==2042==    by 0x4035A0: process_file.isra.3 (main.c:1056)
==2042==    by 0x402C9E: main (main.c:1212)
==2042==
potrace: warning: 1.bmp: premature end of file

valgrind: m_mallocfree.c:268 (mk_plain_bszB): Assertion 'bszB != 0' failed.
valgrind: This is probably caused by your program erroneously writing 
past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.

==2042==    at 0x38057AAF: report_and_quit (m_libcassert.c:235)
==2042==    by 0x38057BF2: vgPlain_assert_fail (m_libcassert.c:309)
==2042==    by 0x380009EC: mk_plain_bszB.part.5 (m_mallocfree.c:268)
==2042==    by 0x38063C87: vgPlain_arena_malloc (m_mallocfree.c:1563)
==2042==    by 0x380299A4: vgMemCheck_new_block (mc_malloc_wrappers.c:263)
==2042==    by 0x38029B3A: vgMemCheck_malloc (mc_malloc_wrappers.c:301)
==2042==    by 0x3809E490: vgPlain_scheduler (scheduler.c:1667)
==2042==    by 0x380AD6F9: run_a_thread_NORETURN (syswrap-linux.c:103)

sched status:
   running_tid=1

Thread 1: status = VgTs_Runnable
==2042==    at 0x4A0887C: malloc (vg_replace_malloc.c:270)
==2042==    by 0x354AE062B0: potrace_trace (potracelib.c:68)
==2042==    by 0x40361A: process_file.isra.3 (main.c:1100)
==2042==    by 0x402C9E: main (main.c:1212)