In LibTIFF 4.0.8, there is a denial of service vulnerability in the TIFFOpen function. A crafted input will lead to a denial of service attack. During the TIFFOpen process, td_imagelength is not checked. The value of td_imagelength can be directly controlled by an input file. In the ChopUpSingleUncompressedStrip function, the _TIFFCheckMalloc function is called based on td_imagelength. If we set the value of td_imagelength close to the amount of system memory, it will hang the system or trigger the OOM killer.
Created libtiff tracking bugs for this issue:
Affects: fedora-all [bug 1475531]
Created mingw-libtiff tracking bugs for this issue:
Affects: fedora-all [bug 1475532]
LibTIFF simply tries to allocate the memory based on the information in the image. If there's not enough RAM, the OOM killer steps in and terminates the process. If you have enough RAM, all is fine.
Although one could implement mechanisms to catch this corner case early in order to handle it in a more graceful manner, I'm not sure if the library itself is the right place for that - I'll leave that to upstream.
Red Hat Product Security determined that this flaw was not a security vulnerability. See the Bugzilla link for more details.