Red Hat Bugzilla – Bug 1537758
CVE-2018-5785 openjpeg: integer overflow in opj_j2k_setup_encoder function in openjp2/j2k.c
Last modified: 2018-01-25 04:59:29 EST
A flaw was found in OpenJPEG 2.3.0, there is an integer overflow caused by an out-of-bounds left shift in the opj_j2k_setup_encoder function (openjp2/j2k.c). Remote attackers could leverage this vulnerability to cause a denial of service via a crafted bmp file. Reference: https://github.com/uclouvain/openjpeg/issues/1057
Created mingw-openjpeg2 tracking bugs for this issue: Affects: fedora-all [bug 1537761] Created openjpeg tracking bugs for this issue: Affects: epel-all [bug 1537762] Affects: fedora-all [bug 1537759] Created openjpeg2 tracking bugs for this issue: Affects: fedora-all [bug 1537760]
Analysis: Running openjpeg compiled with UBSAN, i observe the following: [huzaifas@babylon bin]$ ./opj_compress -n 1 -i /tmp/openjpeg_2-3_opj_compress_integer-overflow_opj_j2k_setup_encoder.bmp -o /tmp/null.j2k /NotBackedUp/oss/openjpeg/src/lib/openjp2/j2k.c:7304:48: runtime error: shift exponent 4294967295 is too large for 32-bit type 'int' [INFO] tile number 1 / 1 /NotBackedUp/oss/openjpeg/src/lib/openjp2/tcd.c:2382:32: runtime error: signed integer overflow: 0 - -2147483648 cannot be represented in type 'int [33]' /NotBackedUp/oss/openjpeg/src/lib/openjp2/mct.c:109:31: runtime error: signed integer overflow: -2147483648 * 2 cannot be represented in type 'int' /NotBackedUp/oss/openjpeg/src/lib/openjp2/mct.c:109:36: runtime error: signed integer overflow: -2147483648 + -2147483648 cannot be represented in type 'int' [INFO] Generated outfile /tmp/null.j2k encode time: 1 ms Which represents multiple integer overflow when converting from BMP to J2K format. However it seems that in the end the operation ends gracefully. There are no related buffer overflows etc. The maximum security impact seems to be limited to invalid conversion to J2K format.