An issue was discovered in LibVNCServer before 0.9.13. libvncserver/scale.c has a pixel_value integer overflow. Upstream commit: https://github.com/LibVNC/libvncserver/commit/a6788d1da719ae006605b78d22f5a9f170b423af
Created libvncserver tracking bugs for this issue: Affects: epel-7 [bug 1860366] Affects: fedora-all [bug 1860365]
in libvncserver/scale.c, rfbScaledScreenUpdateRect() has `pixel_value += (srcptr2[z] << (8 * z));` which performs a shift operation on a 64-bit value using an 8-bit variable, which could cause an unsigned integer wrap. A few lines down, there is a write: `pixel_value = ((red & redMax) << redShift) | ((green & greenMax) << greenShift) | ((blue & blueMax) << blueShift);`. In the event of an unsigned wrap, the write could be incorrect and potentially cause a crash.