An issue was discovered in LibVNCServer before 0.9.13. libvncserver/scale.c has a pixel_value integer overflow.
Created libvncserver tracking bugs for this issue:
Affects: epel-7 [bug 1860366]
Affects: fedora-all [bug 1860365]
in libvncserver/scale.c, rfbScaledScreenUpdateRect() has `pixel_value += (srcptr2[z] << (8 * z));` which performs a shift operation on a 64-bit value using an 8-bit variable, which could cause an unsigned integer wrap. A few lines down, there is a write: `pixel_value = ((red & redMax) << redShift) | ((green & greenMax) << greenShift) | ((blue & blueMax) << blueShift);`. In the event of an unsigned wrap, the write could be incorrect and potentially cause a crash.