Looking at the end binary, it looks like the segvCB() signal handler is compiled out to a NOP.

	(gdb) disassemble segvCB
	Dump of assembler code for function segvCB:
	   0x0000000000400c30 <+0>:     repz retq 
	End of assembler dump.


It looks like the macro below, is not defined at compile time:

	void
	segvCB(int sig)
	{
	#if defined HAVE_XORG_SERVER_1_1_0
	   exit(1);
	#endif
	}


I'm attaching a patch to move the macro to the signal handler registration. That way, if the macro is not defined, the end result would be a death by SIGSEGV, instead of the endless loop.

- Kyle Walker

Marking Regression, as the issue is not present in:

    xorg-x11-drv-vmmouse-13.0.0-11.el7

$ gdb ./usr/bin/vmmouse_detect
(gdb) disassemble segvCB
Dump of assembler code for function segvCB:
   0x0000000000400670 <+0>:     sub    $0x8,%rsp
   0x0000000000400674 <+4>:     mov    $0x1,%edi
   0x0000000000400679 <+9>:     callq  0x400530 <exit@plt>
End of assembler dump.

- Kyle Walker

Redacted the previous patch.

The source of the failure is that in the latest update the following patch was backported:

    Subject: [PATCH vmmouse] Back off if we detect a vmmouse kernel driver v3

    If a vmmouse kernel driver is active, vmmouse input is handled by the Xorg
    evdev driver and not by the vmmouse driver, so make sure the vmmouse_detect
    utility doesn't detect a vmmouse if a kernel driver is active.

    v2: Change the vmmouse kernel device name, fix comment.
    v3: Fix up libudev error handling.


In that patch, the following is present:

	diff --git a/tools/vmmouse_detect.c b/tools/vmmouse_detect.c
	index cfb92e1..6402b16 100644
	--- a/tools/vmmouse_detect.c
	+++ b/tools/vmmouse_detect.c
	@@ -30,9 +30,7 @@
	 #include <signal.h>
	 #include "vmmouse_client.h"

	-#ifdef HAVE_CONFIG_H
	-#include "config.h"
	-#endif
	+extern int vmmouse_uses_kernel_driver(void);

	 void
	 segvCB(int sig)


The above erroneously removes the "ifdef HAVE_CONFIG_H" include statement. When the above is reverted, the previous behaviour is present. Soon to append a corrected patch for the above.

- Kyle Walker

Created attachment 1238054 [details]
Corrected patch to include the config.h header file.

Attaching patch to correct the missing config.h header file. Test build running below:

build (rhel-7.3-z-test, /rpms/xorg-x11-drv-vmmouse:6b363d659e84289bc5f75e4529c4b112c34bda54) | Task Info | Brew
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12315189

Testing results:

	xorg-x11-drv-vmmouse-13.0.0-12.el7_3.bz1410582.x86_64.rpm

	$ rpm2cpio xorg-x11-drv-vmmouse-13.0.0-12.el7_3.bz1410582.x86_64.rpm | cpio -idmv
	./usr/bin/vmmouse_detect
	./usr/lib/udev/rules.d/69-xorg-vmmouse.rules
	./usr/lib64/xorg/modules/input/vmmouse_drv.so
	./usr/share/X11/xorg.conf.d/50-vmmouse.conf
	./usr/share/man/man1/vmmouse_detect.1.gz
	./usr/share/man/man4/vmmouse.4.gz
	69 blocks

	$ ./usr/bin/vmmouse_detect 
	$ echo $?
	1

With the binary function disassembly:

	$ gdb ./usr/bin/vmmouse_detect
	<snip>
	(gdb) disassem segvCB
	Dump of assembler code for function segvCB:
	   0x0000000000400c80 <+0>:	sub    $0x8,%rsp
	   0x0000000000400c84 <+4>:	mov    $0x1,%edi
	   0x0000000000400c89 <+9>:	callq  0x400b40 <exit@plt>
	End of assembler dump.

- Kyle Walker

Rebased driver appears to have the right code in place.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1905