Description of problem:
When hiding and showing the second display of a guest, sometimes the input of the host gets into a strange state which doesn't allow you to move windows around or open menus on certain windows, etc.

Version-Release number of selected component (if applicable):
git commit 67ed4dff9829a1d957aebba9223028f6805dfafb

How reproducible:
always

Steps to Reproduce:
1. Set up a guest with 2 displays
2. Connect to guest with virt/remote-viewer.  2 display windows should be opened.
3. Click "View > Displays > Display 2" to hide the second display window
4. Now repeate step 3 to open the second display window again

Actual results:
The second display window is displayed, but that window cannot be moved or resized, and e.g. the gnome-shell overview cannot be activated

Expected results:
Window is displayed and functional

Additional info:
Activating a menu in the second display window seems to break the mouse grab and the window can then be moved / resized, etc.  The issue also doesn't happen if you run the program with SPICE_NOGRAB=1