Say I am using a 3g connection and SSH -X to my host machine, where I run virt-manager. If I interact with a guest using the vnc console, and then go through a physical tunnel and lose my 3g connection, I am stuck: I cannot get out of the console since Ctrl+Alt are waiting on a reply from the host.
I would recommend either using libvirt remote management capabilities and run virt-manager on your local machine, or ctrl-alt-f2 to drop to a terminal and kill the SSH process (infact I think switching like this will release the mouse for you anyways). Other than those workarounds I don't think there is much we can do if you are running the app remotely and the connection drops. Closing as CANTFIX.