Description of problem: Impossible to reconnect via serial-console to guest vm if previous session unexpectedly dropped (stuck). I've established the serial-console connection to the RHEL7.2 guest VM successfully, then just closed the ssh session window, while session was still active, thus making that session remain active, although it actually have no one connected at the cusomer's side. Then I tried to re-establish the connection to the same RHEL7.2 guest VM over serial console and failed to connect, as my previous serial-console session was still active (stuck), I received a black screen and nothing more, I couldn't kick out the stuck session in order to connect with the new session over an old session. Version-Release number of selected component (if applicable): Engine: rhevm-3.6.0.3-0.1.el6.noarch rhevm-guest-agent-common-1.0.11-2.el6ev.noarch qemu-guest-agent-0.12.1.2-2.479.el6_7.2.x86_64 jboss-as-console-2.5.10-4.Final_redhat_2.1.ep6.el6.noarch ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch ovirt-vmconsole-1.0.0-1.el6ev.noarch rhevm-setup-plugin-vmconsole-proxy-helper-3.6.0.3-0.1.el6.noarch rhevm-vmconsole-proxy-helper-3.6.0.3-0.1.el6.noarch Linux version 2.6.32-573.7.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Sep 10 13:42:16 EDT 2015 Host: ovirt-vmconsole-1.0.0-0.0.6.master.el7ev.noarch ovirt-release36-001-0.5.beta.noarch mom-0.5.1-2.el7.noarch ovirt-hosted-engine-setup-1.3.1-0.0.master.20151020145724.git565c3f9.el7.centos.noarch ovirt-setup-lib-1.0.0-1.20150922141000.git147e275.el7.centos.noarch ovirt-host-deploy-1.5.0-0.0.master.20151015221110.gitc2abfed.el7.noarch ovirt-release36-snapshot-001-0.5.beta.noarch vdsm-4.17.10-13.gite438b03.el7.noarch qemu-kvm-rhev-2.3.0-31.el7.x86_64 libvirt-client-1.2.17-12.el7.x86_64 ovirt-hosted-engine-ha-1.3.1-1.20151016090950.git5ea5093.el7.noarch sanlock-3.2.4-1.el7.x86_64 ovirt-engine-sdk-python-3.6.0.4-0.1.20151014.git117764a.el7.centos.noarch ovirt-vmconsole-host-1.0.0-0.0.6.master.el7ev.noarch Linux version 3.10.0-324.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Oct 13 21:28:31 EDT 2015 How reproducible: 100% Steps to Reproduce: 1.Connect to the RHEL7.2 guest vm via serial console from your laptop. 2.Close the ssh terminal by closing it's window in your laptop. 3.Try reconnecting to the same RHEL7.2 guest vm via serial console from your laptop. Actual results: You may choose the VM, but can't connect to it as previously established serial-console session is stuck. Expected results: Stuck serial-console sessions should be kicked off by the new serial-console sessions or there should be manual possibility of dropping the stuck sessions. Additional info:
This has been tested. After a tcp timeout the sshd process should detect this condition and exit, freeing the console.
(In reply to Alon Bar-Lev from comment #1) > This has been tested. > After a tcp timeout the sshd process should detect this condition and exit, > freeing the console. After some time and after migrating the guest-VM to another host, session was freed and I re-established serial connectivity with guest-VM, although I was not asked for login/password ans guest-VM had no Idea of session initiator was disconnected and then reconnected.
Just for the reference: 1)tried to restart VM and reconnect - failed, session still stuck; 2) tried to restart console service on engine and then reconnect- failed with session still stuck; 3) tried to restart console on host and reconnect - failed, session still stuck; 4) tried to restart VM and reconnect - failed, session still stuck; 5)powered-off the VM and tried to reconnect-debug1: Exit status 255 I've tried this on customized TCP 3334 ports on both engine to laptop and engine to host. [root@nsednev ~]# ssh -v -t -i $HOME/.ssh/id_rsa -p 3334 ovirt-vmconsole.lab.tlv.redhat.com connect OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: Connecting to nsednev-he-1.qa.lab.tlv.redhat.com [10.35.97.61] port 3334. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/id_rsa type 1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5-etm none debug1: kex: client->server aes128-ctr hmac-md5-etm none debug1: kex: curve25519-sha256 need=16 dh_need=16 debug1: kex: curve25519-sha256 need=16 dh_need=16 debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: ssh_rsa_verify: signature correct debug1: Server host key: RSA-CERT 4f:c9:ce:8b:74:ef:2e:2e:95:ad:33:28:1e:83:44:04 debug1: checking without port identifier debug1: No matching CA found. Retry with plain key debug1: No matching CA found. Retry with plain key debug1: Host '[nsednev-he-1.qa.lab.tlv.redhat.com]:3334' is known and matches the RSA host key. debug1: Found key in /root/.ssh/known_hosts:4 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /root/.ssh/id_rsa debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: key_parse_private2: missing begin marker debug1: read PEM private key done: type RSA debug1: Authentication succeeded (publickey). Authenticated to nsednev-he-1.qa.lab.tlv.redhat.com ([10.35.97.61]:3334). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. debug1: Remote: Forced command. debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Remote: Forced command. debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Sending environment. debug1: Sending env LANG = en_US.UTF-8 debug1: Sending env LANGUAGE = debug1: Sending command: connect Available Serial Consoles: 00 A[6a1183b9-24ea-444e-bfd8-3a117dbd9e4e] 01 B[07d90003-c6d5-4bc0-8642-c001f892aa5f] 02 HostedEngine[280c4195-08ee-4385-a031-6288702a6aad] 03 VM1[43500052-354c-4217-957d-ca0d9d7612db] SELECT> 03 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow reply 0 ssh: connect to host alma03.qa.lab.tlv.redhat.com port 3334: Connection refused debug1: channel 0: free: client-session, nchannels 1 Connection to nsednev-he-1.qa.lab.tlv.redhat.com closed. Transferred: sent 3820, received 4476 bytes, in 3.0 seconds Bytes per second: sent 1264.8, received 1482.0 debug1: Exit status 255 Components on host: ovirt-setup-lib-1.0.2-1.el7ev.noarch rhev-release-4.0.0-19-001.noarch ovirt-vmconsole-host-1.0.3-1.el7ev.noarch ovirt-hosted-engine-ha-2.0.0-1.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch qemu-kvm-rhev-2.3.0-31.el7_2.17.x86_64 ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch libvirt-client-1.2.17-13.el7_2.5.x86_64 ovirt-host-deploy-1.5.0-1.el7ev.noarch ovirt-imageio-daemon-0.3.0-0.el7ev.noarch ovirt-hosted-engine-setup-2.0.0.2-1.el7ev.noarch sanlock-3.2.4-2.el7_2.x86_64 mom-0.5.5-1.el7ev.noarch ovirt-vmconsole-1.0.3-1.el7ev.noarch rhev-release-4.0.1-1-001.noarch vdsm-4.18.5.1-1.el7ev.x86_64 Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) On engine: rhevm-doc-4.0.0-2.el7ev.noarch rhevm-setup-plugins-4.0.0.1-1.el7ev.noarch rhevm-spice-client-x64-msi-4.0-2.el7ev.noarch rhevm-4.0.2-0.2.rc1.el7ev.noarch rhev-release-4.0.0-19-001.noarch rhev-release-4.0.1-1-001.noarch rhevm-guest-agent-common-1.0.12-2.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhevm-branding-rhev-4.0.0-2.el7ev.noarch rhevm-spice-client-x86-msi-4.0-2.el7ev.noarch rhev-guest-tools-iso-4.0-2.el7ev.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo)
I see that now this message is being printed continuously on open serial console session with "-v" parameter: debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive reply 1 I suppose that this bug is fixed partially now, please consider changing the status to WONTFIX, as eventually it was partially fixed. Looks like message is all about KeepAlive for the ssh session, so if it's closed unexpectedly, then dropped from the engine to the customer.
closing resolution as per https://bugzilla.redhat.com/show_bug.cgi?id=1278466#c4