Description of problem: I was trying to connect to VM using serial console, but failed with error "Jun 9, 2016 12:23:07 PM Failed to execute login on behalf - for user admin." in WEBUI. This is what I've received from CLI on my laptop: # ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.lab.tlv.redhat.com connect OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: Connecting to nsednev-he-1.qa.lab.tlv.redhat.com [10.35.97.61] port 2222. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/id_rsa type 1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5-etm none debug1: kex: client->server aes128-ctr hmac-md5-etm none debug1: kex: curve25519-sha256 need=16 dh_need=16 debug1: kex: curve25519-sha256 need=16 dh_need=16 debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: ssh_rsa_verify: signature correct debug1: Server host key: RSA-CERT d3:c0:9f:e9:eb:d4:4c:64:00:9d:1a:55:a2:86:8b:af debug1: checking without port identifier debug1: No matching CA found. Retry with plain key debug1: No matching CA found. Retry with plain key debug1: Host '[nsednev-he-1.qa.lab.tlv.redhat.com]:2222' is known and matches the RSA host key. debug1: Found key in /root/.ssh/known_hosts:2 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /root/.ssh/id_rsa debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: key_parse_private2: missing begin marker debug1: read PEM private key done: type RSA debug1: Authentication succeeded (publickey). Authenticated to nsednev-he-1.qa.lab.tlv.redhat.com ([10.35.97.61]:2222). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. debug1: Remote: Forced command. debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Remote: Forced command. debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Sending environment. debug1: Sending env LANG = en_US.UTF-8 debug1: Sending env LANGUAGE = debug1: Sending command: connect debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow reply 0 ERROR: Internal error debug1: channel 0: free: client-session, nchannels 1 Connection to nsednev-he-1.qa.lab.tlv.redhat.com closed. Transferred: sent 3760, received 4124 bytes, in 0.9 seconds Bytes per second: sent 4323.8, received 4742.4 debug1: Exit status 1 Version-Release number of selected component (if applicable): Engine: rhevm-4.0.0.2-0.1.el7ev.noarch rhevm-guest-agent-common-1.0.12-1.el7ev.noarch rhevm-branding-rhev-4.0.0-0.0.master.20160531161414.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhevm-setup-plugins-4.0.0-1.el7ev.noarch rhevm-doc-4.0.0-2.el7ev.noarch ovirt-engine-setup-plugin-ovirt-engine-4.0.0.2-0.1.el7ev.noarch ovirt-vmconsole-1.0.3-1.el7ev.noarch ovirt-engine-extension-aaa-jdbc-1.1.0-1.el7ev.noarch ovirt-engine-setup-base-4.0.0.2-0.1.el7ev.noarch ovirt-engine-websocket-proxy-4.0.0.2-0.1.el7ev.noarch ovirt-image-uploader-4.0.0-1.el7ev.noarch ovirt-engine-backend-4.0.0.2-0.1.el7ev.noarch ovirt-engine-tools-4.0.0.2-0.1.el7ev.noarch ovirt-engine-lib-4.0.0.2-0.1.el7ev.noarch ovirt-engine-dwh-setup-4.0.0-2.el7ev.noarch ovirt-log-collector-4.0.0-1.el7ev.noarch ovirt-engine-vmconsole-proxy-helper-4.0.0.2-0.1.el7ev.noarch ovirt-host-deploy-java-1.5.0-1.el7ev.noarch ovirt-engine-dbscripts-4.0.0.2-0.1.el7ev.noarch ovirt-engine-4.0.0.2-0.1.el7ev.noarch ovirt-engine-setup-plugin-websocket-proxy-4.0.0.2-0.1.el7ev.noarch ovirt-engine-tools-backup-4.0.0.2-0.1.el7ev.noarch ovirt-engine-userportal-4.0.0.2-0.1.el7ev.noarch ovirt-engine-setup-4.0.0.2-0.1.el7ev.noarch ovirt-vmconsole-proxy-1.0.3-1.el7ev.noarch ovirt-engine-restapi-4.0.0.2-0.1.el7ev.noarch ovirt-engine-cli-3.6.2.0-1.el7ev.noarch ovirt-engine-setup-plugin-ovirt-engine-common-4.0.0.2-0.1.el7ev.noarch ovirt-engine-extensions-api-impl-4.0.0.2-0.1.el7ev.noarch ovirt-iso-uploader-4.0.0-1.el7ev.noarch ovirt-engine-webadmin-portal-4.0.0.2-0.1.el7ev.noarch ovirt-engine-dwh-4.0.0-2.el7ev.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.0.2-0.1.el7ev.noarch ovirt-host-deploy-1.5.0-1.el7ev.noarch ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7ev.noarch Linux version 3.10.0-327.22.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon May 16 13:31:48 EDT 2016 Linux 3.10.0-327.22.1.el7.x86_64 #1 SMP Mon May 16 13:31:48 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Hosts: mom-0.5.4-1.el7ev.noarch ovirt-vmconsole-1.0.3-1.el7ev.noarch sanlock-3.2.4-2.el7_2.x86_64 libvirt-client-1.2.17-13.el7_2.5.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.15.x86_64 vdsm-4.18.1-11.gita92976e.el7ev.x86_64 ovirt-hosted-engine-setup-2.0.0-1.el7ev.noarch ovirt-host-deploy-1.5.0-1.el7ev.noarch ovirt-hosted-engine-ha-2.0.0-1.el7ev.noarch ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-vmconsole-host-1.0.3-1.el7ev.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7ev.noarch Linux version 3.10.0-327.22.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon May 16 13:31:48 EDT 2016 Linux 3.10.0-327.22.1.el7.x86_64 #1 SMP Mon May 16 13:31:48 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) How reproducible: 100% Steps to Reproduce: 1.Establish HE environment with at least one host. 2.Create VM and make sure it has all required permissions for serial console. 3.Enable serial virtIO console for VM. 4.Copy your public key to the engine. 5.Establish serial console connectivity from your PC to the engine. Actual results: Serial console connection fails. Expected results: Serial console connection should succeed. Additional info: sosreports from both hosts and the engine attached.
Created attachment 1166228 [details] sosreport from host alma03
Created attachment 1166229 [details] sosreport from engine
External link for sosreport from second host alma04 provided here, because of Bugzilla file size limitations: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88RVRHLURlcmVQNUk/view?usp=sharing
If you see error in engine log why are you pasting tons of data _other_ than e.g. excerpt from engine.log or webadmin events?
I was able to reproduce it and as far as I can tell it looks ok in vmconsole code, LoginOnBehalfCommand is invoked with the right userGuid which does exist. It looks like the command is simply failing when invoked from the vmconsole servlet now.
(In reply to Michal Skrivanek from comment #4) > If you see error in engine log why are you pasting tons of data _other_ than > e.g. excerpt from engine.log or webadmin events? I see several errors and providing fully detailed logs, so nothing will be missing, especially as my environment is running on top of the HE, and there are several related components on hosts and engine like ovirt-vmconsole-proxy-sshd service on engine and ovirt-vmconsole-host-sshd service on hosts.
Using SSH to a physical machine and to a VM should work the same. Are you sure this is related to HE and will not happen on non-HE setup?
(In reply to Doron Fediuck from comment #7) > Using SSH to a physical machine and to a VM should work the same. > Are you sure this is related to HE and will not happen on non-HE setup? Its not related to HE only, but was seen first on it. The authentication AAA component, which being used by serial-console is not working properly.
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Moving back to POST, this needs to be backported to ovirt-engine-4.0
Please specify in which version of ovirt-engine this bug was fixed.
Works for me on these components on host: ovirt-setup-lib-1.0.2-1.el7ev.noarch qemu-kvm-rhev-2.3.0-31.el7_2.16.x86_64 mom-0.5.4-1.el7ev.noarch rhev-release-4.0.0-19-001.noarch vdsm-4.18.4-2.el7ev.x86_64 ovirt-vmconsole-host-1.0.3-1.el7ev.noarch ovirt-hosted-engine-ha-2.0.0-1.el7ev.noarch ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch libvirt-client-1.2.17-13.el7_2.5.x86_64 ovirt-host-deploy-1.5.0-1.el7ev.noarch ovirt-hosted-engine-setup-2.0.0.2-1.el7ev.noarch sanlock-3.2.4-2.el7_2.x86_64 ovirt-vmconsole-1.0.3-1.el7ev.noarch rhev-release-4.0.1-1-001.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Engine: rhevm-doc-4.0.0-2.el7ev.noarch rhevm-setup-plugins-4.0.0.1-1.el7ev.noarch rhevm-spice-client-x64-msi-4.0-2.el7ev.noarch rhevm-4.0.2-0.2.rc1.el7ev.noarch rhev-release-4.0.0-19-001.noarch rhev-release-4.0.1-1-001.noarch rhevm-guest-agent-common-1.0.12-2.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhevm-branding-rhev-4.0.0-2.el7ev.noarch rhevm-spice-client-x86-msi-4.0-2.el7ev.noarch rhev-guest-tools-iso-4.0-2.el7ev.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) I've successfully established serial-console connectivity with one of the VMs, that was running on host and my user passed authentication on engine.
It should be in 4.0.1
Since the problem described in this bug report should be resolved in oVirt 4.0.1 released on July 19th 2016, it has been closed with a resolution of CURRENT RELEASE. For information on the release, and how to update to this release, follow the link below. If the solution does not work for you, open a new bug report. http://www.ovirt.org/release/4.0.1/