Bug 1261519
Summary: | Can't connect to ovirt-vmconsole-proxy over ssh from my RHEL6.6 client node | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Nikolai Sednev <nsednev> | ||||||||||
Component: | ovirt-engine | Assignee: | Francesco Romani <fromani> | ||||||||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Nikolai Sednev <nsednev> | ||||||||||
Severity: | urgent | Docs Contact: | |||||||||||
Priority: | high | ||||||||||||
Version: | 3.6.0 | CC: | fromani, gklein, lsurette, mavital, mgoldboi, michal.skrivanek, nsednev, rbalakri, Rhev-m-bugs, srevivo, ykaul | ||||||||||
Target Milestone: | ovirt-3.6.0-rc3 | Keywords: | Triaged | ||||||||||
Target Release: | 3.6.0 | ||||||||||||
Hardware: | x86_64 | ||||||||||||
OS: | Linux | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2016-04-20 01:34:02 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | 1262003 | ||||||||||||
Bug Blocks: | 1223671, 1255222 | ||||||||||||
Attachments: |
|
Description
Nikolai Sednev
2015-09-09 14:04:30 UTC
Created attachment 1071776 [details]
engine.log
This is ssh-agent issue. Maybe it's just me, but I don't see how can this can be RHEV bug. You should be able to workaround it easily using a different ssh agent, for example the default one which comes with ssh itself: man ssh-agent See these instructions for a workaround. This is for github, but it is clearly explained, and the tools are the same. https://help.github.com/articles/error-agent-admitted-failure-to-sign/ I've tried also the WA provided here https://help.github.com/articles/error-agent-admitted-failure-to-sign/ and it didn't worked for me: ssh-add Identity added: /home/nsednev/.ssh/id_rsa (/home/nsednev/.ssh/id_rsa) $ ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222. debug1: Connection established. debug1: identity file /home/nsednev/.ssh/id_rsa type 1 debug1: identity file /home/nsednev/.ssh/id_rsa-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.3 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: ssh_rsa_verify: signature correct debug1: checking without port identifier debug1: No matching CA found. Retry with plain key debug1: No matching CA found. Retry with plain key debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key. debug1: Found key in /home/nsednev/.ssh/known_hosts:20 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/nsednev/.ssh/id_rsa debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell" accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id" debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Server accepts key: pkalg ssh-rsa blen 277 debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell" accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id" debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Authentication succeeded (publickey). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. debug1: Sending environment. debug1: Sending env XMODIFIERS = @im=none debug1: Sending env LANG = en_US.utf8 debug1: Sending env LANGUAGE = debug1: Sending command: connect debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow reply 0 /bin/sh: Permission denied debug1: channel 0: free: client-session, nchannels 1 Connection to 10.35.160.204 closed. Transferred: sent 3048, received 4112 bytes, in 0.3 seconds Bytes per second: sent 9585.7, received 12931.8 debug1: Exit status 1 Disabling the Selinux on engine not helped as well as also restarting the engine. I also tried to connect to ssh serial console from one of my pure cli RHEL7.1 servers and also failed: ]# ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 list OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/id_rsa type 1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: ssh_rsa_verify: signature correct debug1: Server host key: RSA-CERT 04:52:9f:4b:cf:11:67:8c:8d:ee:1b:48:84:2f:dc:b7 debug1: checking without port identifier debug1: No matching CA found. Retry with plain key debug1: No matching CA found. Retry with plain key debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key. debug1: Found key in /root/.ssh/known_hosts:2 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /root/.ssh/id_rsa debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell" accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id" debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: key_parse_private2: missing begin marker debug1: read PEM private key done: type RSA debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell" accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id" debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Authentication succeeded (publickey). Authenticated to 10.35.160.204 ([10.35.160.204]:2222). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. debug1: Sending environment. debug1: Sending env XMODIFIERS = @im=none debug1: Sending env LANG = en_US.utf8 debug1: Sending env LANGUAGE = debug1: Sending command: list 8ab54a0a-7eb1-4112-9456-0499e654ed9a test debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow reply 0 debug1: channel 0: free: client-session, nchannels 1 Connection to 10.35.160.204 closed. Transferred: sent 4040, received 4128 bytes, in 0.6 seconds Bytes per second: sent 6313.9, received 6451.5 debug1: Exit status 0 Disabling Selinux on engine actually helped, as now I could get the VM running and visible by the engine: Available Serial Consoles: 00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a] But connecting to it still failed: Available Serial Consoles: 00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a] SELECT> 00 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow reply 0 ssh: connect to host 10.35.117.22 port 2223: Connection refused debug1: channel 0: free: client-session, nchannels 1 Connection to 10.35.160.204 closed. Transferred: sent 4728, received 5184 bytes, in 85.0 seconds Bytes per second: sent 55.7, received 61.0 debug1: Exit status 255 There was found that service ovirt-vmconsole-host-sshd wasn't running on host, so it was restarted and the I've tried again to connect to the VM, failed because of : Certificate invalid: name is not a listed principal <fromani_> Host key verification failed. -> need to reinstall the host from engine webadmin <fromani_> so that certificate is enrolled again I reinstalled both hosts and retried, the result was also failure: # ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/id_rsa type 1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: ssh_rsa_verify: signature correct debug1: Server host key: RSA-CERT 04:52:9f:4b:cf:11:67:8c:8d:ee:1b:48:84:2f:dc:b7 debug1: checking without port identifier debug1: No matching CA found. Retry with plain key debug1: No matching CA found. Retry with plain key debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key. debug1: Found key in /root/.ssh/known_hosts:2 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /root/.ssh/id_rsa debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell" accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id" debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: key_parse_private2: missing begin marker debug1: read PEM private key done: type RSA debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell" accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id" debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc file execution disabled. debug1: Remote: X11 forwarding disabled. debug1: Authentication succeeded (publickey). Authenticated to 10.35.160.204 ([10.35.160.204]:2222). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. debug1: Sending environment. debug1: Sending env XMODIFIERS = @im=none debug1: Sending env LANG = en_US.utf8 debug1: Sending env LANGUAGE = debug1: Sending command: connect Available Serial Consoles: 00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a] SELECT> 00 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow reply 0 Certificate invalid: name is not a listed principal Host key verification failed. debug1: channel 0: free: client-session, nchannels 1 Connection to 10.35.160.204 closed. Transferred: sent 4152, received 4384 bytes, in 3.3 seconds Bytes per second: sent 1240.2, received 1309.5 debug1: Exit status 255 iptables on host are fine: # iptables -n -L -v Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 28715 106M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 26 1592 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 3 180 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:111 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:111 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:161 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:16514 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 2223 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 5900:6923 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 49152:49216 35 13416 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 22435 packets, 3149K bytes) pkts bytes target prot opt in out source destination Nikolai, it starts to get chaotic a bit, can you please clarify the exact versions in play so we rule out a possibility of outdated packages? engine - engine-setup logs (to see certificates enrollment)? selinux status? host - ovirt-vmconsole version? installed from scratch? logs from host deploy? selinux status? client - os version, ssh rpm version? exact command line Hosts and engine being installed from scratch/reprovisioned from Foreman. Engine is actually a VM running from RHEV3. Selinux is enabled on both hosts. I also tried disabling it on one of the hosts and it also not helped. Clients: Clients were several, one my laptop with as described within the bug (RHEL6.6) Linux version 2.6.32-504.30.3.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-9) (GCC) ) #1 SMP Thu Jul 9 15:20:47 EDT 2015. openssh-server-5.3p1-104.el6_6.1.x86_64 libssh2-1.4.2-1.el6_6.1.i686 libssh2-1.4.2-1.el6_6.1.x86_64 openssh-clients-5.3p1-104.el6_6.1.x86_64 fuse-sshfs-2.4-1.el6.x86_64 openssh-5.3p1-104.el6_6.1.x86_64 openssh-askpass-5.3p1-104.el6_6.1.x86_64 $ ps -p $$ PID TTY TIME CMD 24372 pts/13 00:00:00 bash Command line used: "ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect" I aslo tried to establish connections from one of my servers, which is running on top of RHEL7.2 (Linux version 3.10.0-229.15.1.el7_1.1227871.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Aug 27 17:02:53 EDT 2015). openssh-server-6.6.1p1-12.el7_1.x86_64 libssh2-1.4.3-8.el7.x86_64 openssh-clients-6.6.1p1-12.el7_1.x86_64 fence-agents-ilo-ssh-4.0.11-13.el7_1.2.x86_64 openssh-6.6.1p1-12.el7_1.x86_64 ps -p $$ PID TTY TIME CMD 50535 pts/0 00:00:00 bash Command line used: "ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect" Packages actually were provided at the top of this bug, they had changed a bit as I progressed with the findings, so here they are: On hosts: mom-0.5.0-1.el7ev.noarch qemu-kvm-rhev-2.3.0-22.el7.x86_64 libvirt-client-1.2.17-6.el7.x86_64 sanlock-3.2.4-1.el7.x86_64 vdsm-4.17.5-1.el7ev.noarch Red Hat Enterprise Linux Server release 7.2 Beta (Maipo) rubygem-io-console-0.4.2-25.el7_1.x86_64 ovirt-vmconsole-host-1.0.0-0.0.1.master.el7ev.noarch ovirt-vmconsole-1.0.0-0.0.1.master.el7ev.noarch abrt-console-notification-2.1.11-31.el7.x86_64 On engine: ovirt-host-deploy-java-1.4.0-0.0.5.master.el6ev.noarch qemu-guest-agent-0.12.1.2-2.479.el6.x86_64 ovirt-host-deploy-1.4.0-0.0.5.master.el6ev.noarch ovirt-vmconsole-proxy-1.0.0-0.0.1.master.el6ev.noarch rhevm-3.6.0-0.13.master.el6.noarch ovirt-engine-extension-aaa-jdbc-0.0.0-6.el6ev.noarch ovirt-vmconsole-1.0.0-0.0.1.master.el6ev.noarch Red Hat Enterprise Linux Server release 6.7 (Santiago) Selinux currently disabled on engine itself, on hosts it's enabled. # getenforce Permissive This was made because we have it blocking the serial console to get connected to the engine from the customer, once we disabled it, the list of running VMs in engine was reported. Logs attached. Created attachment 1072055 [details]
engine_setup_logs
Created attachment 1072056 [details]
hostdeploy.tar.gz
Created attachment 1072057 [details]
alma02_host_deploy
(In reply to Nikolai Sednev from comment #5) > Disabling Selinux on engine actually helped, as now I could get the VM > running and visible by the engine: > Available Serial Consoles: > 00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a] selinux-related fix tracked in bug 1262003 so the merged patches should be good enough to get it work (selinux, proper hostname reported) Works for me while connecting from Linux version 2.6.32-573.7.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Sep 10 13:42:16 EDT 2015, while using ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole@FQDN_of_my_engine connect I've received the selection of running guest VMs and chosen one of them, the RHEL7.2 vm, then connected to it and after pressing "enter" I've got the login prompt. This issue was found and fixed during development; users should never face it, hace I don't think it deserves mention in documentation. |