Bug 1261519 - Can't connect to ovirt-vmconsole-proxy over ssh from my RHEL6.6 client node
Summary: Can't connect to ovirt-vmconsole-proxy over ssh from my RHEL6.6 client node
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.0
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ovirt-3.6.0-rc3
: 3.6.0
Assignee: Francesco Romani
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On: 1262003
Blocks: 1223671 1255222
TreeView+ depends on / blocked
 
Reported: 2015-09-09 14:04 UTC by Nikolai Sednev
Modified: 2016-04-20 01:34 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-04-20 01:34:02 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
engine.log (268.88 KB, application/x-gzip)
2015-09-09 14:08 UTC, Nikolai Sednev
no flags Details
engine_setup_logs (502.64 KB, application/x-gzip)
2015-09-10 08:14 UTC, Nikolai Sednev
no flags Details
hostdeploy.tar.gz (351.77 KB, application/x-gzip)
2015-09-10 08:14 UTC, Nikolai Sednev
no flags Details
alma02_host_deploy (27.32 KB, application/x-gzip)
2015-09-10 08:15 UTC, Nikolai Sednev
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 46052 0 'None' MERGED sercon: servlet: report correct host name 2020-10-27 11:31:48 UTC
oVirt gerrit 46215 0 'None' MERGED sercon: servlet: report correct host name 2020-10-27 11:31:47 UTC

Description Nikolai Sednev 2015-09-09 14:04:30 UTC
Description of problem:
I'm trying to connect to ovirt-vmconsole-proxy from my laptop running on RHEL6.6.
Connection fails with this error:
$ ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect                        
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013                                                                           
debug1: Reading configuration data /etc/ssh/ssh_config                                                                   
debug1: Applying options for *                                                                                           
debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222.                                                           
debug1: Connection established.                                                                                          
debug1: identity file /home/nsednev/.ssh/id_rsa type 1                                                                   
debug1: identity file /home/nsednev/.ssh/id_rsa-cert type -1                                                             
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3                                                 
debug1: match: OpenSSH_5.3 pat OpenSSH*                                                                                  
debug1: Enabling compatibility mode for protocol 2.0                                                                     
debug1: Local version string SSH-2.0-OpenSSH_5.3                                                                         
debug1: SSH2_MSG_KEXINIT sent                                                                                            
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: ssh_rsa_verify: signature correct
debug1: checking without port identifier
debug1: No matching CA found. Retry with plain key
debug1: No matching CA found. Retry with plain key
debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key.
debug1: Found key in /home/nsednev/.ssh/known_hosts:20
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/nsednev/.ssh/id_rsa
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 277
Agent admitted failure to sign using the key.
debug1: No more authentication methods to try.
Permission denied (publickey).


On my laptop I also see this:
$ ps axf | grep gnome-keyring
 3400 ?        Sl     0:03 /usr/bin/gnome-keyring-daemon --daemonize --login
32073 pts/6    S+     0:00  |   \_ grep gnome-keyring

It looks like pretty common to https://bugzilla.redhat.com/show_bug.cgi?id=921513

Version-Release number of selected component (if applicable):
On 3.6engine:
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.0-0.13.master.el6.noarch
ovirt-vmconsole-proxy-1.0.0-0.0.1.master.el6ev.noarch
rhevm-vmconsole-proxy-helper-3.6.0-0.13.master.el6.noarch
jboss-as-console-2.5.6-2.Final_redhat_2.1.ep6.el6.noarch
ovirt-vmconsole-1.0.0-0.0.1.master.el6ev.noarch
rhevm-3.6.0-0.13.master.el6.noarch
Linux version 2.6.32-573.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015


How reproducible:
100%

Steps to Reproduce:
1.Install RHEL6.7 latest on VM.
2.Install ovirt-vmconsole-proxy on VM.
3.Install 3.6 ovirt-engine on VM.
4.engine-setup on VM.
5.Add one clean RHEL7.2 host to the engine, so it could be managed by the engine.
6.Create on your client pc/laptop on RHEL6.6 public and secret keys using "ssh-keygen", you should get created the $HOME/.ssh/id_rsa and $HOME/.ssh/id_rsa.pub
7.Copy the key from id_rsa.pub, e.g. "cat $HOME/.ssh/id_rsa.pub"
8.Log in to the WEBUI of the engine and then go to Virtual machines tab->on the top just righter than "Export" you will find symbol of "Play"; click on it and select "Set Serial Console Key", then paste the public key there.
9.Try to connect to serial console from you PC using " ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole@<IP_OF_YOUR_ENGINE> connect".


Actual results:
Permission denied (publickey).

Expected results:
SSH session should be opened to the engine successfully.

Additional info:

Comment 1 Nikolai Sednev 2015-09-09 14:08:29 UTC
Created attachment 1071776 [details]
engine.log

Comment 2 Francesco Romani 2015-09-09 14:15:23 UTC
This is ssh-agent issue.
Maybe it's just me, but I don't see how can this can be RHEV bug.

You should be able to workaround it easily using a different ssh agent, for example the default one which comes with ssh itself:

man ssh-agent

See these instructions for a workaround. This is for github, but it is clearly explained, and the tools are the same.

https://help.github.com/articles/error-agent-admitted-failure-to-sign/

Comment 3 Nikolai Sednev 2015-09-09 14:18:08 UTC
I've tried also the WA provided here https://help.github.com/articles/error-agent-admitted-failure-to-sign/ and it didn't worked for me:
ssh-add
Identity added: /home/nsednev/.ssh/id_rsa (/home/nsednev/.ssh/id_rsa)
$ ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013                                                   
debug1: Reading configuration data /etc/ssh/ssh_config                                           
debug1: Applying options for *                                                                   
debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222.                                   
debug1: Connection established.                                                                  
debug1: identity file /home/nsednev/.ssh/id_rsa type 1                                           
debug1: identity file /home/nsednev/.ssh/id_rsa-cert type -1                                     
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3                         
debug1: match: OpenSSH_5.3 pat OpenSSH*                                                          
debug1: Enabling compatibility mode for protocol 2.0                                             
debug1: Local version string SSH-2.0-OpenSSH_5.3                                                 
debug1: SSH2_MSG_KEXINIT sent                                                                    
debug1: SSH2_MSG_KEXINIT received                                                                
debug1: kex: server->client aes128-ctr hmac-md5 none                                             
debug1: kex: client->server aes128-ctr hmac-md5 none                                             
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent                                         
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP                                                      
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent                                                            
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY                                                      
debug1: ssh_rsa_verify: signature correct                                                        
debug1: checking without port identifier                                                         
debug1: No matching CA found. Retry with plain key                                               
debug1: No matching CA found. Retry with plain key                                               
debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key.                       
debug1: Found key in /home/nsednev/.ssh/known_hosts:20                                           
debug1: ssh_rsa_verify: signature correct                                                        
debug1: SSH2_MSG_NEWKEYS sent                                                                    
debug1: expecting SSH2_MSG_NEWKEYS                                                               
debug1: SSH2_MSG_NEWKEYS received                                                                
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/nsednev/.ssh/id_rsa
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env XMODIFIERS = @im=none
debug1: Sending env LANG = en_US.utf8
debug1: Sending env LANGUAGE =
debug1: Sending command: connect
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
/bin/sh: Permission denied
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.35.160.204 closed.
Transferred: sent 3048, received 4112 bytes, in 0.3 seconds
Bytes per second: sent 9585.7, received 12931.8
debug1: Exit status 1

Comment 4 Nikolai Sednev 2015-09-09 14:41:07 UTC
Disabling the Selinux on engine not helped as well as also restarting the engine.

I also tried to connect to ssh serial console from one of my pure cli RHEL7.1 servers and also failed:

]# ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 list
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013                                            
debug1: Reading configuration data /etc/ssh/ssh_config                                    
debug1: /etc/ssh/ssh_config line 56: Applying options for *                               
debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222.                            
debug1: Connection established.                                                           
debug1: permanently_set_uid: 0/0                                                          
debug1: identity file /root/.ssh/id_rsa type 1                                            
debug1: identity file /root/.ssh/id_rsa-cert type -1                                      
debug1: Enabling compatibility mode for protocol 2.0                                      
debug1: Local version string SSH-2.0-OpenSSH_6.6.1                                        
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3                  
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000                               
debug1: SSH2_MSG_KEXINIT sent                                                             
debug1: SSH2_MSG_KEXINIT received                                                         
debug1: kex: server->client aes128-ctr hmac-md5 none                                      
debug1: kex: client->server aes128-ctr hmac-md5 none                                      
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                      
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                      
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent                                  
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP                                               
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent                                                     
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY                                               
debug1: ssh_rsa_verify: signature correct                                                 
debug1: Server host key: RSA-CERT 04:52:9f:4b:cf:11:67:8c:8d:ee:1b:48:84:2f:dc:b7         
debug1: checking without port identifier                                                  
debug1: No matching CA found. Retry with plain key                                        
debug1: No matching CA found. Retry with plain key                                        
debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key.                
debug1: Found key in /root/.ssh/known_hosts:2                                             
debug1: ssh_rsa_verify: signature correct                                                 
debug1: SSH2_MSG_NEWKEYS sent                                                             
debug1: expecting SSH2_MSG_NEWKEYS                                                        
debug1: SSH2_MSG_NEWKEYS received                                                         
debug1: Roaming not allowed by server                                                     
debug1: SSH2_MSG_SERVICE_REQUEST sent                                                     
debug1: SSH2_MSG_SERVICE_ACCEPT received                                                  
debug1: Authentications that can continue: publickey                                      
debug1: Next authentication method: publickey                                             
debug1: Offering RSA public key: /root/.ssh/id_rsa                                        
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"                                                                                                      
debug1: Remote: Agent forwarding disabled.                                                                                       
debug1: Remote: Port forwarding disabled.                                                                                        
debug1: Remote: User rc file execution disabled.                                                                                 
debug1: Remote: X11 forwarding disabled.                                                                                         
debug1: Server accepts key: pkalg ssh-rsa blen 279                                                                               
debug1: key_parse_private2: missing begin marker                                                                                 
debug1: read PEM private key done: type RSA                                                                                      
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"                                                                                                      
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
Authenticated to 10.35.160.204 ([10.35.160.204]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env XMODIFIERS = @im=none
debug1: Sending env LANG = en_US.utf8
debug1: Sending env LANGUAGE =
debug1: Sending command: list
8ab54a0a-7eb1-4112-9456-0499e654ed9a    test
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.35.160.204 closed.
Transferred: sent 4040, received 4128 bytes, in 0.6 seconds
Bytes per second: sent 6313.9, received 6451.5
debug1: Exit status 0

Comment 5 Nikolai Sednev 2015-09-09 15:05:23 UTC
Disabling Selinux on engine actually helped, as now I could get the VM running and visible by the engine:
Available Serial Consoles:
00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a]

But connecting to it still failed:
Available Serial Consoles:
00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a]
SELECT> 00
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
ssh: connect to host 10.35.117.22 port 2223: Connection refused
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.35.160.204 closed.
Transferred: sent 4728, received 5184 bytes, in 85.0 seconds
Bytes per second: sent 55.7, received 61.0
debug1: Exit status 255

There was found that service ovirt-vmconsole-host-sshd wasn't running on host, so it was restarted and the I've tried again to connect to the VM, failed because of :
Certificate invalid: name is not a listed principal
<fromani_> Host key verification failed. -> need to reinstall the host from engine webadmin
<fromani_> so that certificate is enrolled again

I reinstalled both hosts and retried, the result was also failure:
# ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect                                    
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013                                                                                   
debug1: Reading configuration data /etc/ssh/ssh_config                                                                           
debug1: /etc/ssh/ssh_config line 56: Applying options for *                                                                      
debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222.                                                                   
debug1: Connection established.                                                                                                  
debug1: permanently_set_uid: 0/0                                                                                                 
debug1: identity file /root/.ssh/id_rsa type 1                                                                                   
debug1: identity file /root/.ssh/id_rsa-cert type -1                                                                             
debug1: Enabling compatibility mode for protocol 2.0                                                                             
debug1: Local version string SSH-2.0-OpenSSH_6.6.1                                                                               
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3                                                         
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000                                                                      
debug1: SSH2_MSG_KEXINIT sent                                                                                                    
debug1: SSH2_MSG_KEXINIT received                                                                                                
debug1: kex: server->client aes128-ctr hmac-md5 none                                                                             
debug1: kex: client->server aes128-ctr hmac-md5 none                                                                             
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                                                             
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                                                             
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent                                                                         
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP                                                                                      
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent                                                                                            
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY                                                                                      
debug1: ssh_rsa_verify: signature correct                                                                                        
debug1: Server host key: RSA-CERT 04:52:9f:4b:cf:11:67:8c:8d:ee:1b:48:84:2f:dc:b7                                                
debug1: checking without port identifier                                                                                         
debug1: No matching CA found. Retry with plain key                                                                               
debug1: No matching CA found. Retry with plain key                                                                               
debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key.                                                       
debug1: Found key in /root/.ssh/known_hosts:2                                                                                    
debug1: ssh_rsa_verify: signature correct                                                                                        
debug1: SSH2_MSG_NEWKEYS sent                                                                                                    
debug1: expecting SSH2_MSG_NEWKEYS                                                                                               
debug1: SSH2_MSG_NEWKEYS received                                                                                                
debug1: Roaming not allowed by server                                                                                            
debug1: SSH2_MSG_SERVICE_REQUEST sent                                                                                            
debug1: SSH2_MSG_SERVICE_ACCEPT received                                                                                         
debug1: Authentications that can continue: publickey                                                                             
debug1: Next authentication method: publickey                                                                                    
debug1: Offering RSA public key: /root/.ssh/id_rsa                                                                               
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"                                                                                                      
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
Authenticated to 10.35.160.204 ([10.35.160.204]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env XMODIFIERS = @im=none
debug1: Sending env LANG = en_US.utf8
debug1: Sending env LANGUAGE =
debug1: Sending command: connect
Available Serial Consoles:
00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a]
SELECT> 00
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
Certificate invalid: name is not a listed principal
Host key verification failed.
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.35.160.204 closed.
Transferred: sent 4152, received 4384 bytes, in 3.3 seconds
Bytes per second: sent 1240.2, received 1309.5
debug1: Exit status 255

iptables on host are fine:
# iptables -n -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
28715  106M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
   26  1592 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    3   180 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:54321
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:111
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:111
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:161
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:16514
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 2223
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 5900:6923
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 49152:49216
   35 13416 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 22435 packets, 3149K bytes)
 pkts bytes target     prot opt in     out     source               destination

Comment 6 Michal Skrivanek 2015-09-10 07:25:26 UTC
Nikolai, it starts to get chaotic a bit, can you please clarify the exact versions in play so we rule out a possibility of outdated packages?

engine -
engine-setup logs (to see certificates enrollment)? 
selinux status?

host - 
ovirt-vmconsole version?
installed from scratch?
logs from host deploy?
selinux status?

client -
os version, ssh rpm version?
exact command line

Comment 7 Nikolai Sednev 2015-09-10 08:13:46 UTC
Hosts and engine being installed from scratch/reprovisioned from Foreman.
Engine is actually a VM running from RHEV3.
Selinux is enabled on both hosts. I also tried disabling it on one of the hosts and it also not helped.

Clients:
Clients were several, one my laptop with as described within the bug (RHEL6.6)
Linux version 2.6.32-504.30.3.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-9) (GCC) ) #1 SMP Thu Jul 9 15:20:47 EDT 2015.
openssh-server-5.3p1-104.el6_6.1.x86_64
libssh2-1.4.2-1.el6_6.1.i686
libssh2-1.4.2-1.el6_6.1.x86_64
openssh-clients-5.3p1-104.el6_6.1.x86_64
fuse-sshfs-2.4-1.el6.x86_64
openssh-5.3p1-104.el6_6.1.x86_64
openssh-askpass-5.3p1-104.el6_6.1.x86_64
$ ps -p $$
  PID TTY          TIME CMD
24372 pts/13   00:00:00 bash
Command line used: "ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect"


I aslo tried to establish connections from one of my servers, which is running on top of RHEL7.2 (Linux version 3.10.0-229.15.1.el7_1.1227871.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Aug 27 17:02:53 EDT 2015).
openssh-server-6.6.1p1-12.el7_1.x86_64
libssh2-1.4.3-8.el7.x86_64
openssh-clients-6.6.1p1-12.el7_1.x86_64
fence-agents-ilo-ssh-4.0.11-13.el7_1.2.x86_64
openssh-6.6.1p1-12.el7_1.x86_64
ps -p $$
   PID TTY          TIME CMD
 50535 pts/0    00:00:00 bash


Command line used: "ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect"

Packages actually were provided at the top of this bug, they had changed a bit as I progressed with the findings, so here they are:
On hosts:
mom-0.5.0-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-22.el7.x86_64
libvirt-client-1.2.17-6.el7.x86_64
sanlock-3.2.4-1.el7.x86_64
vdsm-4.17.5-1.el7ev.noarch
Red Hat Enterprise Linux Server release 7.2 Beta (Maipo)
rubygem-io-console-0.4.2-25.el7_1.x86_64
ovirt-vmconsole-host-1.0.0-0.0.1.master.el7ev.noarch
ovirt-vmconsole-1.0.0-0.0.1.master.el7ev.noarch
abrt-console-notification-2.1.11-31.el7.x86_64

On engine:
ovirt-host-deploy-java-1.4.0-0.0.5.master.el6ev.noarch
qemu-guest-agent-0.12.1.2-2.479.el6.x86_64
ovirt-host-deploy-1.4.0-0.0.5.master.el6ev.noarch
ovirt-vmconsole-proxy-1.0.0-0.0.1.master.el6ev.noarch
rhevm-3.6.0-0.13.master.el6.noarch
ovirt-engine-extension-aaa-jdbc-0.0.0-6.el6ev.noarch
ovirt-vmconsole-1.0.0-0.0.1.master.el6ev.noarch
Red Hat Enterprise Linux Server release 6.7 (Santiago)

Selinux currently disabled on engine itself, on hosts it's enabled.
# getenforce
Permissive
This was made because we have it blocking the serial console to get connected to the engine from the customer, once we disabled it, the list of running VMs in engine was reported.



Logs attached.

Comment 8 Nikolai Sednev 2015-09-10 08:14:26 UTC
Created attachment 1072055 [details]
engine_setup_logs

Comment 9 Nikolai Sednev 2015-09-10 08:14:52 UTC
Created attachment 1072056 [details]
hostdeploy.tar.gz

Comment 10 Nikolai Sednev 2015-09-10 08:15:33 UTC
Created attachment 1072057 [details]
alma02_host_deploy

Comment 11 Michal Skrivanek 2015-09-14 14:48:30 UTC
(In reply to Nikolai Sednev from comment #5)
> Disabling Selinux on engine actually helped, as now I could get the VM
> running and visible by the engine:
> Available Serial Consoles:
> 00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a]

selinux-related fix tracked in bug 1262003

Comment 12 Michal Skrivanek 2015-09-17 08:53:42 UTC
so the merged patches should be good enough to get it work (selinux, proper hostname reported)

Comment 13 Nikolai Sednev 2015-11-05 16:18:35 UTC
Works for me while connecting from Linux version 2.6.32-573.7.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Sep 10 13:42:16 EDT 2015, while using ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole@FQDN_of_my_engine connect

I've received the selection of running guest VMs and chosen one of them, the RHEL7.2 vm, then connected to it and after pressing "enter" I've got the login prompt.

Comment 14 Francesco Romani 2016-01-19 15:28:12 UTC
This issue was found and fixed during development; users should never face it, hace I don't think it deserves mention in documentation.


Note You need to log in before you can comment on or make changes to this bug.