Bug 1329164 - Cannot add Host. Connecting to host via SSH has failed
Summary: Cannot add Host. Connecting to host via SSH has failed
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Host-Deploy
Version: 3.6.5
Hardware: x86_64
OS: Other
unspecified
high vote
Target Milestone: ovirt-4.0.0-beta
: ---
Assignee: Martin Perina
QA Contact: Pavel Stehlik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-21 10:09 UTC by simonp
Modified: 2016-05-06 08:45 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-06 08:45:15 UTC
oVirt Team: Infra
oourfali: ovirt-4.0.0?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
engine.log (487 bytes, text/plain)
2016-04-21 20:15 UTC, simonp
no flags Details
server.log (2.00 KB, text/plain)
2016-04-21 20:16 UTC, simonp
no flags Details

Description simonp 2016-04-21 10:09:05 UTC
Description of problem:

Failing to add host with:
Error while executing action: Cannot add Host.
Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.)
You may refer to the engine.log file for further details.

Version-Release number of selected component (if applicable):
All machine are CentOS Linux release 7.2.1511 (Core)
All package are up to date. (installed from yum)

*sshd_config include:*
KexAlgorithms curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

apache-sshd package is not exists in CentOS system - not applicable.

How reproducible:
Add new host to Cluster.

Additional info:
*engine.log*
2016-04-21 12:18:29,806 ERROR [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-38) [109aa374] Failed to establish session with host 'kvm01': SSH session closed during connection 'root@<host_address>'
2016-04-21 12:18:29,806 WARN  [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-38) [109aa374] CanDoAction of action 'AddVds' failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server <host_address>,VDS_CANNOT_CONNECT_TO_SERVER

Manually ssh from engine to host works without problem.

Package version:
yum list installed | grep ovirt
ebay-cors-filter.noarch                 1.0.1-3.el7              @centos-ovirt36
libtomcrypt.x86_64                      1.17-23.el7              @ovirt-3.6-epel
libtommath.x86_64                       0.42.0-4.el7             @ovirt-3.6-epel
novnc.noarch                            0.5.1-2.el7              @centos-ovirt36
otopi.noarch                            1.4.1-1.el7.centos       @ovirt-3.6
otopi-java.noarch                       1.4.1-1.el7.centos       @ovirt-3.6
ovirt-engine.noarch                     3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-backend.noarch             3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-cli.noarch                 3.6.2.0-1.el7.centos     @ovirt-3.6
ovirt-engine-dbscripts.noarch           3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-extension-aaa-jdbc.noarch  1.0.6-1.el7              @ovirt-3.6
ovirt-engine-extensions-api-impl.noarch 3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-lib.noarch                 3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-restapi.noarch             3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-sdk-python.noarch          3.6.5.0-1.el7.centos     @ovirt-3.6
ovirt-engine-setup.noarch               3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-setup-base.noarch          3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-setup-plugin-ovirt-engine.noarch
                                        3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-setup-plugin-ovirt-engine-common.noarch
                                        3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch
                                        3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-setup-plugin-websocket-proxy.noarch
                                        3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-tools.noarch               3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-tools-backup.noarch        3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-userportal.noarch          3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-vmconsole-proxy-helper.noarch
                                        3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-webadmin-portal.noarch     3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-websocket-proxy.noarch     3.6.5.3-1.el7.centos     @ovirt-3.6
ovirt-engine-wildfly.x86_64             8.2.1-1.el7              @ovirt-3.6
ovirt-engine-wildfly-overlay.noarch     8.0.5-1.el7              @ovirt-3.6
ovirt-host-deploy.noarch                1.4.1-1.el7.centos       @ovirt-3.6
ovirt-host-deploy-java.noarch           1.4.1-1.el7.centos       @ovirt-3.6
ovirt-image-uploader.noarch             3.6.0-1.el7.centos       @ovirt-3.6
ovirt-iso-uploader.noarch               3.6.0-1.el7.centos       @ovirt-3.6
ovirt-release36.noarch                  007-1                    @/ovirt-release36
ovirt-setup-lib.noarch                  1.0.1-1.el7.centos       @ovirt-3.6
ovirt-vmconsole.noarch                  1.0.0-1.el7.centos       @ovirt-3.6
ovirt-vmconsole-proxy.noarch            1.0.0-1.el7.centos       @ovirt-3.6
patternfly1.noarch                      1.3.0-1.el7.centos       @ovirt-3.6-patternfly1-noarch-epel
python-daemon.noarch                    1.6-4.el7                @centos-ovirt36
python-paramiko.noarch                  1.15.1-1.el7             @centos-ovirt36
python-websockify.noarch                0.6.0-2.el7              @centos-ovirt36
python2-crypto.x86_64                   2.6.1-9.el7              @ovirt-3.6-epel
spice-html5.noarch                      0.1.6-1.el7              @centos-ovirt36
vdsm-jsonrpc-java.noarch                1.1.9-1.el7.centos       @ovirt-3.6

Comment 1 Oved Ourfali 2016-04-21 12:02:13 UTC
Please attach the logs.

Comment 2 simonp 2016-04-21 20:15:49 UTC
Created attachment 1149565 [details]
engine.log

/var/log/ovirt-engine/engine.log - messages after try add the host.

Comment 3 simonp 2016-04-21 20:16:23 UTC
Created attachment 1149566 [details]
server.log

Comment 4 Martin Perina 2016-04-22 11:36:49 UTC
> Description of problem:
> 
> Failing to add host with:
> Error while executing action: Cannot add Host.
> Connecting to host via SSH has failed, verify that the host is reachable (IP
> address, routable address etc.)
> You may refer to the engine.log file for further details.
> 
> Version-Release number of selected component (if applicable):
> All machine are CentOS Linux release 7.2.1511 (Core)
> All package are up to date. (installed from yum)
> 
> *sshd_config include:*
> KexAlgorithms
> curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-
> nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,
> diffie-hellman-group1-sha1

So, you have changed sshd configuration on the host and restricting algorithms only to the above? If so, could you please try to add the host to the engine with default algorithms?

Comment 5 simonp 2016-04-24 15:41:29 UTC
(In reply to Martin Perina from comment #4)
> > Description of problem:
> > 
> > Failing to add host with:
> > Error while executing action: Cannot add Host.
> > Connecting to host via SSH has failed, verify that the host is reachable (IP
> > address, routable address etc.)
> > You may refer to the engine.log file for further details.
> > 
> > Version-Release number of selected component (if applicable):
> > All machine are CentOS Linux release 7.2.1511 (Core)
> > All package are up to date. (installed from yum)
> > 
> > *sshd_config include:*
> > KexAlgorithms
> > curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-
> > nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,
> > diffie-hellman-group1-sha1
> 
> So, you have changed sshd configuration on the host and restricting
> algorithms only to the above? If so, could you please try to add the host to
> the engine with default algorithms?

Hi,

Yes, I restrict algorithms to this list but when I remove nothing change. 
I also tried to set this list to engine host (ssh_config) without success.

Regards,
Simon

Comment 6 simonp 2016-04-24 15:59:33 UTC
(In reply to Martin Perina from comment #4)
> > Description of problem:
> > 
> > Failing to add host with:
> > Error while executing action: Cannot add Host.
> > Connecting to host via SSH has failed, verify that the host is reachable (IP
> > address, routable address etc.)
> > You may refer to the engine.log file for further details.
> > 
> > Version-Release number of selected component (if applicable):
> > All machine are CentOS Linux release 7.2.1511 (Core)
> > All package are up to date. (installed from yum)
> > 
> > *sshd_config include:*
> > KexAlgorithms
> > curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-
> > nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,
> > diffie-hellman-group1-sha1
> 
> So, you have changed sshd configuration on the host and restricting
> algorithms only to the above? If so, could you please try to add the host to
> the engine with default algorithms?

And another information about java:
java-1.7.0-openjdk.x86_64            1:1.7.0.101-2.6.6.1.el7_2   @updates
java-1.7.0-openjdk-devel.x86_64      1:1.7.0.101-2.6.6.1.el7_2   @updates
java-1.7.0-openjdk-headless.x86_64   1:1.7.0.101-2.6.6.1.el7_2   @updates
java-1.8.0-openjdk.x86_64            1:1.8.0.91-0.b14.el7_2      @updates
java-1.8.0-openjdk-headless.x86_64   1:1.8.0.91-0.b14.el7_2      @updates
javamail.noarch                      1.4.6-8.el7                 @base
javapackages-tools.noarch            3.4.1-11.el7                @base
javassist.noarch                     3.16.1-10.el7               @base
otopi-java.noarch                    1.4.1-1.el7.centos          @ovirt-3.6
ovirt-host-deploy-java.noarch        1.4.1-1.el7.centos          @ovirt-3.6
python-javapackages.noarch           3.4.1-11.el7                @base
tzdata-java.noarch                   2016d-1.el7                 @updates
vdsm-jsonrpc-java.noarch             1.1.9-1.el7.centos          @ovirt-3.6

Comment 7 Oved Ourfali 2016-04-25 18:17:31 UTC
Seems like an openjdk bug. See dependency. 
Workaround can be to downgrade to previous version.

Comment 9 Severin Gehwolf 2016-04-25 20:12:28 UTC
(In reply to simonp from comment #6)
> (In reply to Martin Perina from comment #4)
> > > Description of problem:
> > > 
> > > Failing to add host with:
> > > Error while executing action: Cannot add Host.
> > > Connecting to host via SSH has failed, verify that the host is reachable (IP
> > > address, routable address etc.)
> > > You may refer to the engine.log file for further details.
> > > 
> > > Version-Release number of selected component (if applicable):
> > > All machine are CentOS Linux release 7.2.1511 (Core)
> > > All package are up to date. (installed from yum)
> > > 
> > > *sshd_config include:*
> > > KexAlgorithms
> > > curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-
> > > nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,
> > > diffie-hellman-group1-sha1
> > 
> > So, you have changed sshd configuration on the host and restricting
> > algorithms only to the above? If so, could you please try to add the host to
> > the engine with default algorithms?
> 
> And another information about java:
> java-1.7.0-openjdk.x86_64            1:1.7.0.101-2.6.6.1.el7_2   @updates
> java-1.7.0-openjdk-devel.x86_64      1:1.7.0.101-2.6.6.1.el7_2   @updates
> java-1.7.0-openjdk-headless.x86_64   1:1.7.0.101-2.6.6.1.el7_2   @updates
> java-1.8.0-openjdk.x86_64            1:1.8.0.91-0.b14.el7_2      @updates
> java-1.8.0-openjdk-headless.x86_64   1:1.8.0.91-0.b14.el7_2      @updates
> javamail.noarch                      1.4.6-8.el7                 @base
> javapackages-tools.noarch            3.4.1-11.el7                @base
> javassist.noarch                     3.16.1-10.el7               @base
> otopi-java.noarch                    1.4.1-1.el7.centos          @ovirt-3.6
> ovirt-host-deploy-java.noarch        1.4.1-1.el7.centos          @ovirt-3.6
> python-javapackages.noarch           3.4.1-11.el7                @base
> tzdata-java.noarch                   2016d-1.el7                 @updates
> vdsm-jsonrpc-java.noarch             1.1.9-1.el7.centos          @ovirt-3.6

Which java version is ovirt using? java-1.7.0-openjdk or java-1.8.0-openjdk? Before we mark this bug dependent on bug 1329342 it would be good if simonp could verify that the reproducer in that bug fails for him.

Note that bug 1329342 is a Fedora bug.

Comment 10 Martin Perina 2016-04-26 12:17:02 UTC
Unfortunately I'm not able to reproduce the error even when I set KexAlgorithms on host as you suggested. Are you sure you posted the correct value for KexAlgorithms? Because the one you posted is the default.

Anyway if you want to be sure if the issue is in KexAlgorithms, please turn debug logging for apache-sshd library which is used by engine:

1. Please edit /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in
2. Locate the lines (lines 207-209)

     <logger category="org.springframework.ldap">
       <level name="ERROR"/>
     </logger>

3. Add following lines to enable debug log for apache-sshd

     <logger category="org.apache.sshd">
       <level name="DEBUG"/>
     </logger>

4. Restart ovirt-engine service, try to install the host again and post server.log

Comment 11 Severin Gehwolf 2016-04-26 13:07:23 UTC
Removing dependency bug as it's not clear (nor has it been confirmed) it's actually that bug.

Comment 12 Oved Ourfali 2016-04-27 06:29:07 UTC
Simon - Can you also try to downgrade openjdk 1.8 and see if it works?

Comment 13 simonp 2016-04-28 18:55:32 UTC
(In reply to Oved Ourfali from comment #12)
> Simon - Can you also try to downgrade openjdk 1.8 and see if it works?

Hi,

Sorry for delay.

It seems that add this line to /etc/ssh/sshd_config solve the problem:

KexAlgorithms curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305,aes256-gcm,aes128-gcm,aes256-ctr,aes192-ctr,aes128-ctr

Regards,
Simon

Comment 14 Sandro Bonazzola 2016-05-02 10:08:05 UTC
Moving from 4.0 alpha to 4.0 beta since 4.0 alpha has been already released and bug is not ON_QA.

Comment 15 Martin Perina 2016-05-06 08:45:15 UTC
Unfortunately I haven't been able to reproduce this one, I was able successfully install the host with both default sshd parameters and custom one specified in this bug. I'm going to close this one WORKSFORME, feel free to reopen and share new logs with sshd config from the host if this issue reappears.


Note You need to log in before you can comment on or make changes to this bug.