Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1154862

Summary: RHEV-H upgrade fail with ssh timeout error when trying to copy the ISO to the host
Product: Red Hat Enterprise Virtualization Manager Reporter: Marina Kalinin <mkalinin>
Component: ovirt-engineAssignee: Nobody <nobody>
Status: CLOSED DUPLICATE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.4.1-1CC: ecohen, fdeutsch, gklein, iheim, lpeer, lsurette, oourfali, rbalakri, Rhev-m-bugs, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-21 08:44:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marina Kalinin 2014-10-20 21:06:35 UTC
Description of problem:
RHEV-H upgrade fail with ssh timeout error when trying to copy the ISO to the host.

Version-Release number of selected component (if applicable):
rhevh-6.5-20140930.1.el6ev.iso
rhevm-3.4.2-0.2.el6ev.noarch

How reproducible:
Always reproducible on my environment. 
But I didn't hear about other cases.

Steps to Reproduce:
1. Put host in maintenance.
2. Click "Upgrade" link, use the default option to use 'SSH publick key' option -> Ok.
3. Installation would fail with time out error: 
Host rhevh4.example.com installation failed. SSH session timeout host 'root.10.11'.

Additional info:
From engine.log:
~~~
2014-10-20 16:22:40,079 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-4-thread-10) [5591cfac] Installation 10.10.10.11: Connected to host 10.10.10.11 with SSH key fingerprint: 18:ab:2c:a5:ea:75:d7:0b:80:3f:df:f2:9f:db:00:0b
2014-10-20 16:22:40,086 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-10) [5591cfac] Correlation ID: 5591cfac, Call Stack: null, Custom Event ID: -1, Message: Installing Host rhevh4.example.com. Connected to host 10.10.10.11 with SSH key fingerprint: 18:ab:2c:a5:ea:75:d7:0b:80:3f:df:f2:9f:db:00:0b.
2014-10-20 16:22:40,119 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-4-thread-10) [5591cfac] Installation 10.10.10.11: Sending file /usr/share/rhev-hypervisor/rhevh-6.5-20140930.1.el6ev.iso to /data/updates/ovirt-node-image.iso
2014-10-20 16:22:40,127 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-10) [5591cfac] Correlation ID: 5591cfac, Call Stack: null, Custom Event ID: -1, Message: Installing Host rhevh4.example.com. Sending file /usr/share/rhev-hypervisor/rhevh-6.5-20140930.1.el6ev.iso to /data/updates/ovirt-node-image.iso.
2014-10-20 16:22:40,127 INFO  [org.ovirt.engine.core.utils.ssh.SSHDialog] (org.ovirt.thread.pool-4-thread-10) SSH execute root.10.11 'mkdir -p '/data/updates'
2014-10-20 16:27:40,218 ERROR [org.ovirt.engine.core.bll.OVirtNodeUpgrade] (org.ovirt.thread.pool-4-thread-10) [5591cfac] Timeout during node 10.10.10.11 upgrade: javax.naming.TimeLimitExceede
dException: SSH session timeout host 'root.10.11'
        at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:499) [utils.jar:]
        at org.ovirt.engine.core.utils.ssh.SSHClient.sendFile(SSHClient.java:633) [utils.jar:]
        at org.ovirt.engine.core.utils.ssh.SSHDialog.sendFile(SSHDialog.java:374) [utils.jar:]
        at org.ovirt.engine.core.bll.OVirtNodeUpgrade.execute(OVirtNodeUpgrade.java:200) [bll.jar:]
        at org.ovirt.engine.core.bll.InstallVdsCommand.upgradeNode(InstallVdsCommand.java:287) [bll.jar:]
        at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:154) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1133) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1218) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1894) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1238) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:189) [bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:156) [bll.jar:]
        at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:165) [bll.jar:]
        at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:97) [utils.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]
~~~

- I do not see any ssh connection from the manager opened to the host:
# netstat -plant |grep sshd | grep <rhevm-fqdn>
- The connection is good, the copy is very fast when connected.
- I can ssh to the host from the manager.
- When running upgrade, on the host I can see:
# grep 'rhevm-ip' /var/log/secure
Oct 20 21:00:14 rhevh-4 sshd[32340]: Accepted publickey for root from <rhevm-ip> port 41707 ssh2
- On the manager, sslv3 is disabled on httpd (to prevent poodle attack).

Also, sometimes, after I ssh from the manager to the host manually, the upgrade succeeds.
Maybe it is something wrong with my specific environment, I do not know.
I would appreciate some hints understanding this problem.

Comment 1 Oved Ourfali 2014-10-21 06:19:32 UTC
Fabian - this sounds familiar to me. Do you happen to remember this issue?

Comment 2 Fabian Deutsch 2014-10-21 08:44:23 UTC
oved, yes, I believe this is a dupe of bug 1093882

*** This bug has been marked as a duplicate of bug 1093882 ***