Bug 1033706

Summary: Failure to add new host to oVirt 3.3.1 Server (engine) on Fedora 19 with recent yum update
Product: [Retired] oVirt Reporter: Boris Derzhavets <bderzhavets>
Component: ovirt-engine-webadminAssignee: Nobody's working on this, feel free to take it <nobody>
Status: CLOSED DUPLICATE QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: 3.3CC: acathrow, alonbl, bderzhavets, ecohen, iheim, mgoldboi, oourfali, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-27 10:09:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Boris Derzhavets 2013-11-22 16:45:50 UTC 
Description of problem:

Target host is also F19 (the most recent yum update)
Like in case 3.3.0 ran on new host :-
  $ sudo yum localinstall http://ovirt.org/releases/ovirt-release-fedora.noarch.rpm -y
Then set up ovirtmgmt bridge, disabled firewalld and enabled iptables firewall manager. oVirt 3.3.1 engine on server set up for iptables firewall manager.
Attempt to add new host generates :

2013-Nov-22, 20:15 Host ovirt4.localdomain installation failed. Unexpected connection termination.

Version-Release number of selected component (if applicable):

oVirt 3.3.1 on Fedora 19 

How reproducible:

Setup 3.3.1 Ovirt 3.3.1 on F19 per http://community.redhat.com/up-and-running-with-ovirt-3-3/ & http://community.redhat.com/ovirt-3-3-glusterized/

Steps to Reproduce:
1. Select Hosts
2. Select New
3. Submit credentials of new F19 node
4. Install terminates immediately with no diagnostic 

Actual results:

New host rejected.

Expected results:

Setup performs install packages on new host with protocol coming to bottom
bar of the Web Admin console. Runs all required yum installs and initializes
new node in cluster

Additional info:
Comment 1 Boris Derzhavets 2013-11-22 16:59:46 UTC 
Server.log

2013-11-22 20:15:11,454 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-1) Session created...
2013-11-22 20:15:11,464 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-2) Server version string: SSH-2.0-OpenSSH_6.2
2013-11-22 20:15:11,467 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-3) Received SSH_MSG_KEXINIT
2013-11-22 20:15:11,470 INFO  [org.apache.sshd.client.kex.DHG1] (pool-96-thread-3) Send SSH_MSG_KEXDH_INIT
2013-11-22 20:15:11,475 INFO  [org.apache.sshd.client.kex.DHG1] (pool-96-thread-4) Received SSH_MSG_KEXDH_REPLY
2013-11-22 20:15:11,478 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-4) Received SSH_MSG_NEWKEYS
2013-11-22 20:15:11,479 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-4) Send SSH_MSG_SERVICE_REQUEST for ssh-userauth
2013-11-22 20:15:11,561 INFO  [org.apache.sshd.client.auth.UserAuthPublicKey] (pool-6-thread-49) Send SSH_MSG_USERAUTH_REQUEST for publickey
2013-11-22 20:15:11,581 INFO  [org.apache.sshd.client.auth.UserAuthPublicKey] (pool-96-thread-2) Received SSH_MSG_USERAUTH_SUCCESS
2013-11-22 20:15:11,587 INFO  [org.apache.sshd.client.channel.ChannelExec] (pool-6-thread-49) Send SSH_MSG_CHANNEL_OPEN on channel 134
2013-11-22 20:15:11,636 INFO  [org.apache.sshd.client.channel.ChannelExec] (pool-96-thread-4) Send SSH_MSG_CHANNEL_REQUEST exec
2013-11-22 20:15:11,645 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-4) Received SSH_MSG_DISCONNECT (reason=2, msg=Packet corrupt)
2013-11-22 20:15:11,650 INFO  [org.apache.sshd.client.session.ClientSessionImpl] (pool-96-thread-4) Session root@/192.168.1.140:22 closed
2013-11-22 20:16:04,593 INFO  [org.quartz.core.JobRunShell] (DefaultQuartzScheduler_Worker-86) Job DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.gluster_async_task_poll_event#-9223372036854775765 threw a JobExecutionException: : org.quartz.JobExecutionException: failed to execute job
        at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.7.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]

2013-11-22 20:17:04,605 INFO  [org.quartz.core.JobRunShell] (DefaultQuartzScheduler_Worker-99) Job DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.gluster_async_task_poll_event#-9223372036854775765 threw a JobExecutionException: : org.quartz.JobExecutionException: failed to execute job
        at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.7.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
Comment 2 Boris Derzhavets 2013-11-22 17:15:42 UTC 
tcpdump -vvv ( started for attempt to add ovirt4 to ovirt1 ):

    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [S], cksum 0x849c (incorrect -> 0x4a13), seq 4100426124, win 29200, options [mss 1460,sackOK,TS val 18726944 ecr 0,nop,wscale 7], length 0
21:08:43.055009 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    ovirt4.localdomain.ssh > ovirt1.localdomain.51238: Flags [S.], cksum 0x1593 (correct), seq 984864340, ack 4100426125, win 28960, options [mss 1460,sackOK,TS val 4858893 ecr 18726944,nop,wscale 7], length 0
21:08:43.055053 IP (tos 0x0, ttl 64, id 18861, offset 0, flags [DF], proto TCP (6), length 52)
    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [.], cksum 0x8494 (incorrect -> 0xb49a), seq 1, ack 1, win 229, options [nop,nop,TS val 18726944 ecr 4858893], length 0
21:08:43.055626 IP (tos 0x0, ttl 64, id 18862, offset 0, flags [DF], proto TCP (6), length 77)
    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [P.], cksum 0x84ad (incorrect -> 0xb5d8), seq 1:26, ack 1, win 229, options [nop,nop,TS val 18726945 ecr 4858893], length 25
21:08:43.055808 IP (tos 0x0, ttl 64, id 62040, offset 0, flags [DF], proto TCP (6), length 52)
    ovirt4.localdomain.ssh > ovirt1.localdomain.51238: Flags [.], cksum 0xb481 (correct), seq 1, ack 26, win 227, options [nop,nop,TS val 4858894 ecr 18726945], length 0
21:08:43.055830 IP (tos 0x0, ttl 64, id 18863, offset 0, flags [DF], proto TCP (6), length 420)
    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [P.], cksum 0x8604 (incorrect -> 0x6fe8), seq 26:394, ack 1, win 229, options [nop,nop,TS val 18726945 ecr 4858894], length 368
21:08:43.056055 IP (tos 0x0, ttl 64, id 62041, offset 0, flags [DF], proto TCP (6), length 52)
    ovirt4.localdomain.ssh > ovirt1.localdomain.51238: Flags [.], cksum 0xb309 (correct), seq 1, ack 394, win 235, options [nop,nop,TS val 4858894 ecr 18726945], length 0
21:08:43.065901 IP (tos 0x0, ttl 64, id 62042, offset 0, flags [DF], proto TCP (6), length 73)
    ovirt4.localdomain.ssh > ovirt1.localdomain.51238: Flags [P.], cksum 0xf239 (correct), seq 1:22, ack 394, win 235, options [nop,nop,TS val 4858904 ecr 18726945], length 21
21:08:43.065932 IP (tos 0x0, ttl 64, id 18864, offset 0, flags [DF], proto TCP (6), length 52)
    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [.], cksum 0x8494 (incorrect -> 0xb2e6), seq 394, ack 22, win 229, options [nop,nop,TS val 18726955 ecr 4858904], length 0
21:08:43.068671 IP (tos 0x0, ttl 64, id 62043, offset 0, flags [DF], proto TCP (6), length 1596)
    ovirt4.localdomain.ssh > ovirt1.localdomain.51238: Flags [P.], cksum 0x8a9c (incorrect -> 0x38d8), seq 22:1566, ack 394, win 235, options [nop,nop,TS val 4858906 ecr 18726955], length 1544
21:08:43.068682 IP (tos 0x0, ttl 64, id 18865, offset 0, flags [DF], proto TCP (6), length 52)
    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [.], cksum 0x8494 (incorrect -> 0xacc1), seq 394, ack 1566, win 253, options [nop,nop,TS val 18726958 ecr 4858906], length 0
21:08:43.072226 IP (tos 0x0, ttl 64, id 18866, offset 0, flags [DF], proto TCP (6), length 204)
    ovirt1.localdomain.51238 > ovirt4.localdomain.ssh: Flags [P.], cksum 0x852c (incorrect -> 0xb98f), seq 394:546, ack 1566, win 253, options [nop,nop,TS val 18726961 ecr 4858906], length 152
Comment 3 Boris Derzhavets 2013-11-24 15:07:14 UTC 
Workaround :-

# yum downgrade apache-sshd

0.9 -> 0.7 - this one works
Comment 4 Alon Bar-Lev 2013-11-27 10:09:58 UTC 

*** This bug has been marked as a duplicate of bug 1021273 ***