Bug 1288206

Summary: rhel-osp-director: 7.2 - Cannot ssh into the launched instance, despite being able to reach port 22.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: hbrock, jslagle, mburns, rhel-osp-director-maint, sasha, srevivo
Target Milestone: y3   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-03 21:45:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
neutron conf and logs from one controller none

Description Alexander Chuzhoy 2015-12-03 20:09:29 UTC 
rhel-osp-director: 7.2 - Cannot ssh into the launched instance, despite being able to reach port 22.

Environment:
openstack-neutron-bigswitch-lldp-2015.1.38-1.el7ost.noarch
openstack-neutron-lbaas-2015.1.2-1.el7ost.noarch
python-neutronclient-2.4.0-2.el7ost.noarch
python-neutron-2015.1.2-2.el7ost.noarch
openstack-neutron-2015.1.2-2.el7ost.noarch
openstack-neutron-ml2-2015.1.2-2.el7ost.noarch
openstack-neutron-common-2015.1.2-2.el7ost.noarch
python-neutron-lbaas-2015.1.2-1.el7ost.noarch
openstack-neutron-openvswitch-2015.1.2-2.el7ost.noarch
openstack-neutron-metering-agent-2015.1.2-2.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-85.el7ost.noarch
instack-undercloud-2.1.2-34.el7ost.noarch

Steps to reproduce:

1. Deploy HA overcloud with network isolation.
2. Allow ICMP,SSH in the default security group.
3. Launch an instance and verify its reachable via ping.
4. Attempt to ssh into the instance.


Result:
Gets stuck:

ssh  192.168.200.101 -l cirros -vvv
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013        
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0                            
debug1: Connecting to 192.168.200.101 [192.168.200.101] port 22.
debug1: Connection established.                                 
debug3: Incorrect RSA1 identifier                               
debug3: Could not load "/home/stack/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/stack/.ssh/id_rsa type 1                 
debug1: identity file /home/stack/.ssh/id_rsa-cert type -1           
debug1: identity file /home/stack/.ssh/id_dsa type -1                
debug1: identity file /home/stack/.ssh/id_dsa-cert type -1           
debug1: identity file /home/stack/.ssh/id_ecdsa type -1              
debug1: identity file /home/stack/.ssh/id_ecdsa-cert type -1         
debug1: identity file /home/stack/.ssh/id_ed25519 type -1            
debug1: identity file /home/stack/.ssh/id_ed25519-cert type -1       
debug1: Enabling compatibility mode for protocol 2.0                 
debug1: Local version string SSH-2.0-OpenSSH_6.6.1                   
debug1: Remote protocol version 2.0, remote software version dropbear_0.53.1
debug1: no match: dropbear_0.53.1                                           
debug2: fd 3 setting O_NONBLOCK                                             
debug3: load_hostkeys: loading entries for host "192.168.200.101" from file "/home/stack/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys                                                                      
debug1: SSH2_MSG_KEXINIT sent                                                                             
debug1: SSH2_MSG_KEXINIT received                                                                         
debug2: kex_parse_kexinit: curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1                                                                                                                                                                                           
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,ssh-ed25519-cert-v01,ssh-rsa-cert-v01,ssh-dss-cert-v01,ssh-rsa-cert-v00,ssh-dss-cert-v00,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss                                        
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm,aes256-gcm,chacha20-poly1305,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se                                                                                                                                                                      
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm,aes256-gcm,chacha20-poly1305,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se                                                                                                                                                                      
debug2: kex_parse_kexinit: hmac-md5-etm,hmac-sha1-etm,umac-64-etm,umac-128-etm,hmac-sha2-256-etm,hmac-sha2-512-etm,hmac-ripemd160-etm,hmac-sha1-96-etm,hmac-md5-96-etm,hmac-md5,hmac-sha1,umac-64,umac-128,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm,hmac-sha1-etm,umac-64-etm,umac-128-etm,hmac-sha2-256-etm,hmac-sha2-512-etm,hmac-ripemd160-etm,hmac-sha1-96-etm,hmac-md5-96-etm,hmac-md5,hmac-sha1,umac-64,umac-128,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib,zlib
debug2: kex_parse_kexinit: none,zlib,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,3des-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,twofish256-cbc,twofish-cbc,twofish128-cbc
debug2: kex_parse_kexinit: aes128-ctr,3des-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,twofish256-cbc,twofish-cbc,twofish128-cbc
debug2: kex_parse_kexinit: hmac-sha1-96,hmac-sha1,hmac-md5
debug2: kex_parse_kexinit: hmac-sha1-96,hmac-sha1,hmac-md5
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: setup hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: diffie-hellman-group14-sha1 need=16 dh_need=16
debug1: kex: diffie-hellman-group14-sha1 need=16 dh_need=16
debug2: bits set: 1024/2048
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY



Expected result:
Able to login via ssh.


Note: MTU was suggested as the cause.
Comment 2 Alexander Chuzhoy 2015-12-03 20:14:52 UTC 
Created attachment 1101957 [details]
neutron conf and logs from one controller
Comment 3 James Slagle 2015-12-03 20:57:04 UTC 
if mtu was suggested as the root cause (which it very well may be), can you try a fedora instance instead of a cirros instance?

cirros only honored mtu as of 0.3.3:
https://bugs.launchpad.net/cirros/+bug/1301958

so if you used the qcow2 download from here:
https://launchpad.net/cirros/+download
the latest you'd have would be 0.3.0.
Comment 4 James Slagle 2015-12-03 20:58:45 UTC 
you can download fedora cloud images from https://getfedora.org/en/cloud/download/
Comment 5 Alexander Chuzhoy 2015-12-03 21:45:36 UTC 
Used a newer cirros image 0.3.3 (as suggested) and it worked fine.
Thanks.