Bug 1118865 - SSH key injection fails with no route to host
Summary: SSH key injection fails with no route to host
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Linux
unspecified
low
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Russell Bryant
QA Contact: Ami Jeain
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-11 18:30 UTC by jliberma@redhat.com
Modified: 2014-09-08 05:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-07-30 15:05:06 UTC


Attachments (Terms of Use)

Description jliberma@redhat.com 2014-07-11 18:30:41 UTC
Description of problem: Deployed osp5 via packstack on rhel 7. One controller, one compute. Launched instance using RHEL 7 guest image. SSH key injection fails with "no route to host." Hosts get IP address, can SSH and ping. tried running nova-api on controller, compute, and both. Works fine with Neutron network. Both in VLAN mode.

I added a root password with guestfish and checked /var/log/cloud-init.log on the instance. Error: no route to host http://172.16.3.1/openstack/meta-data

I tried curl http://169.254.169.254:8775/ and got no route to host.

This worked from the cirros image with same configuration.
 
Version-Release number of selected component (if applicable):
GA1 (July 03.1 puddle)

How reproducible:
I can reproduce it every time.

Steps to Reproduce:
1. Install rhel7 on 2 systems
2. Install OSP 5 via Packstack using answer file below
3. Add RHEL7 guest image
4. Create key pair
5. Launch instance with keypair
6. ssh to instance

Actual results:
Permission denied.

Expected results:
Logs in.

Additional info:

I use Vlan network manager with Vlans 1000:1010 configured on my switches. But I have same error with FlatDHCPManager.

In this answer file I try controller and compute as network hosts. (Both running nova-api service on 8775) but it does not work. I also tried each individually.

Also, I do not see a meta-data service or package which you used to install in multi_host mode.

CONFIG_SSH_KEY=/root/.ssh/id_rsa.pub
CONFIG_MYSQL_INSTALL=y
CONFIG_GLANCE_INSTALL=y
CONFIG_CINDER_INSTALL=y
CONFIG_NOVA_INSTALL=y
CONFIG_NEUTRON_INSTALL=n
CONFIG_HORIZON_INSTALL=n
CONFIG_SWIFT_INSTALL=n
CONFIG_CEILOMETER_INSTALL=n
CONFIG_HEAT_INSTALL=n
CONFIG_CLIENT_INSTALL=y
CONFIG_NTP_SERVERS=10.16.255.1,10.16.255.2
CONFIG_NAGIOS_INSTALL=n
EXCLUDE_SERVERS=
CONFIG_DEBUG_MODE=n
CONFIG_CONTROLLER_HOST=10.19.137.100
CONFIG_COMPUTE_HOSTS=10.19.137.101
CONFIG_NETWORK_HOSTS=10.19.137.100,10.19.137.101
CONFIG_VMWARE_BACKEND=n
CONFIG_VCENTER_HOST=
CONFIG_VCENTER_USER=
CONFIG_VCENTER_PASSWORD=
CONFIG_VCENTER_CLUSTER_NAME=
CONFIG_USE_EPEL=n
CONFIG_REPO=
CONFIG_RH_USER=
CONFIG_RH_PW=
CONFIG_RH_OPTIONAL=y
CONFIG_SATELLITE_URL=
CONFIG_SATELLITE_USER=
CONFIG_SATELLITE_PW=
CONFIG_SATELLITE_AKEY=
CONFIG_SATELLITE_CACERT=
CONFIG_SATELLITE_PROFILE=
CONFIG_SATELLITE_FLAGS=
CONFIG_SATELLITE_PROXY=
CONFIG_SATELLITE_PROXY_USER=
CONFIG_SATELLITE_PROXY_PW=
CONFIG_AMQP_BACKEND=rabbitmq
CONFIG_AMQP_HOST=10.19.137.100
CONFIG_AMQP_ENABLE_SSL=n
CONFIG_AMQP_ENABLE_AUTH=n
CONFIG_AMQP_NSS_CERTDB_PW=redhat
CONFIG_AMQP_SSL_PORT=5671
CONFIG_AMQP_SSL_CERT_FILE=/etc/pki/tls/certs/amqp_selfcert.pem
CONFIG_AMQP_SSL_KEY_FILE=/etc/pki/tls/private/amqp_selfkey.pem
CONFIG_AMQP_SSL_SELF_SIGNED=y
CONFIG_AMQP_AUTH_USER=amqp_user
CONFIG_AMQP_AUTH_PASSWORD=9effc100c017489e
CONFIG_MYSQL_HOST=10.19.137.100
CONFIG_MYSQL_USER=root
CONFIG_MYSQL_PW=redhat
CONFIG_KEYSTONE_DB_PW=redhat
CONFIG_KEYSTONE_ADMIN_TOKEN=123456
CONFIG_KEYSTONE_ADMIN_PW=redhat
CONFIG_KEYSTONE_DEMO_PW=redhat
CONFIG_KEYSTONE_TOKEN_FORMAT=PKI
CONFIG_GLANCE_DB_PW=redhat
CONFIG_GLANCE_KS_PW=redhat
CONFIG_CINDER_DB_PW=redhat
CONFIG_CINDER_KS_PW=redhat
CONFIG_CINDER_BACKEND=lvm
CONFIG_CINDER_VOLUMES_CREATE=y
CONFIG_CINDER_VOLUMES_SIZE=20G
CONFIG_CINDER_GLUSTER_MOUNTS=
CONFIG_CINDER_NFS_MOUNTS=
CONFIG_NOVA_DB_PW=redhat
CONFIG_NOVA_KS_PW=redhat
CONFIG_NOVA_SCHED_CPU_ALLOC_RATIO=16.0
CONFIG_NOVA_SCHED_RAM_ALLOC_RATIO=1.5
CONFIG_NOVA_COMPUTE_PRIVIF=p3p1
CONFIG_NOVA_NETWORK_MANAGER=nova.network.manager.VlanManager
CONFIG_NOVA_NETWORK_PUBIF=em1
CONFIG_NOVA_NETWORK_PRIVIF=p3p1
CONFIG_NOVA_NETWORK_FIXEDRANGE=172.16.3.0/24
CONFIG_NOVA_NETWORK_FLOATRANGE=10.19.137.112/28
CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL=nova
CONFIG_NOVA_NETWORK_AUTOASSIGNFLOATINGIP=n
CONFIG_NOVA_NETWORK_VLAN_START=1000
CONFIG_NOVA_NETWORK_NUMBER=1
CONFIG_NOVA_NETWORK_SIZE=255
CONFIG_NEUTRON_KS_PW=redhat
CONFIG_NEUTRON_DB_PW=redhat
CONFIG_NEUTRON_L3_EXT_BRIDGE=
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_METADATA_PW=redhat
CONFIG_LBAAS_INSTALL=n
CONFIG_NEUTRON_METERING_AGENT_INSTALL=n
CONFIG_NEUTRON_FWAAS=n
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_FLAT_NETWORKS=*
CONFIG_NEUTRON_ML2_VLAN_RANGES=
CONFIG_NEUTRON_ML2_TUNNEL_ID_RANGES=
CONFIG_NEUTRON_ML2_VXLAN_GROUP=
CONFIG_NEUTRON_ML2_VNI_RANGES=10:100
CONFIG_NEUTRON_L2_AGENT=openvswitch
CONFIG_NEUTRON_LB_TENANT_NETWORK_TYPE=local
CONFIG_NEUTRON_LB_VLAN_RANGES=
CONFIG_NEUTRON_LB_INTERFACE_MAPPINGS=
CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vxlan
CONFIG_NEUTRON_OVS_VLAN_RANGES=
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=
CONFIG_NEUTRON_OVS_BRIDGE_IFACES=
CONFIG_NEUTRON_OVS_TUNNEL_RANGES=
CONFIG_NEUTRON_OVS_TUNNEL_IF=
CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT=4789
CONFIG_HORIZON_SSL=n
CONFIG_SSL_CERT=
CONFIG_SSL_KEY=
CONFIG_SSL_CACHAIN=
CONFIG_SWIFT_KS_PW=redhat
CONFIG_SWIFT_STORAGES=
CONFIG_SWIFT_STORAGE_ZONES=1
CONFIG_SWIFT_STORAGE_REPLICAS=1
CONFIG_SWIFT_STORAGE_FSTYPE=ext4
CONFIG_SWIFT_HASH=df8084b5dc2349bd
CONFIG_SWIFT_STORAGE_SIZE=2G
CONFIG_PROVISION_DEMO=n
CONFIG_PROVISION_TEMPEST=n
CONFIG_PROVISION_TEMPEST_USER=
CONFIG_PROVISION_TEMPEST_USER_PW=redhat
CONFIG_PROVISION_DEMO_FLOATRANGE=10.19.137.112/28
CONFIG_PROVISION_TEMPEST_REPO_URI=https://github.com/openstack/tempest.git
CONFIG_PROVISION_TEMPEST_REPO_REVISION=master
CONFIG_PROVISION_ALL_IN_ONE_OVS_BRIDGE=n
CONFIG_HEAT_DB_PW=5redhat
CONFIG_HEAT_AUTH_ENC_KEY=797232e24a3a473c
CONFIG_HEAT_KS_PW=redhat
CONFIG_HEAT_CLOUDWATCH_INSTALL=n
CONFIG_HEAT_CFN_INSTALL=n
CONFIG_HEAT_DOMAIN=heat
CONFIG_HEAT_DOMAIN_ADMIN=heat_admin
CONFIG_HEAT_DOMAIN_PASSWORD=9f5708fd05fa43a7
CONFIG_CEILOMETER_SECRET=647452a90a9e4d30
CONFIG_CEILOMETER_KS_PW=redhat
CONFIG_MONGODB_HOST=10.19.137.100
CONFIG_NAGIOS_PW=redhat

# steps to reproduce
source /root/keystonerc_admin
env | grep OS_

# import the virtual machine disk image
rhn-channel --user=admin --password=password- --add --channel=rhel-x86_64-server-rh-common-7
yum install -y rhel-guest-image-7
glance image-create --progress --name rhel-server7 --is-public true --disk-format qcow2  --container-format bare --file /usr/share/rhel-guest-image-7/rhel-guest-image-7.0-20140506.1.x86_64.qcow2

# create tenant, user
keystone user-create --name refarch --pass refarch
keystone tenant-create --name refarch-tenant

# Add member role to user
keystone user-role-add --user-id refarch --tenant-id refarch-tenant --role-id _member_
keystone user-role-list --user-id refarch --tenant-id refarch-tenant

# create a keystonerc file for the refarch user
cat > /root/keystonerc_refarch << "EOF"
export OS_USERNAME=refarch
export OS_TENANT_NAME=refarch-tenant
export OS_PASSWORD=refarch
export OS_AUTH_URL=http://10.19.137.100:35357/v2.0/
export PS1='[\u@\h \W(refarch_member)]\$ '
EOF

# switch to refarch user
source /root/keystonerc_refarch
env | grep OS_

# create a keypair
nova keypair-add refarchkp > /root/refarchkp.pem
chmod 600 /root/refarchkp.pem

# boot an instance on each compute node
nova boot --poll --flavor 2 --image rhel-server7 --key-name refarchkp inst1

# add ssh and icmp rules to default secgroup
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0   

# add a floating ip
float_ip=$(nova floating-ip-create | awk ' /nova/ { print $2 } ')
nova add-floating-ip inst1 $float_ip
ssh -i /root/refarchkp.pem $float_ip uptime

Comment 2 jliberma@redhat.com 2014-07-11 19:16:15 UTC
Update:

I changed the public Nova IP address to the bridge private IP address (from the fixed pool) and restarted openstack services.

#metadata_host=10.19.137.100
metadata_host=172.16.3.1

After making that change the SSH key injection worked fine.

I will investigate how packstack chose the metadata_host address. I would expect it to start 0.0.0.0:8775 instead of the public IP address. Maybe I need to configure a metadata_proxy? I don't remember having to do this with previous versions of openstack.

[root@rhos0 nova(refarch_member)]# ssh -l cloud-user -i /root/refarchkp.pem  10.19.137.114 uptime
Warning: Permanently added '10.19.137.114' (ECDSA) to the list of known hosts.
 15:12:08 up 6 min,  0 users,  load average: 0.00, 0.04, 0.04

[root@rhos0 nova(refarch_member)]# nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                              |
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------+
| 1e400f91-2fb5-423c-aa63-48ae25f378ba | inst0 | ACTIVE | -          | Running     | novanetwork=172.16.3.4                |
| 73fd2289-e160-4511-a69c-daa66bea2ec0 | inst1 | ACTIVE | -          | Running     | novanetwork=172.16.3.3, 10.19.137.113 |
| b3d2f621-44e5-4244-8934-ab048d264d35 | inst2 | ACTIVE | -          | Running     | novanetwork=172.16.3.5                |
| 27619ab6-4066-40f5-95e4-b7f0285efde6 | inst3 | ACTIVE | -          | Running     | novanetwork=172.16.3.6, 10.19.137.114 |
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------+

[root@rhos0 nova(refarch_member)]# ssh -l cloud-user -i /root/refarchkp.pem  10.19.137.114 uptime
Warning: Permanently added '10.19.137.114' (ECDSA) to the list of known hosts.
 15:12:17 up 6 min,  0 users,  load average: 0.00, 0.04, 0.04

[root@rhos0 nova(refarch_member)]# netstat -na | grep 8775
tcp        0      0 0.0.0.0:8775            0.0.0.0:*               LISTEN

Comment 3 Russell Bryant 2014-07-16 15:23:02 UTC
Based on the latest update here, this appears to be a packstack issue(In reply to jliberma@redhat.com from comment #0)
> This worked from the cirros image with same configuration.

Was this with the same deployment where you saw the failure with the RHEL 7 image?

Based on your later update, it appears to have been an issue with how packstack configured your deployment.  I want to make sure the problem isn't specific to using the RHEL 7 image for some reason.

Comment 4 Russell Bryant 2014-07-30 15:05:06 UTC
Based on the lack of response here I'm going to assume that this is no longer an issue.  Feel free to reopen if there's something we need to address.

Comment 5 jliberma@redhat.com 2014-07-31 22:09:25 UTC
Russel, I think the problem might have been this:

https://bugzilla.redhat.com/show_bug.cgi?id=1105770

As I recall, I needed to reboot the controller after the install to use the latest kernel. 

Thanks, Jacob


Note You need to log in before you can comment on or make changes to this bug.