Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 914648

Summary: Packstack should create and distribute nova keys
Product: Red Hat OpenStack Reporter: Jaroslav Henner <jhenner>
Component: openstack-packstackAssignee: Martin Magr <mmagr>
Status: CLOSED ERRATA QA Contact: Jaroslav Henner <jhenner>
Severity: medium Docs Contact:
Priority: high    
Version: 2.0 (Folsom)CC: adahms, aortega, dasmith, derekh, ichavero, jkt, lars, mmagr, yfried
Target Milestone: rcKeywords: FutureFeature, Triaged
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-packstack-2014.1.1-0.26.dev1157.el6ost Doc Type: Enhancement
Doc Text:
This update introduces support for generating and distributing SSH keys to provide support for migrating instances via connections using QEMU and SSH to remote hypervisors. This update also ensures QEMU migration ports are open for Nova compute nodes and that Nova passes 'no_verify' to libvirt, making it possible for Nova to access compute nodes correctly when resizing instances.
Story Points: ---
Clone Of:
: 1082628 (view as bug list) Environment:
Last Closed: 2014-07-08 15:37:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1082628    
Attachments:
Description Flags
migration fails none

Description Jaroslav Henner 2013-02-22 12:01:09 UTC
Description of problem:
When resizing the instances, nova tries to access nova-compute hosts using ssh. If there are no keys, it will fail:
2013-02-22 11:21:34 ERROR nova.compute.manager [req-44ea0be9-8c20-46de-9aba-bd362bf65cd9 9e8db273de33472b9aca6d0d3adf08e4 6b60892144aa4704bde2f66e2fa3ab7c] [instance: 1d6a5dbb-f774-424b-81f0-6117caaee943] Unexpected error while running command.
Command: ssh 10.34.69.3 mkdir -p /var/lib/nova/instances/instance-00000041
Exit code: 255
Stdout: ''
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to ERROR

I think it is also needed to do:
semanage permissive -a sshd_t
which AFAIK is only a workaround for missing selinux policies.


Version-Release number of selected component (if applicable):
openstack-nova-2012.2.3-1.el6ost.noarch
openstack-packstack-2012.2.2-1.0.dev408.el6ost.noarch


How reproducible:
100%


Steps to Reproduce:
1. Have controller and two hosts.
2. nosetests --tests tempest.tests.compute.servers.test_disk_config:TestServerDiskConfig.test_resize_server_from_auto_to_manual
  

Actual results:
Instance set to ERROR.

Expected results:
Instance resized.

Additional info:

Comment 2 Solly Ross 2014-01-23 20:39:56 UTC
There is a real fix (making the SSHing here go away) in the works

Comment 3 Perry Myers 2014-03-31 13:56:56 UTC
Adding some additional thoughts/context here...

Presently Packstack only distributes the public keys to all remote hosts, to allow the main node running packstack to talk to all remote hosts.  Adding the private key as well would allow passwordless ssh between all compute nodes, which would enable support for Nova migrate/resize using libvirt migration over root passwordless ssh tunnels.

NOTE: This is not a good idea for production environments, but since Packstack is meant for PoC and demos, this is a reasonable option.  Even still, we probably want to make enabling of distributing the private key files an explicitly opt-in vs. making it done by default.

There is (as mentioned in comment #2) separate work ongoing to make it so that Nova doesn't rely on libvirt/ssh tunnels, but until that work completes we probably need to rely on this mechanism at least for Packstack.

Comment 4 Jaroslav Henner 2014-04-01 11:53:38 UTC
(In reply to Perry Myers from comment #3)
> Adding some additional thoughts/context here...
> 
> Presently Packstack only distributes the public keys to all remote hosts, to
> allow the main node running packstack to talk to all remote hosts.  Adding
> the private key as well would allow passwordless ssh between all compute
> nodes, which would enable support for Nova migrate/resize using libvirt
> migration over root passwordless ssh tunnels.
> 
> NOTE: This is not a good idea for production environments, but since
> Packstack is meant for PoC and demos, this is a reasonable option.  Even
> still, we probably want to make enabling of distributing the private key
> files an explicitly opt-in vs. making it done by default.
> 
> There is (as mentioned in comment #2) separate work ongoing to make it so
> that Nova doesn't rely on libvirt/ssh tunnels, but until that work completes
> we probably need to rely on this mechanism at least for Packstack.

We don't need to distribute root keys but nova keys. On grizzly, nova tries to ssh as nova, not root. If it is different on newer releases, I would call that a bug.

Comment 5 Dan Smith 2014-04-06 15:16:38 UTC
Nova doesn't specify a user for ssh, so whatever user nova is running as is what is used for the copy. So yes, still in icehouse, this means it happens as the nova user for us.

Comment 6 Lars Kellogg-Stedman 2014-04-09 20:39:38 UTC
Going back to the "don't fix things in packstack" thread, it seems like this should actually go into the Nova puppet module (which is already responsible for creating the nova user).

Here's my proposal for how to handle this in the puppet module:

  https://github.com/larsks/puppet-nova/blob/feature/sshkeys/manifests/init.pp#L304

Does that make sense?  Given an appropriate packstack template these changes appear to Do the Right Thing, modulo some selinux issues (the nova home directory is labelled nova_var_lib_t, to which sshd does not have access...this would probably require a fix in the selinux package).

If people like this solution, I'll push it upstream and work on the corresponding support in packstack.  We would need to generate an ssh keypair here and then include both the private and public keys as literals in the generated manifests.

Comment 7 Dan Smith 2014-04-09 20:43:42 UTC
Note that for live migrations, we need keys for root as well. Also note that both types of migrations will require that host keys for all nodes be installed on every node before migrations can work.

Comment 12 Jaroslav Henner 2014-07-01 10:34:55 UTC
I have some problems with this change:

 * The no_verify=1 is used in the uri, which I think enables MITM attack.
 * The key is in /etc/nova/ssh while the nova home is ~nova (/var/lib/nova). I am not sure it is wise or not to have the key in /etc instead of ~nova.

Comment 13 yfried 2014-07-06 14:37:47 UTC
is there a new option for packstack to enable this?
right now this still fails with
openstack-packstack-2014.1.1-0.32.1.dev1209.el7ost

Comment 14 Jaroslav Henner 2014-07-06 14:51:42 UTC
Created attachment 914967 [details]
migration fails

As yfried found, this is still broken. When doing nova migrate (resize to another host), nova tries to ssh to the other host, resulting permissions denied.

Comment 16 errata-xmlrpc 2014-07-08 15:37:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0846.html