Bug 1156000

Summary: Resize of flavor and migrate fails due to insufficient SSH keys distribution
Product: Red Hat OpenStack Reporter: Marian Krcmarik <mkrcmari>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED DUPLICATE QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact: Mike Burns <mburns>
Priority: high    
Version: 7.0 (Kilo)CC: hbrock, mburns, oblaut, pasik, rhel-osp-director-maint, rhos-maint, sgordon, srevivo, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: Director   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-28 16:43:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1156010    

Description Marian Krcmarik 2014-10-23 11:27:40 UTC
Description of problem:
SSH keys distribution is needed for some Openstack features (such as resize of an instance to diffent flavor which uses SSH migration and There are some checks through SSH of existence of storage). SSh keys should be generated on the compute nodes for nova user as well as It should be added to authorized keys of other nodes and a node should has other nodes in known_hosts for nova user, so the ssh commands does not fail.
This is implemted in packstack in RHOS5.0 but not in staypuft

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. In RHOS setup deployed by staypuft try to resize running instance.

Actual results:
It fails due to missing ssh keys distributed.
Command: ssh 192.168.0.7 mkdir -p /var/lib/nova/instances/07ff020f-c1db-41d9-a55d-0dec2448d2cd
2014-10-22 12:34:33.738 22767 TRACE nova.compute.manager [instance: 07ff020f-c1db-41d9-a55d-0dec2448d2cd] Exit code: 255
2014-10-22 12:34:33.738 22767 TRACE nova.compute.manager [instance: 07ff020f-c1db-41d9-a55d-0dec2448d2cd] Stdout: u''
2014-10-22 12:34:33.738 22767 TRACE nova.compute.manager [instance: 07ff020f-c1db-41d9-a55d-0dec2448d2cd] Stderr: u'Host key verification failed.\r\n'


Expected results:
SSH keys should be generated and distributed correctly.

Additional info:

Comment 3 Marian Krcmarik 2015-01-22 14:04:29 UTC
To clarify more:
One of the other features which uses SSH is offline migration triggered by command "nova migrate instance". Staypuft does not generate any SSH keys for nodes (compute and controller) as well as It does not place such generated keys into authorized_keys on the other nodes - so There is no SSH key distribution. This cause all the feature which depend on SSH to fail - in the case of migration (nova migrate) instance ends up in ERROR state. There is a live migration triggered by command "nova live-migration" which does not use SSH and works smoothly but from user point of perspective I would say that users will find and use "nova migrate" command firstly which will place their instances into ERROR state.

Packstack does generate public SSH keys for this purpose and place it into authorized_keys on the nodes but It did not work for me as well unless I allowed login shell for nova user and places the ssh public keys of the nodes into known_hosts file of nova users on the other nodes. I have separated bug #1156010 for allowing the nova user to have a login shell which may have security impact.

I am raising severity since some of the features do not work out of the box. Possibly this feature "nova migrate" could be taken away since "nova live-migration" works without use of SSH (over TCP).

Comment 4 Ofer Blaut 2015-01-28 10:05:44 UTC
I have tested VM resize on latest rhel-osp 6 and it fails

to reproduce :
create VM
check its flavour 
resize it flavour to bigger one


[root@controller ~(keystone_admin_tenant1)]$nova resize 1781233a-75a8-478d-95f3-bc5149f5a3d2 3
[root@controller ~(keystone_admin_tenant1)]$nova show 1781233a-75a8-478d-95f3-bc5149f5a3d2
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                            |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                           |
| OS-EXT-AZ:availability_zone          | nova                                                                                             |
| OS-EXT-STS:power_state               | 1                                                                                                |
| OS-EXT-STS:task_state                | -                                                                                                |
| OS-EXT-STS:vm_state                  | error                                                                                            |
| OS-SRV-USG:launched_at               | 2015-01-20T06:15:30.000000                                                                       |
| OS-SRV-USG:terminated_at             | -                                                                                                |
| accessIPv4                           |                                                                                                  |
| accessIPv6                           |                                                                                                  |
| config_drive                         |                                                                                                  |
| created                              | 2015-01-20T06:14:10Z                                                                             |
| fault                                | {"message": "Unexpected error while running command.                                             |
|                                      | Command: ssh 10.35.186.231 mkdir -p /var/lib/nova/instances/1781233a-75a8-478d-95f3-bc5149f5a3d2 |
|                                      | Exit code: 255                                                                                   |
|                                      | Stdout: u''                                                                                      |
|                                      | Stderr: u'Host key verification failed.\\r\                                                      |
|                                      | '", "code": 500, "created": "2015-01-28T08:49:48Z"}                                              |
| flavor                               | m1.small (2)                                                                                     |
| hostId                               | 5d600347a8839a6ad387e64de85d88b9aa9318fab380d0e29daa7ef0                                         |
| id                                   | 1781233a-75a8-478d-95f3-bc5149f5a3d2                                                             |
| image                                | cirros (5b099a2c-a682-466f-82c1-73a229118342)                                                    |
| key_name                             | oskey1                                                                                           |
| metadata                             | {}                                                                                               |
| name                                 | C                                                                                                |
| net201 network                       | 3001::f816:3eff:fe52:d005, 21.0.0.5                                                              |
| os-extended-volumes:volumes_attached | []                                                                                               |
| security_groups                      | default                                                                                          |
| status                               | ERROR                                                                                            |
| tenant_id                            | caa7603588c14954b2f4f6616f87d4bc                                                                 |
| updated                              | 2015-01-28T08:49:48Z                                                                             |
| user_id                              | efa9b7baadc149bd8082eede59a3121c

Comment 6 Ivan Chavero 2015-02-09 22:44:55 UTC
This seems to be a Staypuft problem, changing component

Comment 19 Marian Krcmarik 2015-07-29 12:37:07 UTC
I am moving this bug to osp-d since the same problem occurs there as already said this was supported by packstack but it lacks of support by staypuft or osp-d so feature as resize flavor or offline migration does not work out of box on deployments deployed by osp-d.

Comment 20 Mike Burns 2015-08-28 16:43:05 UTC

*** This bug has been marked as a duplicate of bug 1221776 ***