Bug 1289001 - Deployment failed: Deployment to server with non-zero status code
Deployment failed: Deployment to server with non-zero status code
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: chris alfonso
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-07 03:12 EST by Ronnie Rasouli
Modified: 2016-04-26 10:28 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-31 02:55:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
heat-config-puppet that failed (66.40 KB, text/plain)
2015-12-07 03:12 EST, Ronnie Rasouli
no flags Details

  None (edit)
Description Ronnie Rasouli 2015-12-07 03:12:14 EST
Created attachment 1103059 [details]
heat-config-puppet that failed

Description of problem:

version in controller:
openstack-heat-api-2015.1.2-2.el7ost.noarch
openstack-nova-api-2015.1.2-4.el7ost.noarch

version in undercloud:
openstack-heat-api-2015.1.2-4.el7ost.noarch
openstack-nova-api-2015.1.2-7.el7ost.noarch


Update failed from previous 7.2 puddle to latest puddle:

I have had previous 7.2 puddle and tried to update the stacks with update command:

openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-vip.yaml -e /home/stack/firstboot-env.yaml
Version-Release number of selected component (if applicable):

After the update failure I got a deployment failure:
[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| a6598ecd-a322-4a47-a7e7-9744e2bb4d4f | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| f9096fe0-0858-44b6-9234-47b9ab6718ca | overcloud-compute-0    | ERROR  | -          | NOSTATE     |                     |
| 80242f3e-993d-4fa1-b11a-7af02177d805 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| c66cfb6f-20a1-47ae-9c70-0d0dd7812bf3 | overcloud-controller-0 | ERROR  | -          | NOSTATE     |                     |
| 10923f6b-496d-44f3-8c9d-1bc79d80ecb7 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| c559474c-8819-4dbb-8572-937e45907f0a | overcloud-controller-1 | ERROR  | -          | NOSTATE     |                     |
| 4ee94847-7e1c-43fd-aa17-95282cbed3a3 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| fdfb3257-aa61-4ffc-8d14-09bb5bb9b36f | overcloud-controller-2 | ERROR  | -          | NOSTATE     |                     |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+



nova show f9096fe0-0858-44b6-9234-47b9ab6718ca

fault                                | {"message": "No valid host was found. Exceeded max scheduling attempts 3 for instance f9096fe0-0858-44b6-9234-47b9ab6718ca. Last exception: [u'Traceback (most recent call last):\                                         |
|                                      | ', u'  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2261, in _do", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 669, in build_instances |
|                                      |     instances[0].uuid)                                                                                                                                                                                                     |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 172, in populate_retry                                                                                                                           |
|                                      |     raise exception.NoValidHost(reason=msg)                                                                                                                                                                                |
|                                      | ", "created": "2015-12-06T12:13:56Z"}                                                                                                                
I am able to login into the nodes successfully
[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 33f63b8c-af99-42bc-8777-4557453c883f | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| 66e7cad9-31db-445b-af61-755225b49df0 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| 323b7171-47b8-45fc-baff-3f84585006b6 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 84d6266c-24a7-4d0e-943d-721a96c0d601 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+

openstack overcloud deploy --templates -e firstboot-env.yaml --control-scale 3 --compute-scale 1 --debug --log-file=overcloud_deploy.log --libvirt-type qemu --ntp-server clock.redhat.com

From heat-api.log

ck": {"parent": null, "disable_rollback": true, "description": "Nova API,Keystone,Heat Engine and API,Glance,Neutron,Dedicated MySQL server,Dedicated RabbitMQ Server,Group of Nova Computes\n", "links": [{"href": "http://192.0.2.1:8004/v1/ab55a17ce27341e29759d5e5c5d141d7/stacks/overcloud/10862cca-1348-4e11-93aa-6062015134a1", "rel": "self"}], "stack_status_reason": "Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1", "stack_name": "overcloud", "stack_user_project_id": "128fd809cad0423fb35dc7f7e6311c54", "stack_owner": "admin", "creation_time": "2015-12-06T13:55:46Z", "capabilities": [], "notification_topics": [], "updated_time": null, "timeout_mins": 240, "stack_status": "CREATE_FAILED", "parameters": {"NeutronPublicInterfaceRawDevice": "", "KeystoneSignin

[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 71a6dd2c-4ebd-428f-9384-a75ed4275444 | overcloud  | CREATE_FAILED | 2015-12-06T14:30:26Z |
+--------------------------------------+------------+---------------+----------------------+


How reproducible:


Steps to Reproduce:
1.deploy overcloud HA 3 controllers and 1 compute, must be successful deployment
2. update with the  command above
3. delete the stack and re deploy

Actual results:

Exit code 1 failed to deploy

Expected results:

Although the update failed, ability to redeploy the stacks after deleting and re deploying

Additional info:
In controllers 

Error: Could not find data item step in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d0377.pp:36 on node overcloud-controller-1.localdomain

Dec  6 09:42:11 localhost os-collect-config: [2015-12-06 09:42:11,403] (heat-config) [INFO] {"deploy_stdout": "", "deploy_stderr": "\u001b[1;31mWarning: Scope(Class[Keystone]): Execution of 
db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_servic
e or $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host';
 class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; 
class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class
 ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::no
va::compute has not been evaluated\u001b[0m\n\u001b[1;31mError: nova_admin_password must be set. at /etc/puppet/modules/neutron/manifests/server/notifications.pp:80 on node overcloud-control
ler-1.localdomain\u001b[0m\n\u001b[1;31mError: nova_admin_password must be set. at /etc/puppet/modules/neutron/manifests/server/notifications.pp:80 on node overcloud-controller-1.localdomain
\u001b[0m\n", "deploy_status_code": 1}
Dec  6 09:42:11 localhost os-collect-config: [2015-12-06 09:42:11,404] (heat-config) [DEBUG] [2015-12-06 09:42:04,989] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-c
onfig/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d0377"  FACTER_fqdn="overcloud-controller-1.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-4d7k3fypb
c3m-ControllerPuppetConfig-wcechcxqwplg-ControllerPuppetConfigImpl-5vd2qdyleaa6"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d03
77.pp
Dec  6 09:42:11 localhost os-collect-config: [2015-12-06 09:42:11,400] (heat-config) [INFO] Return code 1


Attached 

/var/lib/heat-config/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d0377.pp
Comment 2 Zane Bitter 2015-12-07 08:25:53 EST
Not a Heat bug. If an error occurs and Heat reports it, Heat is doing its job correctly.
Comment 4 James Slagle 2015-12-08 09:22:13 EST
i need some clarification on exactly what happened here.

What are we trying to test?

You said;
> Steps to Reproduce:
> 1.deploy overcloud HA 3 controllers and 1 compute, must be successful deployment
> 2. update with the  command above
> 3. delete the stack and re deploy

Did the update fail at step 2? Can we debug that failure first if so?

Or are you trying to test deleting a stack and redeploying after an update?

Note You need to log in before you can comment on or make changes to this bug.