Bug 1400140

Summary: [Backwards Compatibility] New UC10-OC9 deployment fails with "Deployment exited with non-zero status code: 1"
Product: Red Hat OpenStack Reporter: Dan Yasny <dyasny>
Component: rhosp-directorAssignee: Adriano Petrich <apetrich>
Status: CLOSED CURRENTRELEASE QA Contact: Omri Hochman <ohochman>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: dbecker, dyasny, jcoufal, mandreou, mburns, morazi, rhel-osp-director-maint, sclewis
Target Milestone: rcKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-16 16:50:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dan Yasny 2016-11-30 14:15:54 UTC
Description of problem:
2016-11-30 04Heat Stack create failed.
:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment.0]: SIGNAL_IN_PROGRESS  Signal: deployment 6458f311-4106-46bd-84f0-10b7f2e07b4b failed (1)
2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with no
n-zero status code: 1
2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Dep
loyment exited with non-zero status code: 1
2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment]: CREATE_FAILED  Error: resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_
code: Deployment exited with non-zero status code: 1
2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment]: CREATE_FAILED  Resource CREATE failed: Error: resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_
code: Deployment exited with non-zero status code: 1
2016-11-30 04:17:48Z [overcloud.ComputeNodesPostDeployment]: CREATE_FAILED  Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: d
eploy_status_code: Deployment exited with non-zero status code: 1
2016-11-30 04:17:48Z [overcloud.ControllerNodesPostDeployment]: CREATE_FAILED  CREATE aborted  
2016-11-30 04:17:48Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: depl
oy_status_code: Deployment exited with non-zero status code: 1
2016-11-30 04:17:49Z [overcloud.ControllerNodesPostDeployment.ControllerLoadBalancerDeployment_Step1]: CREATE_FAILED  CREATE aborted
2016-11-30 04:17:49Z [overcloud.ControllerNodesPostDeployment]: CREATE_FAILED  Resource CREATE failed: Operation cancelled

 Stack overcloud CREATE_FAILED


Version-Release number of selected component (if applicable):
openstack-puppet-modules-9.3.0-1.el7ost.noarch
openstack-ceilometer-api-7.0.0-4.el7ost.noarch
openstack-neutron-common-9.1.0-7.el7ost.noarch
openstack-nova-cert-14.0.2-7.el7ost.noarch
openstack-ceilometer-central-7.0.0-4.el7ost.noarch
puppet-openstacklib-9.4.0-3.el7ost.noarch
openstack-glance-13.0.0-1.el7ost.noarch
openstack-ironic-conductor-6.2.2-2.el7ost.noarch
openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch
openstack-heat-common-7.0.0-7.el7ost.noarch
openstack-neutron-9.1.0-7.el7ost.noarch
openstack-heat-templates-0-0.9.1e6015dgit.el7ost.noarch
openstack-ceilometer-notification-7.0.0-4.el7ost.noarch
openstack-ceilometer-collector-7.0.0-4.el7ost.noarch
python-openstacksdk-0.9.5-1.el7ost.noarch
openstack-mistral-engine-3.0.2-9.el7ost.noarch
python-openstackclient-3.2.0-2.el7ost.noarch
openstack-selinux-0.7.12-1.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
python-openstack-mistral-3.0.2-9.el7ost.noarch
openstack-mistral-executor-3.0.2-9.el7ost.noarch
openstack-swift-container-2.10.0-6.el7ost.noarch
openstack-nova-api-14.0.2-7.el7ost.noarch
openstack-tripleo-validations-5.1.0-5.el7ost.noarch
openstack-aodh-listener-3.0.1-4.el7ost.noarch
openstack-mistral-api-3.0.2-9.el7ost.noarch
openstack-ironic-inspector-4.2.1-1.el7ost.noarch
openstack-aodh-api-3.0.1-4.el7ost.noarch
openstack-heat-engine-7.0.0-7.el7ost.noarch
openstack-aodh-notifier-3.0.1-4.el7ost.noarch
openstack-tripleo-puppet-elements-5.1.0-2.el7ost.noarch
openstack-aodh-evaluator-3.0.1-4.el7ost.noarch
openstack-heat-api-7.0.0-7.el7ost.noarch
openstack-nova-scheduler-14.0.2-7.el7ost.noarch
openstack-swift-account-2.10.0-6.el7ost.noarch
openstack-swift-proxy-2.10.0-6.el7ost.noarch
openstack-tempest-13.0.0-7.bafe630git.el7ost.noarch
openstack-ironic-common-6.2.2-2.el7ost.noarch
openstack-zaqar-3.0.0-3.el7ost.noarch
openstack-neutron-ml2-9.1.0-7.el7ost.noarch
openstack-ironic-api-6.2.2-2.el7ost.noarch
openstack-heat-api-cfn-7.0.0-7.el7ost.noarch
openstack-keystone-10.0.0-3.el7ost.noarch
openstack-tripleo-image-elements-5.1.0-1.el7ost.noarch
openstack-ceilometer-polling-7.0.0-4.el7ost.noarch
openstack-tripleo-common-5.4.0-3.el7ost.noarch
openstack-nova-common-14.0.2-7.el7ost.noarch
openstack-nova-compute-14.0.2-7.el7ost.noarch
openstack-aodh-common-3.0.1-4.el7ost.noarch
openstack-nova-conductor-14.0.2-7.el7ost.noarch
openstack-ceilometer-common-7.0.0-4.el7ost.noarch
openstack-swift-object-2.10.0-6.el7ost.noarch
openstack-mistral-common-3.0.2-9.el7ost.noarch
openstack-neutron-openvswitch-9.1.0-7.el7ost.noarch
openstack-tripleo-ui-1.0.5-2.el7ost.noarch
openstack-tripleo-heat-templates-compat-2.0.0-41.el7ost.noarch
puppet-openstack_extras-9.4.0-1.el7ost.noarch


How reproducible:
always (with/without SSL/IPv4/6)

Steps to Reproduce:
1. Deploy mixed environment, install UC10 and OC9
2. Deployment command: openstack overcloud deploy --templates /home/stack/tht --control-scale 3 --compute-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server clock.redhat
.com --timeout 90 -e /home/stack/tht/environments/puppet-pacemaker.yaml -e /home/stack/tht/environments/storage-environment.yaml -e /home/stack/tht/environments/network-isolation.yaml -e network-env
ironment.yaml --ceph-storage-scale 1
3.

Actual results:
fails

Expected results:
create_complete

Additional info:

Tried with two OC9 images: 
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=526505
and rhosp-director-images-9.0-20161031.1.el7ost.noarch.rpm

env available for debug

Comment 1 Dan Yasny 2016-11-30 15:00:15 UTC
    "deploy_stdout": "Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Aug  3 2016 08:33:27'
",
    "deploy_stderr": "exception: connect failed
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[mai
n]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1
;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_confi
g_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova]): Could not look up qualif
ied variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova]): Could not look up qualified 
variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova]): Could not look up qualified vari
able '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Compute]): compute_manager is marked 
as deprecated in Nova but still needed when Ironic is used. It will be removed once Nova removes it.\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified 
variable '::nova::vncproxy::host'; class ::nova::vncproxy has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vn
cproxy::vncproxy_protocol'; class ::nova::vncproxy has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy:
:port'; class ::nova::vncproxy has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_path'; cla
ss ::nova::vncproxy has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class
[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u00
1b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_c
onfig_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qu
alified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[
main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001
b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_co
nfig_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qua
lified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mWarning: Scope(Class[m
ain]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m
\u001b[1;31mError: Duplicate declaration: Nova_config[DEFAULT/host] is already declared in file /etc/puppet/modules/nov
a/manifests/init.pp:653; cannot redeclare at /var/lib/heat-config/heat-config-puppet/cefd8eb6-7a44-45f1-aeb1-aa9d24c411d9.pp:107 on node overcloud-compute-0.localdomain\u001b[0m
\u001b[1;31mError: 
Duplicate declaration: Nova_config[DEFAULT/host] is already declared in file /etc/puppet/modules/nova/manifests/init.pp:653; cannot redeclare at /var/lib/heat-config/heat-config-puppet/cefd8eb6-7a44
-45f1-aeb1-aa9d24c411d9.pp:107 on node overcloud-compute-0.localdomain\u001b[0m

Comment 2 Dan Yasny 2016-11-30 15:11:37 UTC
[stack@instack overcloud_images]$ ./list_nodes_status 
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| d6306966-c550-424d-9e92-b381759460d6 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
| 9810aa3e-ceee-4381-b837-93df60b4df5f | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.6  |
| 0a7f5551-d2f9-4de9-96da-4be86ce7ff40 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 3610d2fa-1d1a-4f70-877f-986c0f3f6cbc | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| dbde31d4-d6a5-4a9b-9987-cb1e8b390114 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
** 192.0.2.12 up and running
** 192.0.2.6 up and running
** 192.0.2.19 up and running
** 192.0.2.18 up and running
** 192.0.2.14 up and running

Comment 3 Dan Yasny 2016-11-30 15:14:16 UTC
[stack@instack overcloud_images]$ heat resource-list --nested-depth 5 overcloud | grep FAILED
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| ComputeNodesPostDeployment                   | ea1602d6-27a0-4142-bd1c-852224c9914b          | OS::TripleO::ComputePostDeployment                                                                    | CREATE_FAILED   | 2016-11-30T04:10:13Z | overcloud                                                                                                                                       |
| ControllerNodesPostDeployment                | 4a287ba9-39a7-453b-b302-f4be7a613e7e          | OS::TripleO::ControllerPostDeployment                                                                 | CREATE_FAILED   | 2016-11-30T04:10:13Z | overcloud                                                                                                                                       |
| ComputePuppetDeployment                      | 959aae89-cd0d-4285-8a4d-6c3159438cb8          | OS::Heat::StructuredDeployments                                                                       | CREATE_FAILED   | 2016-11-30T04:16:59Z | overcloud-ComputeNodesPostDeployment-b4l36sv2cgwz                                                                                               |
| ControllerLoadBalancerDeployment_Step1       | 9a550a39-963d-412f-8258-cfecfbf11536          | OS::Heat::StructuredDeployments                                                                       | CREATE_FAILED   | 2016-11-30T04:17:00Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat                                                                                            |
| 0                                            | 6458f311-4106-46bd-84f0-10b7f2e07b4b          | OS::Heat::StructuredDeployment                                                                        | CREATE_FAILED   | 2016-11-30T04:17:27Z | overcloud-ComputeNodesPostDeployment-b4l36sv2cgwz-ComputePuppetDeployment-nxzpey7ta5av                                                          |
| 0                                            | b7ed4746-e264-49aa-814e-1f24a012ca5e          | OS::Heat::StructuredDeployment                                                                        | CREATE_FAILED   | 2016-11-30T04:17:40Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat-ControllerLoadBalancerDeployment_Step1-4o6vosgln7v6                                        |
| 1                                            | 6d29e39a-192f-4653-bdc4-fefe7c1d0666          | OS::Heat::StructuredDeployment                                                                        | CREATE_FAILED   | 2016-11-30T04:17:40Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat-ControllerLoadBalancerDeployment_Step1-4o6vosgln7v6                                        |
| 2                                            | 82c7d522-74e6-44ad-ae89-1d36c72e30d8          | OS::Heat::StructuredDeployment                                                                        | CREATE_FAILED   | 2016-11-30T04:17:40Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat-ControllerLoadBalancerDeployment_Step1-4o6vosgln7v6                                        |

Comment 4 Marios Andreou 2016-11-30 15:51:16 UTC
readding the DFG:DF based on the fact that this is an OSP9 deployment failing (albeit with OSP10 undercloud). According to dyasny, this was a passing job last week so the concern is this is a regression.

lbezdick from lifecycle is currently trying to triage on the environment with dyasny. 

apetrich will continue to look when he comes in tomorrow morning.

Comment 9 Jaromir Coufal 2016-11-30 23:43:55 UTC
Thanks Dan! Do we need a documentation fix or was the issue in our automation?

Comment 11 Dan Yasny 2016-12-01 16:04:49 UTC
(In reply to Jaromir Coufal from comment #9)
> Thanks Dan! Do we need a documentation fix or was the issue in our
> automation?

I don't think so, though it would be great to be able to see the versions in glance so a user can be certain which image he has uploaded