Hide Forgot
Description of problem: 2016-11-30 04Heat Stack create failed. :17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment.0]: SIGNAL_IN_PROGRESS Signal: deployment 6458f311-4106-46bd-84f0-10b7f2e07b4b failed (1) 2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with no n-zero status code: 1 2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Dep loyment exited with non-zero status code: 1 2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment.ComputePuppetDeployment]: CREATE_FAILED Error: resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_ code: Deployment exited with non-zero status code: 1 2016-11-30 04:17:47Z [overcloud.ComputeNodesPostDeployment]: CREATE_FAILED Resource CREATE failed: Error: resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_ code: Deployment exited with non-zero status code: 1 2016-11-30 04:17:48Z [overcloud.ComputeNodesPostDeployment]: CREATE_FAILED Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: d eploy_status_code: Deployment exited with non-zero status code: 1 2016-11-30 04:17:48Z [overcloud.ControllerNodesPostDeployment]: CREATE_FAILED CREATE aborted 2016-11-30 04:17:48Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: depl oy_status_code: Deployment exited with non-zero status code: 1 2016-11-30 04:17:49Z [overcloud.ControllerNodesPostDeployment.ControllerLoadBalancerDeployment_Step1]: CREATE_FAILED CREATE aborted 2016-11-30 04:17:49Z [overcloud.ControllerNodesPostDeployment]: CREATE_FAILED Resource CREATE failed: Operation cancelled Stack overcloud CREATE_FAILED Version-Release number of selected component (if applicable): openstack-puppet-modules-9.3.0-1.el7ost.noarch openstack-ceilometer-api-7.0.0-4.el7ost.noarch openstack-neutron-common-9.1.0-7.el7ost.noarch openstack-nova-cert-14.0.2-7.el7ost.noarch openstack-ceilometer-central-7.0.0-4.el7ost.noarch puppet-openstacklib-9.4.0-3.el7ost.noarch openstack-glance-13.0.0-1.el7ost.noarch openstack-ironic-conductor-6.2.2-2.el7ost.noarch openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch openstack-heat-common-7.0.0-7.el7ost.noarch openstack-neutron-9.1.0-7.el7ost.noarch openstack-heat-templates-0-0.9.1e6015dgit.el7ost.noarch openstack-ceilometer-notification-7.0.0-4.el7ost.noarch openstack-ceilometer-collector-7.0.0-4.el7ost.noarch python-openstacksdk-0.9.5-1.el7ost.noarch openstack-mistral-engine-3.0.2-9.el7ost.noarch python-openstackclient-3.2.0-2.el7ost.noarch openstack-selinux-0.7.12-1.el7ost.noarch openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch python-openstack-mistral-3.0.2-9.el7ost.noarch openstack-mistral-executor-3.0.2-9.el7ost.noarch openstack-swift-container-2.10.0-6.el7ost.noarch openstack-nova-api-14.0.2-7.el7ost.noarch openstack-tripleo-validations-5.1.0-5.el7ost.noarch openstack-aodh-listener-3.0.1-4.el7ost.noarch openstack-mistral-api-3.0.2-9.el7ost.noarch openstack-ironic-inspector-4.2.1-1.el7ost.noarch openstack-aodh-api-3.0.1-4.el7ost.noarch openstack-heat-engine-7.0.0-7.el7ost.noarch openstack-aodh-notifier-3.0.1-4.el7ost.noarch openstack-tripleo-puppet-elements-5.1.0-2.el7ost.noarch openstack-aodh-evaluator-3.0.1-4.el7ost.noarch openstack-heat-api-7.0.0-7.el7ost.noarch openstack-nova-scheduler-14.0.2-7.el7ost.noarch openstack-swift-account-2.10.0-6.el7ost.noarch openstack-swift-proxy-2.10.0-6.el7ost.noarch openstack-tempest-13.0.0-7.bafe630git.el7ost.noarch openstack-ironic-common-6.2.2-2.el7ost.noarch openstack-zaqar-3.0.0-3.el7ost.noarch openstack-neutron-ml2-9.1.0-7.el7ost.noarch openstack-ironic-api-6.2.2-2.el7ost.noarch openstack-heat-api-cfn-7.0.0-7.el7ost.noarch openstack-keystone-10.0.0-3.el7ost.noarch openstack-tripleo-image-elements-5.1.0-1.el7ost.noarch openstack-ceilometer-polling-7.0.0-4.el7ost.noarch openstack-tripleo-common-5.4.0-3.el7ost.noarch openstack-nova-common-14.0.2-7.el7ost.noarch openstack-nova-compute-14.0.2-7.el7ost.noarch openstack-aodh-common-3.0.1-4.el7ost.noarch openstack-nova-conductor-14.0.2-7.el7ost.noarch openstack-ceilometer-common-7.0.0-4.el7ost.noarch openstack-swift-object-2.10.0-6.el7ost.noarch openstack-mistral-common-3.0.2-9.el7ost.noarch openstack-neutron-openvswitch-9.1.0-7.el7ost.noarch openstack-tripleo-ui-1.0.5-2.el7ost.noarch openstack-tripleo-heat-templates-compat-2.0.0-41.el7ost.noarch puppet-openstack_extras-9.4.0-1.el7ost.noarch How reproducible: always (with/without SSL/IPv4/6) Steps to Reproduce: 1. Deploy mixed environment, install UC10 and OC9 2. Deployment command: openstack overcloud deploy --templates /home/stack/tht --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat .com --timeout 90 -e /home/stack/tht/environments/puppet-pacemaker.yaml -e /home/stack/tht/environments/storage-environment.yaml -e /home/stack/tht/environments/network-isolation.yaml -e network-env ironment.yaml --ceph-storage-scale 1 3. Actual results: fails Expected results: create_complete Additional info: Tried with two OC9 images: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=526505 and rhosp-director-images-9.0-20161031.1.el7ost.noarch.rpm env available for debug
"deploy_stdout": "Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux) Server built: Aug 3 2016 08:33:27' ", "deploy_stderr": "exception: connect failed \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[mai n]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1 ;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_confi g_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[Nova]): Could not look up qualif ied variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova]): Could not look up qualified vari able '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Compute]): compute_manager is marked as deprecated in Nova but still needed when Ironic is used. It will be removed once Nova removes it.\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::host'; class ::nova::vncproxy has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vn cproxy::vncproxy_protocol'; class ::nova::vncproxy has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy: :port'; class ::nova::vncproxy has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_path'; cla ss ::nova::vncproxy has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class [main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u00 1b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_c onfig_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qu alified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[ main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001 b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_co nfig_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qua lified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[main]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mWarning: Scope(Class[m ain]): Could not look up qualified variable '::deploy_config_name'; \u001b[0m \u001b[1;31mError: Duplicate declaration: Nova_config[DEFAULT/host] is already declared in file /etc/puppet/modules/nov a/manifests/init.pp:653; cannot redeclare at /var/lib/heat-config/heat-config-puppet/cefd8eb6-7a44-45f1-aeb1-aa9d24c411d9.pp:107 on node overcloud-compute-0.localdomain\u001b[0m \u001b[1;31mError: Duplicate declaration: Nova_config[DEFAULT/host] is already declared in file /etc/puppet/modules/nova/manifests/init.pp:653; cannot redeclare at /var/lib/heat-config/heat-config-puppet/cefd8eb6-7a44 -45f1-aeb1-aa9d24c411d9.pp:107 on node overcloud-compute-0.localdomain\u001b[0m
[stack@instack overcloud_images]$ ./list_nodes_status +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+ | d6306966-c550-424d-9e92-b381759460d6 | overcloud-cephstorage-0 | ACTIVE | - | Running | ctlplane=192.0.2.12 | | 9810aa3e-ceee-4381-b837-93df60b4df5f | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.6 | | 0a7f5551-d2f9-4de9-96da-4be86ce7ff40 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.19 | | 3610d2fa-1d1a-4f70-877f-986c0f3f6cbc | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.18 | | dbde31d4-d6a5-4a9b-9987-cb1e8b390114 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.14 | +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+ ** 192.0.2.12 up and running ** 192.0.2.6 up and running ** 192.0.2.19 up and running ** 192.0.2.18 up and running ** 192.0.2.14 up and running
[stack@instack overcloud_images]$ heat resource-list --nested-depth 5 overcloud | grep FAILED WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead | ComputeNodesPostDeployment | ea1602d6-27a0-4142-bd1c-852224c9914b | OS::TripleO::ComputePostDeployment | CREATE_FAILED | 2016-11-30T04:10:13Z | overcloud | | ControllerNodesPostDeployment | 4a287ba9-39a7-453b-b302-f4be7a613e7e | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2016-11-30T04:10:13Z | overcloud | | ComputePuppetDeployment | 959aae89-cd0d-4285-8a4d-6c3159438cb8 | OS::Heat::StructuredDeployments | CREATE_FAILED | 2016-11-30T04:16:59Z | overcloud-ComputeNodesPostDeployment-b4l36sv2cgwz | | ControllerLoadBalancerDeployment_Step1 | 9a550a39-963d-412f-8258-cfecfbf11536 | OS::Heat::StructuredDeployments | CREATE_FAILED | 2016-11-30T04:17:00Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat | | 0 | 6458f311-4106-46bd-84f0-10b7f2e07b4b | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-11-30T04:17:27Z | overcloud-ComputeNodesPostDeployment-b4l36sv2cgwz-ComputePuppetDeployment-nxzpey7ta5av | | 0 | b7ed4746-e264-49aa-814e-1f24a012ca5e | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-11-30T04:17:40Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat-ControllerLoadBalancerDeployment_Step1-4o6vosgln7v6 | | 1 | 6d29e39a-192f-4653-bdc4-fefe7c1d0666 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-11-30T04:17:40Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat-ControllerLoadBalancerDeployment_Step1-4o6vosgln7v6 | | 2 | 82c7d522-74e6-44ad-ae89-1d36c72e30d8 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-11-30T04:17:40Z | overcloud-ControllerNodesPostDeployment-bcosmjmwpiat-ControllerLoadBalancerDeployment_Step1-4o6vosgln7v6 |
readding the DFG:DF based on the fact that this is an OSP9 deployment failing (albeit with OSP10 undercloud). According to dyasny, this was a passing job last week so the concern is this is a regression. lbezdick from lifecycle is currently trying to triage on the environment with dyasny. apetrich will continue to look when he comes in tomorrow morning.
Thanks Dan! Do we need a documentation fix or was the issue in our automation?
(In reply to Jaromir Coufal from comment #9) > Thanks Dan! Do we need a documentation fix or was the issue in our > automation? I don't think so, though it would be great to be able to see the versions in glance so a user can be certain which image he has uploaded