Created attachment 1103059 [details] heat-config-puppet that failed Description of problem: version in controller: openstack-heat-api-2015.1.2-2.el7ost.noarch openstack-nova-api-2015.1.2-4.el7ost.noarch version in undercloud: openstack-heat-api-2015.1.2-4.el7ost.noarch openstack-nova-api-2015.1.2-7.el7ost.noarch Update failed from previous 7.2 puddle to latest puddle: I have had previous 7.2 puddle and tried to update the stacks with update command: openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-vip.yaml -e /home/stack/firstboot-env.yaml Version-Release number of selected component (if applicable): After the update failure I got a deployment failure: [stack@instack ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | a6598ecd-a322-4a47-a7e7-9744e2bb4d4f | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.9 | | f9096fe0-0858-44b6-9234-47b9ab6718ca | overcloud-compute-0 | ERROR | - | NOSTATE | | | 80242f3e-993d-4fa1-b11a-7af02177d805 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.10 | | c66cfb6f-20a1-47ae-9c70-0d0dd7812bf3 | overcloud-controller-0 | ERROR | - | NOSTATE | | | 10923f6b-496d-44f3-8c9d-1bc79d80ecb7 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.8 | | c559474c-8819-4dbb-8572-937e45907f0a | overcloud-controller-1 | ERROR | - | NOSTATE | | | 4ee94847-7e1c-43fd-aa17-95282cbed3a3 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.11 | | fdfb3257-aa61-4ffc-8d14-09bb5bb9b36f | overcloud-controller-2 | ERROR | - | NOSTATE | | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ nova show f9096fe0-0858-44b6-9234-47b9ab6718ca fault | {"message": "No valid host was found. Exceeded max scheduling attempts 3 for instance f9096fe0-0858-44b6-9234-47b9ab6718ca. Last exception: [u'Traceback (most recent call last):\ | | | ', u' File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2261, in _do", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 669, in build_instances | | | instances[0].uuid) | | | File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 172, in populate_retry | | | raise exception.NoValidHost(reason=msg) | | | ", "created": "2015-12-06T12:13:56Z"} I am able to login into the nodes successfully [stack@instack ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | 33f63b8c-af99-42bc-8777-4557453c883f | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.8 | | 66e7cad9-31db-445b-af61-755225b49df0 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.10 | | 323b7171-47b8-45fc-baff-3f84585006b6 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.9 | | 84d6266c-24a7-4d0e-943d-721a96c0d601 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.11 | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ openstack overcloud deploy --templates -e firstboot-env.yaml --control-scale 3 --compute-scale 1 --debug --log-file=overcloud_deploy.log --libvirt-type qemu --ntp-server clock.redhat.com From heat-api.log ck": {"parent": null, "disable_rollback": true, "description": "Nova API,Keystone,Heat Engine and API,Glance,Neutron,Dedicated MySQL server,Dedicated RabbitMQ Server,Group of Nova Computes\n", "links": [{"href": "http://192.0.2.1:8004/v1/ab55a17ce27341e29759d5e5c5d141d7/stacks/overcloud/10862cca-1348-4e11-93aa-6062015134a1", "rel": "self"}], "stack_status_reason": "Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1", "stack_name": "overcloud", "stack_user_project_id": "128fd809cad0423fb35dc7f7e6311c54", "stack_owner": "admin", "creation_time": "2015-12-06T13:55:46Z", "capabilities": [], "notification_topics": [], "updated_time": null, "timeout_mins": 240, "stack_status": "CREATE_FAILED", "parameters": {"NeutronPublicInterfaceRawDevice": "", "KeystoneSignin [stack@instack ~]$ heat stack-list +--------------------------------------+------------+---------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+---------------+----------------------+ | 71a6dd2c-4ebd-428f-9384-a75ed4275444 | overcloud | CREATE_FAILED | 2015-12-06T14:30:26Z | +--------------------------------------+------------+---------------+----------------------+ How reproducible: Steps to Reproduce: 1.deploy overcloud HA 3 controllers and 1 compute, must be successful deployment 2. update with the command above 3. delete the stack and re deploy Actual results: Exit code 1 failed to deploy Expected results: Although the update failed, ability to redeploy the stacks after deleting and re deploying Additional info: In controllers Error: Could not find data item step in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d0377.pp:36 on node overcloud-controller-1.localdomain Dec 6 09:42:11 localhost os-collect-config: [2015-12-06 09:42:11,403] (heat-config) [INFO] {"deploy_stdout": "", "deploy_stderr": "\u001b[1;31mWarning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_servic e or $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::no va::compute has not been evaluated\u001b[0m\n\u001b[1;31mError: nova_admin_password must be set. at /etc/puppet/modules/neutron/manifests/server/notifications.pp:80 on node overcloud-control ler-1.localdomain\u001b[0m\n\u001b[1;31mError: nova_admin_password must be set. at /etc/puppet/modules/neutron/manifests/server/notifications.pp:80 on node overcloud-controller-1.localdomain \u001b[0m\n", "deploy_status_code": 1} Dec 6 09:42:11 localhost os-collect-config: [2015-12-06 09:42:11,404] (heat-config) [DEBUG] [2015-12-06 09:42:04,989] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-c onfig/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d0377" FACTER_fqdn="overcloud-controller-1.localdomain" FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-4d7k3fypb c3m-ControllerPuppetConfig-wcechcxqwplg-ControllerPuppetConfigImpl-5vd2qdyleaa6" puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d03 77.pp Dec 6 09:42:11 localhost os-collect-config: [2015-12-06 09:42:11,400] (heat-config) [INFO] Return code 1 Attached /var/lib/heat-config/heat-config-puppet/fd355e82-666f-43e4-be3f-da7cdb6d0377.pp
Not a Heat bug. If an error occurs and Heat reports it, Heat is doing its job correctly.
i need some clarification on exactly what happened here. What are we trying to test? You said; > Steps to Reproduce: > 1.deploy overcloud HA 3 controllers and 1 compute, must be successful deployment > 2. update with the command above > 3. delete the stack and re deploy Did the update fail at step 2? Can we debug that failure first if so? Or are you trying to test deleting a stack and redeploying after an update?