Bug 1822411
| Summary: | Deployment stuck in loop on AllNodesDeploySteps in heat-db shows all requirements complete | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Stan Toporek <stoporek> |
| Component: | openstack-heat | Assignee: | David Peacock <dpeacock> |
| Status: | CLOSED DUPLICATE | QA Contact: | David Rosenfeld <drosenfe> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 13.0 (Queens) | CC: | apetrich, aschultz, averi, dpeacock, mburns, ramishra, sbaker, shardy |
| Target Milestone: | --- | Keywords: | Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-15 17:59:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
More delopyment script detail:
[stoporek@supportshell overcloud-templates-cee]$ cat generate-environment.sh
upstream_templates=(
environments/ceph-ansible/ceph-ansible.yaml
environments/ceph-ansible/ceph-rgw.yaml
environments/cinder-backup.yaml
environments/services/barbican.yaml
environments/barbican-backend-simple-crypto.yaml
environments/services-docker/octavia.yaml
environments/network-isolation.yaml
extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml
extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml
)
custom_templates=(
net-multiple-nics.yaml
ceph-config.yaml
saml.yaml
fencing.yaml
novnc-ssl-fix.yaml
)
ENVIRONMENT_ARGS=""
UPSTREAM_TEMPLATE_PATH=${UPSTREAM_TEMPLATE_PATH:-"/usr/share/openstack-tripleo-heat-templates"}
CUSTOM_TEMPLATE_PATH=${CUSTOM_TEMPLATE_PATH:-"/home/stack/overcloud-templates/custom"}
for i in "${upstream_templates[@]}" ; do
ENVIRONMENT_ARGS="${ENVIRONMENT_ARGS} -e ${UPSTREAM_TEMPLATE_PATH}/${i}"
done
for i in "${custom_templates[@]}" ; do
ENVIRONMENT_ARGS="${ENVIRONMENT_ARGS} -e ${CUSTOM_TEMPLATE_PATH}/${i}"
done
PLAN_ARGS="-p /home/stack/overcloud-templates/custom/plan-environment.yaml"
NETWORK_ARGS="-n /home/stack/overcloud-templates/custom/network_data.yaml"
ROLE_ARGS="-r /home/stack/overcloud-templates/custom/roles_data.yaml"
Looks like memory upgrade pushed out the loop. The loop may be caused by updates in SELinux rules they applied recently. Waiting for customer to verify. Please close bug. Cu had issues with SELinux rules. For anyone interested, follow-ups at https://bugzilla.redhat.com/show_bug.cgi?id=1823829. cheers, *** This bug has been marked as a duplicate of bug 1823829 *** |
Description of problem: Deployment of OSP 13 stuck in loop on AllNodesDeploySteps and -ControllerDeployment_Step1. Heat-db shows all requirements are complete but heat.engine.scheduler does not report step complete so scheduler is stuck in a loop and never finish deployment. Version-Release number of selected component (if applicable): OSP 13 How reproducible: Every time deployment is attempted Steps to Reproduce: 1.Deploy using script run-deployment.sh 2.Deployment begins normally 3.Deployment stuck in loop and never completes Actual results: loop 2020-04-08 07:44:33.951 4386 DEBUG heat.engine.scheduler [req-4f9b216d-2756-4b80-bac9-b0717b64246a - - - - -] Task update from StructuredDeploymentGroup "ControllerDeployment_Step1" [d31e7fed-138d-4844-9aae-65e510cc708d] Stack "overcloud-AllNodesDeploySteps-yf7a3ogzexmx" [e8b323ae-8535-416c-a7d9-6e86a83223c5] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2020-04-08 07:44:34.094 4388 DEBUG heat.engine.scheduler [req-264fc718-41c7-43f8-b6b2-671cfacac289 - admin - default default] Task create from StructuredDeployment "2" Stack "overcloud-AllNodesDeploySteps-yf7a3ogzexmx-ControllerDeployment_Step1-d45zbxsqeeon" [d31e7fed-138d-4844-9aae-65e510cc708d] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 Expected results: Deployment completes Additional info: director Heat-db info:The following are hung: MariaDB [heat]> select * from resource where status != 'COMPLETE' \G ; *************************** 1. row *************************** id: 13786 uuid: 8d925d97-e0db-4c17-8ee6-cdeb2c4475be nova_instance: e8b323ae-8535-416c-a7d9-6e86a83223c5 name: AllNodesDeploySteps created_at: 2019-02-14 19:56:17 updated_at: 2020-04-08 21:04:56 action: UPDATE status: IN_PROGRESS status_reason: state changed stack_id: f87c43e6-3351-4e78-a979-f062183f7c23 rsrc_metadata: {} properties_data: null engine_id: 6f8e48a1-09b3-487e-b0a6-bf782f7eb589 atomic_key: 110 needed_by: [] requires: [13825, 13826, 13829, 13830, 13838, 13848, 13853, 13856, 13858, 13859, 13862, 13868, 13787, 13789, 13790, 13793, 13794, 13795, 13796, 13798, 13799, 13801, 13802, 13803, 13804, 13807, 13813, 13814, 13816, 13819] replaces: NULL replaced_by: NULL current_template_id: 89240 properties_data_encrypted: NULL root_stack_id: f87c43e6-3351-4e78-a979-f062183f7c23 rsrc_prop_data_id: 139697 attr_data_id: NULL *************************** 2. row *************************** id: 17760 uuid: e50211cd-b90d-41f3-a60e-ad640b2da6d2 nova_instance: d31e7fed-138d-4844-9aae-65e510cc708d name: ControllerDeployment_Step1 created_at: 2020-04-07 21:03:23 updated_at: 2020-04-08 21:05:23 action: UPDATE status: IN_PROGRESS status_reason: state changed stack_id: e8b323ae-8535-416c-a7d9-6e86a83223c5 rsrc_metadata: {} properties_data: null engine_id: 3156b73c-edcd-4263-804c-99250fd2d351 atomic_key: 16 needed_by: [15343, 15988, 15989, 15990, 15991, 15992] requires: [15361, 15363, 15350, 15366, 15372, 15374, 15349, 17750, 15352, 15354, 15357, 15359] replaces: 15982 replaced_by: NULL current_template_id: 89899 properties_data_encrypted: NULL root_stack_id: f87c43e6-3351-4e78-a979-f062183f7c23 rsrc_prop_data_id: 140713 attr_data_id: NULL *************************** 3. row *************************** id: 17816 uuid: 49e6a9e8-72c1-4cd4-a2f0-a69865ded268 nova_instance: 85408deb-6610-46e9-9403-eb4a65fbc75e name: 1 created_at: 2020-04-08 21:05:28 updated_at: NULL action: CREATE status: IN_PROGRESS status_reason: state changed stack_id: d31e7fed-138d-4844-9aae-65e510cc708d rsrc_metadata: {} properties_data: null engine_id: 3156b73c-edcd-4263-804c-99250fd2d351 atomic_key: 1 needed_by: [] requires: [] replaces: 17814 replaced_by: NULL current_template_id: 90591 properties_data_encrypted: NULL root_stack_id: f87c43e6-3351-4e78-a979-f062183f7c23 rsrc_prop_data_id: 146692 attr_data_id: NULL *************************** 4. row *************************** id: 17817 uuid: c872180d-e8f9-4fbe-b705-3ab511c47714 nova_instance: adcfa3b9-4359-434d-b5be-2ceaa3518d1b name: 2 created_at: 2020-04-08 21:05:29 updated_at: NULL action: CREATE status: IN_PROGRESS status_reason: state changed stack_id: d31e7fed-138d-4844-9aae-65e510cc708d rsrc_metadata: {} properties_data: null engine_id: 980ec1f1-fc53-45d9-a898-e6d3877bab0f atomic_key: 1 needed_by: [] requires: [] replaces: 17812 replaced_by: NULL current_template_id: 90591 properties_data_encrypted: NULL root_stack_id: f87c43e6-3351-4e78-a979-f062183f7c23 rsrc_prop_data_id: 146693 attr_data_id: NULL 4 rows in set (0.02 sec) Check of resources required shows all complete: MariaDB [heat]> select * from resource where id in ('13825', '13826', '13829', '13830', '13838', '13848', '13853', '13856', '13858', '13859', '13862', '13868', '13787', '13789', '13790', '13793', '13794', '13795', '13796', '13798', '13799', '13801', '13802', '13803', '13804', '13807', '13813', '13814', '13816', '13819') and status != 'COMPLETE' ; Empty set (0.00 sec) MariaDB [heat]> select * from resource where id in ('15361', '15363', '15350', '15366', '15372', '15374', '15349', '17750', '15352', '15354', '15357', '15359') and status != 'COMPLETE'; Empty set (0.00 sec) Check of conflicting states shows nothing: MariaDB [heat]> select id, name from resource where action='INIT' and status='COMPLETE'; Empty set (0.00 sec) [stoporek@supportshell overcloud-templates-cee]$ cat run-deployment.sh #!/bin/bash UPSTREAM_TEMPLATE_PATH=/usr/share/openstack-tripleo-heat-templates . ./generate-environment.sh set -x time \ openstack overcloud deploy --templates \ ${PLAN_ARGS} \ ${NETWORK_ARGS} \ ${ROLE_ARGS} \ ${ENVIRONMENT_ARGS} \ "${@}"