Bug 1388116

rhel-osp-director:   10 minor update: re-running the deployment command fails with "Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1:


Environment:
openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch
instack-undercloud-5.0.0-0.20161007201832.f044a47.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log


2. Update undercloud.
3. Apply patches: 
https://review.openstack.org/#/c/387531/ 
https://review.openstack.org/#/c/389114/ 

4.Remove openstack-dashboard rpm from all OC nodes.
5. Re-run the deployment command (to update templates)

Result:
xjpdjnht-CephStorageDeployment_Step3-lhxprpd2ntxf.1]: UPDATE_COMPLETE  state changed", "2016-10-23 22:04:36Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-CephStorageDeployment_Step3-lhxprpd2ntxf]: UPDATE_COMPLETE  Stack UPDATE completed successfully", "2016-10-23 22:04:37Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht.CephStorageDeployment_Step3]: UPDATE_COMPLETE  state changed", "2016-10-23 22:04:42Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ComputeDeployment_Step3-sthvk6feanfi.0]: SIGNAL_IN_PROGRESS  Signal: deployment b6fa9286-7f07-4723-a36f-d843f515e40b succeeded", "2016-10-23 22:04:42Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ComputeDeployment_Step3-sthvk6feanfi.0]: UPDATE_COMPLETE  state changed", "2016-10-23 22:04:43Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ComputeDeployment_Step3-sthvk6feanfi]: UPDATE_COMPLETE  Stack UPDATE completed successfully", "2016-10-23 22:04:44Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht.ComputeDeployment_Step3]: UPDATE_COMPLETE  state changed", "2016-10-23 22:04:51Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm.2]: SIGNAL_IN_PROGRESS  Signal: deployment de1486e3-c07c-4175-b15e-f5135082209d succeeded", "2016-10-23 22:04:52Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm.2]: UPDATE_COMPLETE  state changed", "2016-10-23 22:04:58Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm.1]: SIGNAL_IN_PROGRESS  Signal: deployment f1a2c31a-80df-4de4-bb12-32988cee86ef succeeded", "2016-10-23 22:04:58Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm.1]: UPDATE_COMPLETE  state changed", "2016-10-23 22:20:06Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm.0]: SIGNAL_IN_PROGRESS  Signal: deployment 88e12463-db1b-4219-a2ea-828ebd231302 failed (6)", "2016-10-23 22:20:07Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm.0]: UPDATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6", "2016-10-23 22:20:07Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht-ControllerDeployment_Step3-5dkshglkoiqm]: UPDATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6", "2016-10-23 22:20:09Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht.ControllerDeployment_Step3]: UPDATE_FAILED  resources.ControllerDeployment_Step3: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6", "2016-10-23 22:20:09Z [overcloud-AllNodesDeploySteps-c4glxjpdjnht]: UPDATE_FAILED  resources.ControllerDeployment_Step3: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6", "2016-10-23 22:20:10Z [AllNodesDeploySteps]: UPDATE_FAILED  resources.AllNodesDeploySteps: resources.ControllerDeployment_Step3: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6", "2016-10-23 22:20:10Z [overcloud]: UPDATE_FAILED  resources.AllNodesDeploySteps: resources.ControllerDeployment_Step3: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6", "", " Stack overcloud UPDATE_FAILED "], "warnings": []}



Debugging with heat:
Notice: /Firewall[999 drop all]: Dependency Keystone_service[keystone::identity] has failures: true                                                                                                                  
Notice: Finished catalog run in 904.07 seconds                                                                                                                                                                       
", "deploy_stderr": "exception: connect failed                                                                                                                                                                       
Warning: Scope(Class[Cinder::Api]): keystone_enabled is deprecated, use auth_strategy instead.                                                                                                                       
Warning: Scope(Class[Keystone]): Fernet token is recommended in Mitaka release. The default for token_provider will be changed to 'fernet' in O release.                                                             
Warning: Scope(Class[Keystone]): admin_password is required, please set admin_password to a value != admin_token. admin_token will be removed in a later release                                                     
Warning: Scope(Class[Keystone::Roles::Admin]): the main class is setting the admin password differently from this\                                                                                                   
 class when calling bootstrap. This will lead to the password\                                                                                                                                                       
 flip-flopping and cause authentication issues for the admin user.\                                                                                                                                                  
 Please ensure that keystone::roles::admin::password and\                                                                                                                                                            
 keystone::admin_password are set the same.                                                                                                                                                                          
Warning: Scope(Class[Heat]): keystone_user_domain_id is deprecated, use the name option instead.                                                                                                                     
Warning: Scope(Class[Heat]): keystone_project_domain_id is deprecated, use the name option instead.                                                                                                                  
Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated                                          
Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated                                          
Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated                                         
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Class[Ceilometer]): Both $metering_secret and $telemetry_secret defined, using $telemetry_secret
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for httpd.service failed because the control process exited with error code. See \"systemctl status httpd.service\" and \"journalctl -xe\" for details.
Error: /Stage[main]/Apache::Service/Service[httpd]: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for httpd.service failed because the control process exited with error code. See \"systemctl status httpd.service\" and \"journalctl -xe\" for details.
Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.0.2.18:35357/v3/projects (tried 47, for a total of 170 seconds)
Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures.
Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures.
Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
Error: Could not prefetch keystone_role provider 'openstack': Execution of '/usr/bin/openstack role list --quiet --format csv' returned 1: Unable to establish connection to http://192.0.2.18:35357/v3/roles (tried 47, for a total of 170 seconds)
Error: Not managing Keystone_role[admin] due to earlier Keystone API failures.
Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures.
Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://192.0.2.18:35357/v3/domains (tried 47, for a total of 170 seconds)
Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies
Error: Could not prefetch keystone_service provider 'openstack': Execution of '/usr/bin/openstack service list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.0.2.18:35357/v3/services (tried 47, for a total of 170 seconds)
Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures.
Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures.
Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/usr/bin/openstack endpoint list --quiet --format csv' returned 1: Unable to establish connection to http://192.0.2.18:35357/v3/endpoints (tried 47, for a total of 170 seconds)
Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]: Skipping because of failed dependencies
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies



Expected result:
The command should complete successfully.