Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1487743

Summary: [OSP12][Openstack-containers] Bare-metal deployment fails on ERROR Skipping group os-apply-config with no hook script None (ControllerDeployment_Step1.2)
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: openstack-containersAssignee: Dan Prince <dprince>
Status: CLOSED ERRATA QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact: Andrew Burden <aburden>
Priority: high    
Version: 12.0 (Pike)CC: dprince, jcoufal, lruzicka, m.andre, rhallise, shardy
Target Milestone: betaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 19:15:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Omri Hochman 2017-09-01 19:08:04 UTC
[OSP12][Openstack-containers] Bare-metal deployment fails on ERROR Skipping group os-apply-config with no hook script None (ControllerDeployment_Step1.2) 

Environment (osp12 puddle  2017-08-31.3 ):
--------------------------------------------
openstack-tripleo-heat-templates-7.0.0-0.20170821194254.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170712001959.0333c73.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170819032135.23884d3.el7ost.noarch
puppet-4.8.2-2.el7ost.noarch
instack-7.0.1-0.20170726180523.9961055.el7ost.noarch
instack-undercloud-7.2.1-0.20170821194210.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170821132121.22d7142.el7ost.noarch
puppet-heat-11.3.0-0.20170805100105.70232ae.el7ost.noarch
openstack-heat-common-9.0.0-0.20170821132121.22d7142.el7ost.noarch
openstack-heat-api-9.0.0-0.20170821132121.22d7142.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170821132121.22d7142.el7ost.noarch
python-heatclient-1.11.0-0.20170814165725.804ab3e.el7ost.noarch
python-heat-agent-1.4.1-0.20170821130002.ff31eb2.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170821194254.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
docker-client-1.12.6-48.git0fdc778.el7.x86_64
docker-1.12.6-48.git0fdc778.el7.x86_64



Description:
-------------
Attempted to deploy on Bare-Metal : 3 controller ,1 compute , 3 ceph 
deployment failed during ControllerDeployment_Step1.2


failure view:
--------------
18:19:29 2017-09-01 18:11:38Z [0]: CREATE_COMPLETE  state changed
18:19:29 2017-09-01 18:11:38Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step1]: CREATE_COMPLETE  Stack CREATE completed successfully
18:19:29 2017-09-01 18:11:38Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step1]: CREATE_COMPLETE  state changed
18:19:29 2017-09-01 18:19:18Z [2]: SIGNAL_IN_PROGRESS  Signal: deployment f20452fd-c9f0-4487-9348-68a9aad0b6b2 failed (2)
18:19:29 2017-09-01 18:19:18Z [2]: CREATE_FAILED  Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
18:19:29 2017-09-01 18:19:18Z [1]: CREATE_FAILED  CREATE aborted
18:19:29 2017-09-01 18:19:18Z [0]: CREATE_FAILED  CREATE aborted
18:19:29 2017-09-01 18:19:18Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_FAILED  Resource CREATE failed: Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
18:19:29 2017-09-01 18:19:18Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_FAILED  Error: resources.ControllerDeployment_Step1.resources[2]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
18:19:29 2017-09-01 18:19:18Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step1.resources[2]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
18:21:08 2017-09-01 18:19:19Z [overcloud.AllNodesDHeat Stack create failed.
18:21:08 Heat Stack create failed.
18:21:08 eploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step1.resources[2]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
18:21:08 2017-09-01 18:19:19Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step1.resources[2]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
18:21:08 
18:21:08  Stack overcloud CREATE_FAILED 
18:21:08 
18:21:08 overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.1:
18:21:08   resource_type: OS::Heat::StructuredDeployment
18:21:08   physical_resource_id: f6c0f649-4b6f-4214-ab53-b3f1c393cd84
18:21:08   status: CREATE_FAILED
18:21:08   status_reason: |
18:21:08     CREATE aborted
18:21:08   deploy_stdout: |
18:21:08 None
18:21:08   deploy_stderr: |
18:21:08 None
18:21:08 overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
18:21:08   resource_type: OS::Heat::StructuredDeployment
18:21:08   physical_resource_id: e55ec5d1-b117-473e-97c7-c26dee9a91a8
18:21:08   status: CREATE_FAILED
18:21:08   status_reason: |
18:21:08     CREATE aborted
18:21:08   deploy_stdout: |
18:21:08 None
18:21:08   deploy_stderr: |
18:21:08 None
18:21:08 overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.2:
18:21:08   resource_type: OS::Heat::StructuredDeployment
18:21:08   physical_resource_id: f20452fd-c9f0-4487-9348-68a9aad0b6b2
18:21:08   status: CREATE_FAILED
18:21:08   status_reason: |
18:21:08     Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
18:21:08   deploy_stdout: |
18:21:08     ...
18:21:08         ], 
18:21:08         "changed": false, 
18:21:08         "failed": true, 
18:21:08         "failed_when_result": true
18:21:08     }
18:21:08     	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/4e937e71-b48b-4da7-a574-a3bd5d110100_playbook.retry
18:21:08     
18:21:08     PLAY RECAP *********************************************************************
18:21:08     localhost                  : ok=8    changed=1    unreachable=0    failed=1   
18:21:08     
18:21:08     (truncated, view all with --long)
18:21:08   deploy_stderr: |
18:21:08 
18:21:08 Build step 'Virtualenv Builder' marked build as failure
18:21:09 Build step 'Groovy Postbuild' marked build as failure
18:21:09 Build step 'Groovy Postbuild' marked build as failure
18:21:09 [BFA] Scanning build for known causes...
18:21:10 .[BFA] Found failure cause(s):
18:21:10 [BFA] no-such-file-or-directory
18:21:10 [BFA] command not found from category shell
18:21:10 [BFA] Done. 1s
18:21:10 Started calculate disk usage of build
18:21:10 Finished Calculation of disk usage of build in 0 seconds
18:21:10 Started calculate disk usage of workspace
18:21:10 Finished Calculation of disk usage of workspace in 0 seconds
18:21:11 Finished: FAILURE



(undercloud) [stack@undercloud74 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+---------------+----------------------+--------------+----------------------------------+
| id                                   | stack_name | stack_status  | creation_time        | updated_time | project                          |
+--------------------------------------+------------+---------------+----------------------+--------------+----------------------------------+
| 91de1934-fb60-4729-8548-5e3a16c46709 | overcloud  | CREATE_FAILED | 2017-09-01T17:36:35Z | None         | eb397d38fa7a4ec9b08034b9ec2257a7 |


(undercloud) [stack@undercloud74 ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 05eb18a9-7161-4351-9087-1a2a35bbfa14 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.9  |
| a490bfd3-9d7f-40a8-b5e0-3b00776936e6 | overcloud-cephstorage-1 | ACTIVE | -          | Running     | ctlplane=192.168.24.16 |
| c78fc89e-5405-44a1-9efe-b0ae84f5f8a9 | overcloud-cephstorage-2 | ACTIVE | -          | Running     | ctlplane=192.168.24.15 |
| eb349fdd-b6fa-48b5-a946-caddd83ef115 | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.168.24.10 |
| ce393757-8b2b-4177-9e0f-716e3a263a90 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.7  |
| 728f5a6a-013e-4069-8279-491cc7270e6f | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.24.8  |
| 6ceec514-5ed3-45ed-8fc6-a20313916855 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.168.24.17 |


(undercloud) [stack@undercloud74 ~]$ heat resource-list overcloud -n 5 | grep -v COMPLETE


WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------------------------------+---------------------------------------
| AllNodesDeploySteps                       | e9a73d6d-c40d-4419-a315-a38ecd55c046                                                                                                                                                 | OS::TripleO::PostDeploySteps                                                                                                   | CREATE_FAILED   | 2017-09-01T17:36:40Z | overcloud                                                                                                                                                |
| ControllerDeployment_Step1                | 3b718619-8461-4e0f-8137-6953343bad14                                                                                                                                                 | OS::TripleO::DeploymentSteps                                                                                                   | CREATE_FAILED   | 2017-09-01T18:06:41Z | overcloud-AllNodesDeploySteps-gizorgzve5mr                                                                                                               |
| 0                                         | e55ec5d1-b117-473e-97c7-c26dee9a91a8                                                                                                                                                 | OS::Heat::StructuredDeployment                                                                                                 | CREATE_FAILED   | 2017-09-01T18:08:57Z | overcloud-AllNodesDeploySteps-gizorgzve5mr-ControllerDeployment_Step1-5dpfvwzc7yi7                                                                       |
| 1                                         | f6c0f649-4b6f-4214-ab53-b3f1c393cd84                                                                                                                                                 | OS::Heat::StructuredDeployment                                                                                                 | CREATE_FAILED   | 2017-09-01T18:08:57Z | overcloud-AllNodesDeploySteps-gizorgzve5mr-ControllerDeployment_Step1-5dpfvwzc7yi7                                                                       |
| 2                                         | f20452fd-c9f0-4487-9348-68a9aad0b6b2                                                                                                                                                 | OS::Heat::StructuredDeployment                                                                                                 | CREATE_FAILED   | 2017-09-01T18:08:57Z | overcloud-AllNodesDeploySteps-gizorgzve5mr-ControllerDeployment_Step1-5dpfvwzc7yi7                                                                       |
+-------------------------------------------+-----------------------------------------------------------------------




from controller journalctl -u os-collect-config | grep ERROR:
----------------------------------------------------------------
[root@overcloud-controller-0 ~]# journalctl -u os-collect-config | grep ERROR
Sep 01 18:00:14 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:00:14,549] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:00:37 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:00:37,519] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:01:06 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:01:06,739] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:01:35 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:01:35,058] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:04:56 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:04:56,362] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:05:42 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:05:42,479] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:06:31 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:06:31,003] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:07:44 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 14:07:44,654] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
Sep 01 18:19:45 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 18:19:44,968] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-ansible/1c8a3c5f-434e-4f34-b5fe-8ba7b55d4f7f_playbook.yaml. [2]
Sep 01 18:19:47 overcloud-controller-0 os-collect-config[3448]: [2017-09-01 18:19:47,238] (heat-config) [ERROR] Skipping group os-apply-config with no hook script None
[root@overcloud-controller-0 ~]#


/var/lib/heat-config/heat-config-ansible/1c8a3c5f-434e-4f34-b5fe-8ba7b55d4f7f_playbook.yaml
--------------------------------------------------------

- hosttep puppet configuration of the baremetal hoster step puppet configuration of the baremetal host: localhost
  connection: local
  tasks:
      # Note the indentation here is required as it's joined
    # to create a playbook in deploy-steps.j2

    #####################################################
    # Per step puppet configuration of the baremetal host
    #####################################################
    - name: Write the config_step hieradata
      copy: content="{{dict(step=step|int)|to_json}}" dest=/etc/puppet/hieradata/config_step.json force=true
    - name: Run puppet host configuration for step {{step}}
      command: >-
        puppet apply
        --modulepath=/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules
        --logdest syslog --logdest console --color=false
        /var/lib/tripleo-config/puppet_step_config.pp

Comment 1 Omri Hochman 2017-09-01 19:10:30 UTC
full output --> http://pastebin.test.redhat.com/513161



(undercloud) [stack@undercloud74 ~]$ heat deployment-show f20452fd-c9f0-4487-9348-68a9aad0b6b2
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED",
  "server_id": "6ceec514-5ed3-45ed-8fc6-a20313916855",
  "config_id": "4e937e71-b48b-4da7-a574-a3bd5d110100",
  "output_values": {
    "deploy_stdout": "\nPLAY [localhost] ***************************************************************\n\nTASK [Gathering Facts] *********************************************************\nok: [localhost]\n\nTASK [Write the config_step hieradata] *****************************************\nchanged: [localhost]\n\nTASK [Run puppet host configuration for step 1] ********************************\nok: [localhost]\n\nTASK [debug] *******************************************************************\nok: [localhost] => {\n    \"(outputs.stderr|default('')).split('\\n')|union(outputs.stdout_lines|default([]))\": [\n        \"exception: connect failed\", \n        \"Warning: Undefined variable 'deploy_config_name'; \", \n        \"   (file & line not available)\", \n        \"Warning: Scope(Class[Tripleo::Profile::Base::Docker]): The $insecure_registry_address parameter is deprecated. Use $insecure_registries instead.\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 54]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n        \"   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 55]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 56]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 66]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 68]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 89]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n        \"Warning: ModuleLoader: module 'ssh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n        \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ssh/manifests/server.pp\\\", 12]:[\\\"/var/lib/tripleo-config/puppet_step_config.pp\\\", 30]\", \n        \"Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n        \"Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend\", \n        \"Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.\", \n        \"Notice: Compiled catalog for overcloud-controller-2.localdomain in environment production in 4.44 seconds\", \n        \"Notice: /Stage[main]/Tripleo::Profile::Base::Docker/Augeas[docker-sysconfig-options]/returns: executed successfully\", \n        \"Notice: /Stage[main]/Tripleo::Profile::Base::Docker/Augeas[docker-sysconfig-registry]/returns: executed successfully\", \n        \"Notice: /Stage[main]/Tripleo::Profile::Base::Docker/Augeas[docker-sysconfig-storage]/returns: executed successfully\", \n        \"Notice: /Stage[main]/Tripleo::Profile::Base::Docker/Service[docker]/ensure: ensure changed 'stopped' to 'running'\", \n        \"Notice: /Stage[main]/Tripleo::Profile::Base::Database::Mysql::Client/Augeas[tripleo-mysql-client-conf]/returns: executed successfully\", \n        \"Notice: /Stage[main]/Tripleo::Profile::Base::Time::Ntp/Service[chronyd]/ensure: ensure changed 'running' to 'stopped'\", \n        \"Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}913c85f0fde85f83c2d6c030ecf259e9' to '{md5}d8be90f7e269dce8ad2a1e27e621ae09'\", \n        \"Notice: /Stage[main]/Ntp::Service/Service[ntp]/ensure: ensure changed 'stopped' to 'running'\", \n        \"Notice: /Stage[main]/Pacemaker::Service/Service[pcsd]/ensure: ensure changed 'stopped' to 'running'\", \n        \"Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'\", \n        \"Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'\", \n        \"Notice: /Stage[main]/Pacemaker::Corosync/User[hacluster]/password: changed password\", \n        \"Notice: /Stage[main]/Pacemaker::Corosync/User[hacluster]/groups: groups changed '' to ['haclient']\", \n        \"Notice: /Stage[main]/Pacemaker::Corosync/Exec[reauthenticate-across-all-nodes]: Triggered 'refresh' from 2 events\", \n        \"Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker]/ensure: created\", \n        \"Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker-authkey]/ensure: defined content as '{md5}50eae4a58fff0b9c34cf693ac28bef76'\", \n        \"Notice: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: executed successfully\", \n        \"Notice: /Stage[main]/Timezone/File[/etc/localtime]/target: target changed '../usr/share/zoneinfo/America/New_York' to '/usr/share/zoneinfo/UTC'\", \n        \"Notice: /Stage[main]/Firewall::Linux::Redhat/Exec[/usr/bin/systemctl daemon-reload]/returns: executed successfully\", \n        \"Notice: /Stage[main]/Firewall::Linux::Redhat/Service[iptables]/ensure: ensure changed 'stopped' to 'running'\", \n        \"Notice: /Stage[main]/Firewall::Linux::Redhat/Service[ip6tables]/ensure: ensure        
..
..
/var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf\", \n        \"INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck\", \n        \"INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf\", \n        \"INFO:__main__:Writing out command to execute\", \n        \"170901 18:17:58 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295\", \n        \"170901 18:18:00 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295\", \n        \"stdout: 4e6cbe8acb92212ee8b9b30ff045bf49dd432cc75022371175efd5e6c92fb43a\"\n    ], \n    \"changed\": false, \n    \"failed\": true, \n    \"failed_when_result\": true\n}\n\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/4e937e71-b48b-4da7-a574-a3bd5d110100_playbook.retry\n\nPLAY RECAP *********************************************************************\nlocalhost                  : ok=8    changed=1    unreachable=0    failed=1   \n\n",
    "deploy_stderr": "",
    "deploy_status_code": 2
  },
  "creation_time": "2017-09-01T18:09:06Z",
  "updated_time": "2017-09-01T18:19:18Z",
  "input_values": {
    "update_identifier": "1504287325",
    "docker_puppet_debug": "",
    "role_name": "Controller",
    "step": 1,
    "bootstrap_server_id": "ce393757-8b2b-4177-9e0f-716e3a263a90"
  },
  "action": "CREATE",
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 2",
  "id": "f20452fd-c9f0-4487-9348-68a9aad0b6b2"
}

Comment 4 Omri Hochman 2017-09-01 20:10:32 UTC
Your sosreport has been generated and saved in:
  /var/tmp/sosreport-undercloud74.local-20170901154621.tar.xz

The checksum is: 39be5c606e9b48f183ea24a4a75bc265

Please send this file to your support representative.

Copying the results to the publicly available URL
The reports should be available here: http://rhos-release.virt.bos.redhat.com/log/bz1487743

Comment 5 Omri Hochman 2017-09-01 22:07:23 UTC
Unable to reproduce after removing the W/A used for https://bugzilla.redhat.com/show_bug.cgi?id=1483643

Comment 6 Omri Hochman 2017-09-05 16:13:53 UTC
(In reply to Omri Hochman from comment #5)
> Unable to reproduce after removing the W/A used for
> https://bugzilla.redhat.com/show_bug.cgi?id=1483643

Since the W/A might still relevant. Reopen Bz till we will figure it out

Comment 7 Dan Prince 2017-09-05 20:01:06 UTC
Does the baremetal image being deployed (overcloud-full.qcow2) container the python-heat-agent-apply-config package. This packages is what should be supplying the os-apply-config hook that is being requested.

Comment 8 Omri Hochman 2017-09-05 20:49:41 UTC
(In reply to Dan Prince from comment #7)
> Does the baremetal image being deployed (overcloud-full.qcow2) container the
> python-heat-agent-apply-config package. This packages is what should be
> supplying the os-apply-config hook that is being requested.

[root@overcloud-controller-0 ~]# rpm -qa | grep python-heat-agent-apply-config
python-heat-agent-apply-config-1.4.1-0.20170821130002.ff31eb2.el7ost.noarch

Comment 9 Dan Prince 2017-09-05 21:20:37 UTC
Ah, okay. I think I misread the error. The 'os-apply-config' (extra os- on the front) is the important part.

So I'm wondering if perhaps there is an old style custom network configuration template around. Specifically the old network templates used to use 'os-apply-config' to convert the network metadata into the correct on disk format for os-net-config to do its magic. We've since switched to just apply-config (which is now installed by the python-heat-agent-apply-config package I wanted you to check above).

Could you give details about how the network templates for this environment are generated?

Comment 10 Omri Hochman 2017-09-06 14:17:12 UTC
(In reply to Dan Prince from comment #9)
> Ah, okay. I think I misread the error. The 'os-apply-config' (extra os- on
> the front) is the important part.
> 
> So I'm wondering if perhaps there is an old style custom network
> configuration template around. Specifically the old network templates used
> to use 'os-apply-config' to convert the network metadata into the correct on
> disk format for os-net-config to do its magic. We've since switched to just
> apply-config (which is now installed by the python-heat-agent-apply-config
> package I wanted you to check above).
> 
> Could you give details about how the network templates for this environment
> are generated?

we saw that there was a new change in nic-config template,  I would try to make the adjustments to new format and re-test

Comment 11 Dan Prince 2017-09-06 20:56:20 UTC
A couple of updates:

-we are still deploying the os-net-config element. So the old style 'os-apply-config' elements should still work fine. I don't think they are related to the deployment hanging here.

-the error messages is coming from the heat-agents projects heat-config/os-refresh-config/configure.d/55-heat-config. Essentially, the 'os-apply-config' group is sort of a flag value in that it does nothing as far has heat-config is concern. It instead relies on the fact that os-apply-config is executed once per each os-refresh-config run... so the templated file in the old os-net-config element will go away.

-While the warning is concerning, Like I mentioned before I think using newer styled network templates would make it go away.

Comment 12 Dan Prince 2017-09-06 21:39:02 UTC
Okay, I pushed an upstream fix for the error message here:

https://review.openstack.org/501456 whitelist os-apply-config group in 55-heat-config

In the meantime I don't think this is an OSP12 blocker.

Comment 13 Omri Hochman 2017-10-26 19:35:24 UTC
Verified . Unable to reproduce : OpenStack-12.0-RHEL-7 Puddle: 2017-10-24.3

Comment 18 errata-xmlrpc 2017-12-13 19:15:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3457