This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Pasting some of the http://pastebin.test.redhat.com/485452  
errors here (before pastebin link is out-date): 


undercloud) [stack@undercloud-0 ~]$ openstack stack failures list --long overcloud
	overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: a43ed7aa-9b2f-4a8a-a01d-d2170ac5274d
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
    Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 7.35 seconds
    Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_controller3]/ensure: created
    Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/api_paste_config]/ensure: created
    Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/storage_availability_zone]/ensure: created
    Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/default_availability_zone]/ensure: created
    Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/host]/ensure: created
    Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/enable_v3_api]/ensure: created
    Notice: /Stage[main]/Cinder::Glance/Cinder_config[DEFAULT/glance_api_servers]/ensure: created
    Notice: /Stage[main]/Cinder::Glance/Cinder_config[DEFAULT/glance_api_version]/ensure: created
    Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/osapi_volume_listen]/ensure: created
    Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/osapi_volume_workers]/ensure: created
    Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/nova_catalog_info]/ensure: created

...
...
...
...

  Notice: /Stage[main]/Horizon/Exec[refresh_horizon_django_cache]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Horizon/Exec[refresh_horizon_django_compress]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Horizon::Wsgi::Apache/Apache::Vhost[horizon_vhost]/File[/var/www/]/seluser: seluser changed 'unconfined_u' to 'system_u'
    Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/File[/var/www/cgi-bin/cinder]/ensure: created
    Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/File[cinder_wsgi]/ensure: defined content as '{md5}2edefdbaaa05fe52ec75b860834ad713'
    Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[/var/www/cgi-bin/heat]/ensure: created
    Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[heat_api_cloudwatch_wsgi]/ensure: defined content as '{md5}d30e0d9c58374044881079f8dd8132e8'
    Notice: /Stage[main]/Horizon::Wsgi::Apache/Apache::Vhost[horizon_vhost]/Concat[10-horizon_vhost.conf]/File[/etc/httpd/conf.d/10-horizon_vhost.conf]/ensure: defined content as '{md5}4e6723b6cd639932bbf665b59811e5ff'
    Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/Apache::Vhost[cinder_wsgi]/Concat[10-cinder_wsgi.conf]/File[/etc/httpd/conf.d/10-cinder_wsgi.conf]/ensure: defined content as '{md5}febd0df6ecf38d5da85ac2b80fa67e9f'
    Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/Apache::Vhost[heat_api_cloudwatch_wsgi]/Concat[10-heat_api_cloudwatch_wsgi.conf]/File[/etc/httpd/conf.d/10-heat_api_cloudwatch_wsgi.conf]/ensure: defined content as '{md5}62e48fe213e7e4ccbfe6853aeaed691c'
    Notice: /Stage[main]/Apache::Service/Service[httpd]/ensure: ensure changed 'stopped' to 'running'
    Notice: Applied catalog in 660.52 seconds
  deploy_stderr: |
    exception: connect failed
    Warning: Facter: Could not retrieve fact='rabbitmq_nodename', resolution='<anonymous>': undefined method `[]' for nil:NilClass
    Warning: Facter: Could not retrieve fact='rabbitmq_nodename', resolution='<anonymous>': undefined method `[]' for nil:NilClass
    Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/cinder/manifests/db.pp", 64]:["/etc/puppet/modules/cinder/manifests/init.pp", 385]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: Scope(Class[Cinder]): host is deprecated, has no effect and will be removed in a future release, use backend_host instead
    Warning: Scope(Class[Cinder]): cinder::rabbit_host, cinder::rabbit_hosts, cinder::rabbit_password, cinder::rabbit_port, cinder::rabbit_userid and cinder::rabbit_virtual_host are deprecated. Please use cinder::default_transport_url instead.
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/cinder/manifests/config.pp", 38]:["/etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp", 121]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/cinder/manifests/api.pp", 214]:["/etc/puppet/modules/tripleo/manifests/profile/base/cinder/api.pp", 78]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'apache' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/apache/manifests/init.pp", 338]:["/etc/puppet/modules/apache/manifests/mod/ssl.pp", 21]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/apache/manifests/mod/dir.pp", 8]:["/etc/puppet/modules/apache/manifests/default_mods.pp", 131]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/apache/manifests/mod/negotiation.pp", 9]:["/etc/puppet/modules/apache/manifests/default_mods.pp", 133]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/apache/manifests/mod/ssl.pp", 58]:["/etc/puppet/modules/tripleo/manifests/profile/base/cinder/api.pp", 79]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Bool instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/apache/manifests/mod/ssl.pp", 60]:["/etc/puppet/modules/tripleo/manifests/profile/base/cinder/api.pp", 79]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: Scope(Class[Heat]): heat::rabbit_host, heat::rabbit_hosts, heat::rabbit_password, heat::rabbit_port, heat::rabbit_userid and heat::rabbit_virtual_host are deprecated. Please use heat::default_transport_url instead.
    Warning: Unknown variable: 'haproxy_stats_bind_certificate'. at /etc/puppet/modules/tripleo/manifests/haproxy.pp:783:6
    Warning: ModuleLoader: module 'haproxy' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 136]:["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/haproxy.pp", 44]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 76]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'ssh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Scope(Class[Ceilometer]): ceilometer::rabbit_host, ceilometer::rabbit_hosts, ceilometer::rabbit_password, ceilometer::rabbit_port, ceilometer::rabbit_userid and ceilometer::rabbit_virtual_host are deprecated. Please use ceilometer::default_transport_url instead.
    Warning: ModuleLoader: module 'horizon' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Undefined variable ''; 
       (file & line not available)
    Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/pacemaker/haproxy_with_vip.pp", 62]:
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    Error: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Failed to call refresh: Command exceeded timeout
    Error: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Command exceeded timeout
    Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: Command exceeded timeout
    Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Command exceeded timeout
(undercloud) [stack@undercloud-0 ~]$

Reproduced:
    Error: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Failed to call refresh: Command exceeded timeout
    Error: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Command exceeded timeout
    Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: Command exceeded timeout
    Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Command exceeded timeout

Digging on the setup shows that there's a problem connecting to mariadb on a controller using VIP:

[root@overcloud-controller-0 ~]# mysql -u cinder -h 192.168.24.8
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0



Once the following lines were removed in haproxy.conf under "listen mysql" - it became possible to connect to mysql using VIP

  option tcpka
  option httpchk
  stick on dst
  stick-table type ip size 1000

The change requires restarting haproxy.

So running the following before the OC deployment on the undercloud worked for me:

sudo sed -i  's/tripleo::haproxy::mysql_clustercheck: true/tripleo::haproxy::mysql_clustercheck: false/' /usr/share/openstack-tripleo-heat-templates/puppet/services/pacemaker/haproxy.yaml

so just as a note, the "clustercheck" thing is required when haproxy is interfacing to Galera, and it is not supposed be there if it is interfacing to a non-Galera MySQL node.   the behavior here seems to indicate that the overcloud node is not running Galera, because "clustercheck" will always fail against a non-Galera MySQL service.

some internal discussion here seems to indicate that is likely the case here (non Galera running, and this is a non-HA deployment), a ps listing on the overcloud in question would confirm that for us.

just curious, since I use oooq to deploy here, what's in /home/stack/nodes_data.yaml please?  can you share that file?

nevermind, dciabrin is going to walk me through the full steps you were using locally

I got this issue for non-HA deployment.

Without using w/a 
sudo sed -i  's/tripleo::haproxy::mysql_clustercheck: true/tripleo::haproxy::mysql_clustercheck: false/' /usr/share/openstack-tripleo-heat-templates/puppet/services/pacemaker/haproxy.yaml


Redeployment with w/a was successfully

Created attachment 1301071 [details]
stderr to failed deployment

So, to be more specific about why this type of deploy is failing:

 . the default downstream is to deploy with pacemaker enabled, being 1-node or multi-node controllers.
 . the setting used for the deployment is:
    . per docker.yaml: deploy services in by following the non-HA variant (not managed by pacemaker). Do not deploy cinder or haproxy in container, because this is how this file defaults to.
    . per overcloud-resource-registry-puppet.j2.yaml: enable pacemaker on baremetal, as well as haproxy and cinder on baremetal.

Now enabling pacemaker will set some flags in hiera to tell the haproxy profile to generate config like "mariadb is targeted the HA way, and clustercheck monitors the nodes".

However, the mariadb service is configured to be deployed like a non-HA container, i.e. without clustercheck at all. And the deploy fails.

The procedure used to deploy this overcloud is *very* far from the expected way of deploying it. One should deploys it like specific in the docker-ha.yaml, even for 1-node controller.