rhel-osp-director: upgrade 7.3->8.0, that follows update 7.2->7.3, times out during "major-upgrade-pacemaker-converge.yaml" step

Environment:
openstack-tripleo-heat-templates-0.8.14-9.el7ost.noarch
instack-undercloud-2.2.7-4.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-9.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch

Steps to reproduce:
1. Deploy 7.2 HA with netiso (deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml)
2. populate the setup.
3. Update to 7.3
4. Add another instance to the setup.
5. Attempt to upgrade to 8.0
Result:

The upgrade fails after 4 hours run during the major-upgrade-pacemaker-converge step with:
2016-05-19 03:28:25 [overcloud-CephStorageCephDeployment-yxak3hlzkhck]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2016-05-19 03:28:27 [ControllerCephDeployment]: UPDATE_COMPLETE  state changed
2016-05-19 03:28:27 [CephStorageCephDeployment]: UPDATE_COMPLETE  state changed



ERROR: Authentication failed: Authentication required
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$
[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
+--------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-----------------------------------------------------------------------------------------------+
| resource_name                              | physical_resource_id                          | resource_type                                     | resource_status | updated_time        | stack_name                                                                                    |
+--------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-----------------------------------------------------------------------------------------------+
| Compute                                    | 520b5036-48b2-4ff7-9e22-2c0d40d9d6f4          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-05-19T03:24:56 | overcloud                                                                                     |
| 0                                          | bbea6423-3e7f-41d2-b8d9-425f9c9bd088          | OS::TripleO::Compute                              | UPDATE_FAILED   | 2016-05-19T03:25:05 | overcloud-Compute-jfxnp5zhnqu2                                                                |
| UpdateDeployment                           | 2c3e65db-b54b-4c07-aa86-c1d77690747d          | OS::Heat::SoftwareDeployment                      | CREATE_FAILED   | 2016-05-19T03:26:22 | overcloud-Compute-jfxnp5zhnqu2-0-gwvboa4eenug                                                 |
+--------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-----------------------------------------------------------------------------------------------+
[stack@instack ~]$ heat deployment-show 2c3e65db-b54b-4c07-aa86-c1d77690747d
{
  "status": "IN_PROGRESS",
  "server_id": "0c6981c8-0ee7-4df2-be98-50d258992f01",
  "config_id": "573938c4-542d-4f49-a83a-4c825f6ff097",
  "output_values": null,
  "creation_time": "2016-05-19T03:26:29",
  "input_values": {},
  "action": "CREATE",
  "status_reason": "Deploy data available",
  "id": "2c3e65db-b54b-4c07-aa86-c1d77690747d"
}
[stack@instack ~]$



The heat-engine.log (which is too big to attach) has these repeating block of erros:
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service [-] Unhandled exception
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service Traceback (most recent call last):
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 377, in _child_wait_for_exit_or_signal
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     launcher.wait()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 204, in wait
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self.services.wait()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 625, in wait
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     service.wait()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return f(*args, **kwargs)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 591, in wait
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self._done.wait()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return hubs.get_hub().switch()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return self.greenlet.switch()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self.wait(sleep_time)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     presult = self.do_poll(seconds)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return self.poll.poll(seconds)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 160, in _handle_signals
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     handler(signo, frame)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 356, in _sigterm
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self.launcher.stop()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 196, in stop
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self.services.stop()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 612, in stop
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     service.stop()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return f(*args, **kwargs)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 408, in stop
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     service_objects.Service.delete(ctxt, self.service_id)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/heat/objects/service.py", line 78, in delete
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     db_api.service_delete(context, service_id, soft_delete)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 378, in service_delete
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return IMPL.service_delete(context, service_id, soft_delete)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 1006, in service_delete
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     service = service_get(context, service_id)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 1016, in service_get
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     result = model_query(context, models.Service).get(service_id)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 819, in get
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return self._get_impl(ident, loading.load_on_ident)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 852, in _get_impl
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return fallback_fn(self, key)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     return q.one()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2473, in one
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     ret = list(self)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 86, in instances
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     util.raise_from_cause(err)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     reraise(type(exception), exception, tb=exc_tb)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 67, in instances
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     fetch = cursor.fetchall()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 968, in fetchall
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self.cursor, self.context)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     util.raise_from_cause(newraise, exc_info)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     reraise(type(exception), exception, tb=exc_tb)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 963, in fetchall
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self._soft_close()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 667, in _soft_close
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self.connection.close()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in close
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     conn.close()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 882, in close
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self._checkin()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 758, in _checkin
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     self._pool, None, self._echo, fairy=self)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 650, in _finalize_fairy
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     connection_record.checkin()
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 509, in checkin
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     pool.dispatch.checkin(connection, self)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     fn(*args, **kw)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 50, in _thread_yield
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     time.sleep(0)
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 31, in sleep
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service DBError: do not call blocking functions from the mainloop
2016-05-18 17:39:11.156 31851 ERROR oslo_service.service