1129896 – HA: openstack-heat-engine fails to start.

Bug 1129896 - HA: openstack-heat-engine fails to start.

Summary: HA: openstack-heat-engine fails to start.

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-foreman-installer
Sub Component:
Version:	5.0 (RHEL 7)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	ga
Target Release:	Installer
Assignee:	John Eckersberg
QA Contact:	Leonid Natapov
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-08-13 21:42 UTC by Leonid Natapov
Modified:	2014-08-21 18:08 UTC (History)
CC List:	10 users (show)
Fixed In Version:	openstack-foreman-installer-2.0.21-1.el6ost
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2014-08-21 18:08:56 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
messages log from controller1 (4.93 MB, text/plain) 2014-08-15 15:58 UTC, Alexander Chuzhoy	no flags	Details
/var/log/heat/engine.log from controller1 (81.79 KB, text/x-log) 2014-08-15 15:59 UTC, Alexander Chuzhoy	no flags	Details
/var/log/heat/engine.log from controller2 (82.04 KB, text/x-log) 2014-08-15 16:09 UTC, Alexander Chuzhoy	no flags	Details
messages log from controller2 (10.63 MB, text/plain) 2014-08-15 16:10 UTC, Alexander Chuzhoy	no flags	Details
/var/log/heat/engine.log from controller3 (276.15 KB, text/x-log) 2014-08-15 16:27 UTC, Alexander Chuzhoy	no flags	Details
messages log from controller3 (10.20 MB, text/plain) 2014-08-15 16:28 UTC, Alexander Chuzhoy	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2014:1090	0	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory	2014-08-22 15:28:08 UTC

Description Leonid Natapov 2014-08-13 21:42:20 UTC

After successful HA+Neutron deployment heat-engine fails to start

heat engine log shows the following:
------------------------------------------------------------------------------
2014-08-13 21:27:24.603 8537 INFO heat.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.0.99:5672
2014-08-13 21:27:24.655 8537 INFO heat.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.0.99:5672
2014-08-13 21:27:24.678 8537 WARNING heat.openstack.common.db.sqlalchemy.session [req-efec3d95-c38a-480e-8d91-d5e6d1077dad None] This application has not enabled MySQL traditional mode, which means
 silent data corruption may occur. Please encourage the application developers to enable this mode.
2014-08-13 21:27:24.739 8537 ERROR heat.openstack.common.threadgroup [-] (ProgrammingError) (1146, "Table 'heat.stack' doesn't exist") 'SELECT stack.status_reason AS stack_status_reason, stack.crea
ted_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id A
S stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_ow
ner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_at \nFROM sta
ck \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL ORDER BY stack.created_at DESC, stack.id DESC' ()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup Traceback (most recent call last):
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/openstack/common/threadgroup.py", line 125, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     x.wait()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/openstack/common/threadgroup.py", line 47, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self.thread.wait()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self._exit_event.wait()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self.greenlet.switch()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/openstack/common/service.py", line 483, in run_service
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     service.start()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 297, in start
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     stacks = db_api.stack_get_all(admin_context, tenant_safe=False)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 125, in stack_get_all
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     marker, sort_dir, filters, tenant_safe)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 346, in stack_get_all
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     marker, sort_dir, filters).all()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return list(self)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self._execute_and_instances(context)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     result = conn.execute(querycontext.statement, self._params)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     params)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     compiled_sql, distilled_params
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     context)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     exc_info
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     reraise(type(exception), exception, tb=exc_tb)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     context)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     cursor.execute(statement, parameters)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     self.errorhandler(self, exc, value)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     raise errorclass, errorvalue
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup ProgrammingError: (ProgrammingError) (1146, "Table 'heat.stack' doesn't exist") 'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL ORDER BY stack.created_at DESC, stack.id DESC' ()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup 
------------------------------------------------------------------------------

trying to start it manually also fail.
 
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
openstack-foreman-installer-2.0.18-1.el6ost.noarch
rhel-osp-installer-0.1.9-1.el6ost.noarch
openstack-puppet-modules-2014.1-19.9.el6ost.noarch

Comment 2 Brad P. Crochet 2014-08-14 18:53:06 UTC

Please retest with latest puddle.

Comment 3 Alexander Chuzhoy 2014-08-14 19:12:51 UTC

Verified: FailedQA with rhel-osp-installer-0.1.10-2.el6ost.noarch

Failed actions:
    openstack-heat-engine_monitor_60000 on mac047d7b61765c.example.com 'not running' (7): call=208, status=complete, last-rc-change='Thu Aug 14 19:02:47 2014', queued=0ms, exec=0ms
    neutron-openvswitch-agent_start_0 on mac047d7b61765c.example.com 'not running' (7): call=196, status=complete, last-rc-change='Thu Aug 14 18:55:23 2014', queued=4ms, exec=2001ms

Comment 4 Alexander Chuzhoy 2014-08-14 20:14:23 UTC

Will verify this bug as the error is on  openstack-heat-engine_monitor and not  openstack-heat-engine

Comment 5 Alexander Chuzhoy 2014-08-14 20:15:00 UTC

Verified:  rhel-osp-installer-0.1.10-2.el6ost.noarch


this particular error didn't reproduce.

Comment 6 Alexander Chuzhoy 2014-08-15 15:53:47 UTC

reproduced - reopening

Comment 7 Alexander Chuzhoy 2014-08-15 15:58:20 UTC

Created attachment 927147 [details]
messages log from controller1

Comment 8 Alexander Chuzhoy 2014-08-15 15:59:48 UTC

Created attachment 927148 [details]
/var/log/heat/engine.log from controller1

Comment 9 Alexander Chuzhoy 2014-08-15 16:09:14 UTC

Created attachment 927176 [details]
/var/log/heat/engine.log from controller2

Comment 10 Alexander Chuzhoy 2014-08-15 16:10:27 UTC

Created attachment 927177 [details]
messages log from controller2

Comment 11 Jason Guiditta 2014-08-15 16:21:08 UTC

The fix was in openstack-foreman-installer, can you verify what version of tat you were using?

Comment 12 Alexander Chuzhoy 2014-08-15 16:27:41 UTC

Created attachment 927182 [details]
/var/log/heat/engine.log from controller3

Comment 13 Alexander Chuzhoy 2014-08-15 16:28:47 UTC

Created attachment 927183 [details]
messages log from controller3

Comment 14 Alexander Chuzhoy 2014-08-15 16:43:12 UTC

openstack-foreman-installer-2.0.20-1.el6ost.noarch

Comment 15 Jason Guiditta 2014-08-15 17:10:03 UTC

Hmm, this worked fine for me when I tested it.  I see a lot of 'missing table' errors in the attached logs, which are not present in my setup.  Can you verify that there is a heat db user, and that there is a heat keystone endpoint?

Comment 16 Alexander Chuzhoy 2014-08-15 18:07:31 UTC

Verified both exist - confirmed.

Comment 17 John Eckersberg 2014-08-15 18:52:38 UTC

https://github.com/redhat-openstack/astapor/pull/355

Comment 18 Jason Guiditta 2014-08-15 20:28:09 UTC

Merged

Comment 21 Leonid Natapov 2014-08-20 12:28:38 UTC

Can't reproduce openstack-foreman-installer-2.0.21-1.el6ost

Comment 22 errata-xmlrpc 2014-08-21 18:08:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html

Note You need to log in before you can comment on or make changes to this bug.