Bug 1129896

Summary: HA: openstack-heat-engine fails to start.
Product: Red Hat OpenStack Reporter: Leonid Natapov <lnatapov>
Component: openstack-foreman-installerAssignee: John Eckersberg <jeckersb>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: unspecified Docs Contact:
Priority: high    
Version: 5.0 (RHEL 7)CC: brad, jeckersb, jguiditt, lnatapov, mburns, morazi, rhos-maint, rybrown, sasha, yeylon
Target Milestone: ga   
Target Release: Installer   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-foreman-installer-2.0.21-1.el6ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-21 18:08:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
messages log from controller1
none
/var/log/heat/engine.log from controller1
none
/var/log/heat/engine.log from controller2
none
messages log from controller2
none
/var/log/heat/engine.log from controller3
none
messages log from controller3 none

Description Leonid Natapov 2014-08-13 21:42:20 UTC 
After successful HA+Neutron deployment heat-engine fails to start

heat engine log shows the following:
------------------------------------------------------------------------------
2014-08-13 21:27:24.603 8537 INFO heat.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.0.99:5672
2014-08-13 21:27:24.655 8537 INFO heat.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.0.99:5672
2014-08-13 21:27:24.678 8537 WARNING heat.openstack.common.db.sqlalchemy.session [req-efec3d95-c38a-480e-8d91-d5e6d1077dad None] This application has not enabled MySQL traditional mode, which means
 silent data corruption may occur. Please encourage the application developers to enable this mode.
2014-08-13 21:27:24.739 8537 ERROR heat.openstack.common.threadgroup [-] (ProgrammingError) (1146, "Table 'heat.stack' doesn't exist") 'SELECT stack.status_reason AS stack_status_reason, stack.crea
ted_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id A
S stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_ow
ner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_at \nFROM sta
ck \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL ORDER BY stack.created_at DESC, stack.id DESC' ()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup Traceback (most recent call last):
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/openstack/common/threadgroup.py", line 125, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     x.wait()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/openstack/common/threadgroup.py", line 47, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self.thread.wait()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self._exit_event.wait()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self.greenlet.switch()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/openstack/common/service.py", line 483, in run_service
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     service.start()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 297, in start
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     stacks = db_api.stack_get_all(admin_context, tenant_safe=False)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 125, in stack_get_all
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     marker, sort_dir, filters, tenant_safe)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 346, in stack_get_all
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     marker, sort_dir, filters).all()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return list(self)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     return self._execute_and_instances(context)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     result = conn.execute(querycontext.statement, self._params)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     params)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     compiled_sql, distilled_params
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     context)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     exc_info
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     reraise(type(exception), exception, tb=exc_tb)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     context)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     cursor.execute(statement, parameters)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     self.errorhandler(self, exc, value)
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup     raise errorclass, errorvalue
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup ProgrammingError: (ProgrammingError) (1146, "Table 'heat.stack' doesn't exist") 'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL ORDER BY stack.created_at DESC, stack.id DESC' ()
2014-08-13 21:27:24.739 8537 TRACE heat.openstack.common.threadgroup 
------------------------------------------------------------------------------

trying to start it manually also fail.
 
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
openstack-foreman-installer-2.0.18-1.el6ost.noarch
rhel-osp-installer-0.1.9-1.el6ost.noarch
openstack-puppet-modules-2014.1-19.9.el6ost.noarch
Comment 2 Brad P. Crochet 2014-08-14 18:53:06 UTC 
Please retest with latest puddle.
Comment 3 Alexander Chuzhoy 2014-08-14 19:12:51 UTC 
Verified: FailedQA with rhel-osp-installer-0.1.10-2.el6ost.noarch

Failed actions:
    openstack-heat-engine_monitor_60000 on mac047d7b61765c.example.com 'not running' (7): call=208, status=complete, last-rc-change='Thu Aug 14 19:02:47 2014', queued=0ms, exec=0ms
    neutron-openvswitch-agent_start_0 on mac047d7b61765c.example.com 'not running' (7): call=196, status=complete, last-rc-change='Thu Aug 14 18:55:23 2014', queued=4ms, exec=2001ms
Comment 4 Alexander Chuzhoy 2014-08-14 20:14:23 UTC 
Will verify this bug as the error is on  openstack-heat-engine_monitor and not  openstack-heat-engine
Comment 5 Alexander Chuzhoy 2014-08-14 20:15:00 UTC 
Verified:  rhel-osp-installer-0.1.10-2.el6ost.noarch


this particular error didn't reproduce.
Comment 6 Alexander Chuzhoy 2014-08-15 15:53:47 UTC 
reproduced - reopening
Comment 7 Alexander Chuzhoy 2014-08-15 15:58:20 UTC 
Created attachment 927147 [details]
messages log from controller1
Comment 8 Alexander Chuzhoy 2014-08-15 15:59:48 UTC 
Created attachment 927148 [details]
/var/log/heat/engine.log from controller1
Comment 9 Alexander Chuzhoy 2014-08-15 16:09:14 UTC 
Created attachment 927176 [details]
/var/log/heat/engine.log from controller2
Comment 10 Alexander Chuzhoy 2014-08-15 16:10:27 UTC 
Created attachment 927177 [details]
messages log from controller2
Comment 11 Jason Guiditta 2014-08-15 16:21:08 UTC 
The fix was in openstack-foreman-installer, can you verify what version of tat you were using?
Comment 12 Alexander Chuzhoy 2014-08-15 16:27:41 UTC 
Created attachment 927182 [details]
/var/log/heat/engine.log from controller3
Comment 13 Alexander Chuzhoy 2014-08-15 16:28:47 UTC 
Created attachment 927183 [details]
messages log from controller3
Comment 14 Alexander Chuzhoy 2014-08-15 16:43:12 UTC 
openstack-foreman-installer-2.0.20-1.el6ost.noarch
Comment 15 Jason Guiditta 2014-08-15 17:10:03 UTC 
Hmm, this worked fine for me when I tested it.  I see a lot of 'missing table' errors in the attached logs, which are not present in my setup.  Can you verify that there is a heat db user, and that there is a heat keystone endpoint?
Comment 16 Alexander Chuzhoy 2014-08-15 18:07:31 UTC 
Verified both exist - confirmed.
Comment 17 John Eckersberg 2014-08-15 18:52:38 UTC 
https://github.com/redhat-openstack/astapor/pull/355
Comment 18 Jason Guiditta 2014-08-15 20:28:09 UTC 
Merged
Comment 21 Leonid Natapov 2014-08-20 12:28:38 UTC 
Can't reproduce openstack-foreman-installer-2.0.21-1.el6ost
Comment 22 errata-xmlrpc 2014-08-21 18:08:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html