Bug 1308576

Summary: Heat engine crashes during osp-d update from 7.1 and from 7.0 to 7.3
Product: Red Hat OpenStack Reporter: Angus Thomas <athomas>
Component: openstack-heatAssignee: Zane Bitter <zbitter>
Status: CLOSED CURRENTRELEASE QA Contact: Amit Ugol <augol>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: augol, mburns, ohochman, rhel-osp-director-maint, sasha, sbaker, shardy, tdunnon, yeylon, zbitter
Target Milestone: gaKeywords: ZStream
Target Release: 8.0 (Liberty)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1307125 Environment:
Last Closed: 2016-02-15 17:39:53 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On: 1307125    
Bug Blocks:    

Comment 1 Zane Bitter 2016-02-15 17:37:25 EST
2016-02-12 13:18:46.487 13936 CRITICAL heat-engine [req-33db6d70-974c-4210-8311-b85ba49fb031 - -] DBConnectionError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT watch
_rule.created_at AS watch_rule_created_at, watch_rule.updated_at AS watch_rule_updated_at, watch_rule.id AS watch_rule_id, watch_rule.name AS watch_rule_name, watch_rule.rule AS watch_rule_rule, watch_rule.state
 AS watch_rule_state, watch_rule.last_evaluated AS watch_rule_last_evaluated, watch_rule.stack_id AS watch_rule_stack_id \nFROM watch_rule \nWHERE watch_rule.stack_id = %s'] [parameters: ('fda3526e-7dbe-4f23-bb3
d-a1b8b3bd0fa8',)]
2016-02-12 13:18:46.487 13936 TRACE heat-engine Traceback (most recent call last):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/bin/heat-engine", line 83, in <module>
2016-02-12 13:18:46.487 13936 TRACE heat-engine     srv.create_periodic_tasks()
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return f(*args, **kwargs)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 332, in create_periodic_tasks
2016-02-12 13:18:46.487 13936 TRACE heat-engine     self.stack_watch.start_watch_task(s.id, admin_context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 57, in start_watch_task
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(stack_id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(child.id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(child.id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(child.id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 36, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     wrs = watch_rule_object.WatchRule.get_all_by_stack(cnxt, sid)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/objects/watch_rule.py", line 77, in get_all_by_stack
2016-02-12 13:18:46.487 13936 TRACE heat-engine     stack_id)]
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 248, in watch_rule_get_all_by_stack
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return IMPL.watch_rule_get_all_by_stack(context, stack_id)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 718, in watch_rule_get_all_by_stack
2016-02-12 13:18:46.487 13936 TRACE heat-engine     context, models.WatchRule).filter_by(stack_id=stack_id).all()
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2399, in all
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return list(self)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return self._execute_and_instances(context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
2016-02-12 13:18:46.487 13936 TRACE heat-engine     result = conn.execute(querycontext.statement, self._params)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return meth(self, multiparams, params)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return connection._execute_clauseelement(self, multiparams, params)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-02-12 13:18:46.487 13936 TRACE heat-engine     compiled_sql, distilled_params
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-02-12 13:18:46.487 13936 TRACE heat-engine     context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-02-12 13:18:46.487 13936 TRACE heat-engine     util.raise_from_cause(newraise, exc_info)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2016-02-12 13:18:46.487 13936 TRACE heat-engine     reraise(type(exception), exception, tb=exc_tb)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-02-12 13:18:46.487 13936 TRACE heat-engine     context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-02-12 13:18:46.487 13936 TRACE heat-engine     cursor.execute(statement, parameters)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-02-12 13:18:46.487 13936 TRACE heat-engine     self.errorhandler(self, exc, value)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-02-12 13:18:46.487 13936 TRACE heat-engine     raise errorclass, errorvalue
2016-02-12 13:18:46.487 13936 TRACE heat-engine DBConnectionError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT watch_rule.created_at AS watch_rule_created_at, watch_rule.updated_at AS watch_rule_updated_at, watch_rule.id AS watch_rule_id, watch_rule.name AS watch_rule_name, watch_rule.rule AS watch_rule_rule, watch_rule.state AS watch_rule_state, watch_rule.last_evaluated AS watch_rule_last_evaluated, watch_rule.stack_id AS watch_rule_stack_id \nFROM watch_rule \nWHERE watch_rule.stack_id = %s'] [parameters: ('fda3526e-7dbe-4f23-bb3d-a1b8b3bd0fa8',)]
2016-02-12 13:18:46.487 13936 TRACE heat-engine 
2016-02-12 13:18:46.611 13995 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting
2016-02-12 13:18:46.611 13993 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting
2016-02-12 13:18:46.612 13995 DEBUG heat.engine.service [-] Attempting to stop engine service... _stop_rpc_server /usr/lib/python2.7/site-packages/heat/engine/service.py:373
2016-02-12 13:18:46.612 13993 DEBUG heat.engine.service [-] Attempting to stop engine service... _stop_rpc_server /usr/lib/python2.7/site-packages/heat/engine/service.py:373
2016-02-12 13:18:46.613 13992 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting
Comment 2 Zane Bitter 2016-02-15 17:39:53 EST
This issue does not affect RHOS 8 because the code in question is disabled by default from Liberty onward: https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=bbf29137282769f6c96123fe92babe41ea64e625