Bug 1308576 - Heat engine crashes during osp-d update from 7.1 and from 7.0 to 7.3
Heat engine crashes during osp-d update from 7.1 and from 7.0 to 7.3
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
7.0 (Kilo)
x86_64 Linux
high Severity high
: ga
: 8.0 (Liberty)
Assigned To: Zane Bitter
Amit Ugol
: ZStream
Depends On: 1307125
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-15 09:29 EST by Angus Thomas
Modified: 2016-04-26 10:21 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1307125
Environment:
Last Closed: 2016-02-15 17:39:53 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 1 Zane Bitter 2016-02-15 17:37:25 EST
2016-02-12 13:18:46.487 13936 CRITICAL heat-engine [req-33db6d70-974c-4210-8311-b85ba49fb031 - -] DBConnectionError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT watch
_rule.created_at AS watch_rule_created_at, watch_rule.updated_at AS watch_rule_updated_at, watch_rule.id AS watch_rule_id, watch_rule.name AS watch_rule_name, watch_rule.rule AS watch_rule_rule, watch_rule.state
 AS watch_rule_state, watch_rule.last_evaluated AS watch_rule_last_evaluated, watch_rule.stack_id AS watch_rule_stack_id \nFROM watch_rule \nWHERE watch_rule.stack_id = %s'] [parameters: ('fda3526e-7dbe-4f23-bb3
d-a1b8b3bd0fa8',)]
2016-02-12 13:18:46.487 13936 TRACE heat-engine Traceback (most recent call last):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/bin/heat-engine", line 83, in <module>
2016-02-12 13:18:46.487 13936 TRACE heat-engine     srv.create_periodic_tasks()
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return f(*args, **kwargs)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 332, in create_periodic_tasks
2016-02-12 13:18:46.487 13936 TRACE heat-engine     self.stack_watch.start_watch_task(s.id, admin_context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 57, in start_watch_task
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(stack_id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(child.id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(child.id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     if stack_has_a_watchrule(child.id):
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 36, in stack_has_a_watchrule
2016-02-12 13:18:46.487 13936 TRACE heat-engine     wrs = watch_rule_object.WatchRule.get_all_by_stack(cnxt, sid)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/objects/watch_rule.py", line 77, in get_all_by_stack
2016-02-12 13:18:46.487 13936 TRACE heat-engine     stack_id)]
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 248, in watch_rule_get_all_by_stack
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return IMPL.watch_rule_get_all_by_stack(context, stack_id)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 718, in watch_rule_get_all_by_stack
2016-02-12 13:18:46.487 13936 TRACE heat-engine     context, models.WatchRule).filter_by(stack_id=stack_id).all()
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2399, in all
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return list(self)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return self._execute_and_instances(context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
2016-02-12 13:18:46.487 13936 TRACE heat-engine     result = conn.execute(querycontext.statement, self._params)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return meth(self, multiparams, params)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-02-12 13:18:46.487 13936 TRACE heat-engine     return connection._execute_clauseelement(self, multiparams, params)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-02-12 13:18:46.487 13936 TRACE heat-engine     compiled_sql, distilled_params
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-02-12 13:18:46.487 13936 TRACE heat-engine     context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-02-12 13:18:46.487 13936 TRACE heat-engine     util.raise_from_cause(newraise, exc_info)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2016-02-12 13:18:46.487 13936 TRACE heat-engine     reraise(type(exception), exception, tb=exc_tb)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-02-12 13:18:46.487 13936 TRACE heat-engine     context)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-02-12 13:18:46.487 13936 TRACE heat-engine     cursor.execute(statement, parameters)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-02-12 13:18:46.487 13936 TRACE heat-engine     self.errorhandler(self, exc, value)
2016-02-12 13:18:46.487 13936 TRACE heat-engine   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-02-12 13:18:46.487 13936 TRACE heat-engine     raise errorclass, errorvalue
2016-02-12 13:18:46.487 13936 TRACE heat-engine DBConnectionError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT watch_rule.created_at AS watch_rule_created_at, watch_rule.updated_at AS watch_rule_updated_at, watch_rule.id AS watch_rule_id, watch_rule.name AS watch_rule_name, watch_rule.rule AS watch_rule_rule, watch_rule.state AS watch_rule_state, watch_rule.last_evaluated AS watch_rule_last_evaluated, watch_rule.stack_id AS watch_rule_stack_id \nFROM watch_rule \nWHERE watch_rule.stack_id = %s'] [parameters: ('fda3526e-7dbe-4f23-bb3d-a1b8b3bd0fa8',)]
2016-02-12 13:18:46.487 13936 TRACE heat-engine 
2016-02-12 13:18:46.611 13995 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting
2016-02-12 13:18:46.611 13993 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting
2016-02-12 13:18:46.612 13995 DEBUG heat.engine.service [-] Attempting to stop engine service... _stop_rpc_server /usr/lib/python2.7/site-packages/heat/engine/service.py:373
2016-02-12 13:18:46.612 13993 DEBUG heat.engine.service [-] Attempting to stop engine service... _stop_rpc_server /usr/lib/python2.7/site-packages/heat/engine/service.py:373
2016-02-12 13:18:46.613 13992 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting
Comment 2 Zane Bitter 2016-02-15 17:39:53 EST
This issue does not affect RHOS 8 because the code in question is disabled by default from Liberty onward: https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=bbf29137282769f6c96123fe92babe41ea64e625

Note You need to log in before you can comment on or make changes to this bug.