Bug 1308576
| Summary: | Heat engine crashes during osp-d update from 7.1 and from 7.0 to 7.3 | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Angus Thomas <athomas> |
| Component: | openstack-heat | Assignee: | Zane Bitter <zbitter> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Amit Ugol <augol> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 (Kilo) | CC: | augol, mburns, ohochman, rhel-osp-director-maint, sasha, sbaker, shardy, tdunnon, yeylon, zbitter |
| Target Milestone: | ga | Keywords: | ZStream |
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1307125 | Environment: | |
| Last Closed: | 2016-02-15 22:39:53 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1307125 | ||
| Bug Blocks: | |||
This issue does not affect RHOS 8 because the code in question is disabled by default from Liberty onward: https://review.openstack.org/gitweb?p=openstack/heat.git;a=commitdiff;h=bbf29137282769f6c96123fe92babe41ea64e625 |
2016-02-12 13:18:46.487 13936 CRITICAL heat-engine [req-33db6d70-974c-4210-8311-b85ba49fb031 - -] DBConnectionError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT watch _rule.created_at AS watch_rule_created_at, watch_rule.updated_at AS watch_rule_updated_at, watch_rule.id AS watch_rule_id, watch_rule.name AS watch_rule_name, watch_rule.rule AS watch_rule_rule, watch_rule.state AS watch_rule_state, watch_rule.last_evaluated AS watch_rule_last_evaluated, watch_rule.stack_id AS watch_rule_stack_id \nFROM watch_rule \nWHERE watch_rule.stack_id = %s'] [parameters: ('fda3526e-7dbe-4f23-bb3 d-a1b8b3bd0fa8',)] 2016-02-12 13:18:46.487 13936 TRACE heat-engine Traceback (most recent call last): 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/bin/heat-engine", line 83, in <module> 2016-02-12 13:18:46.487 13936 TRACE heat-engine srv.create_periodic_tasks() 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-02-12 13:18:46.487 13936 TRACE heat-engine return f(*args, **kwargs) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 332, in create_periodic_tasks 2016-02-12 13:18:46.487 13936 TRACE heat-engine self.stack_watch.start_watch_task(s.id, admin_context) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 57, in start_watch_task 2016-02-12 13:18:46.487 13936 TRACE heat-engine if stack_has_a_watchrule(stack_id): 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule 2016-02-12 13:18:46.487 13936 TRACE heat-engine if stack_has_a_watchrule(child.id): 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule 2016-02-12 13:18:46.487 13936 TRACE heat-engine if stack_has_a_watchrule(child.id): 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 52, in stack_has_a_watchrule 2016-02-12 13:18:46.487 13936 TRACE heat-engine if stack_has_a_watchrule(child.id): 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/engine/service_stack_watch.py", line 36, in stack_has_a_watchrule 2016-02-12 13:18:46.487 13936 TRACE heat-engine wrs = watch_rule_object.WatchRule.get_all_by_stack(cnxt, sid) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/objects/watch_rule.py", line 77, in get_all_by_stack 2016-02-12 13:18:46.487 13936 TRACE heat-engine stack_id)] 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 248, in watch_rule_get_all_by_stack 2016-02-12 13:18:46.487 13936 TRACE heat-engine return IMPL.watch_rule_get_all_by_stack(context, stack_id) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 718, in watch_rule_get_all_by_stack 2016-02-12 13:18:46.487 13936 TRACE heat-engine context, models.WatchRule).filter_by(stack_id=stack_id).all() 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2399, in all 2016-02-12 13:18:46.487 13936 TRACE heat-engine return list(self) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__ 2016-02-12 13:18:46.487 13936 TRACE heat-engine return self._execute_and_instances(context) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances 2016-02-12 13:18:46.487 13936 TRACE heat-engine result = conn.execute(querycontext.statement, self._params) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2016-02-12 13:18:46.487 13936 TRACE heat-engine return meth(self, multiparams, params) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2016-02-12 13:18:46.487 13936 TRACE heat-engine return connection._execute_clauseelement(self, multiparams, params) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2016-02-12 13:18:46.487 13936 TRACE heat-engine compiled_sql, distilled_params 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2016-02-12 13:18:46.487 13936 TRACE heat-engine context) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception 2016-02-12 13:18:46.487 13936 TRACE heat-engine util.raise_from_cause(newraise, exc_info) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-02-12 13:18:46.487 13936 TRACE heat-engine reraise(type(exception), exception, tb=exc_tb) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2016-02-12 13:18:46.487 13936 TRACE heat-engine context) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2016-02-12 13:18:46.487 13936 TRACE heat-engine cursor.execute(statement, parameters) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute 2016-02-12 13:18:46.487 13936 TRACE heat-engine self.errorhandler(self, exc, value) 2016-02-12 13:18:46.487 13936 TRACE heat-engine File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2016-02-12 13:18:46.487 13936 TRACE heat-engine raise errorclass, errorvalue 2016-02-12 13:18:46.487 13936 TRACE heat-engine DBConnectionError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT watch_rule.created_at AS watch_rule_created_at, watch_rule.updated_at AS watch_rule_updated_at, watch_rule.id AS watch_rule_id, watch_rule.name AS watch_rule_name, watch_rule.rule AS watch_rule_rule, watch_rule.state AS watch_rule_state, watch_rule.last_evaluated AS watch_rule_last_evaluated, watch_rule.stack_id AS watch_rule_stack_id \nFROM watch_rule \nWHERE watch_rule.stack_id = %s'] [parameters: ('fda3526e-7dbe-4f23-bb3d-a1b8b3bd0fa8',)] 2016-02-12 13:18:46.487 13936 TRACE heat-engine 2016-02-12 13:18:46.611 13995 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting 2016-02-12 13:18:46.611 13993 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting 2016-02-12 13:18:46.612 13995 DEBUG heat.engine.service [-] Attempting to stop engine service... _stop_rpc_server /usr/lib/python2.7/site-packages/heat/engine/service.py:373 2016-02-12 13:18:46.612 13993 DEBUG heat.engine.service [-] Attempting to stop engine service... _stop_rpc_server /usr/lib/python2.7/site-packages/heat/engine/service.py:373 2016-02-12 13:18:46.613 13992 INFO heat.openstack.common.service [-] Parent process has died unexpectedly, exiting