Bug 1603138

Summary: Controller replacement with Octavia- After controller replacement amphora is in ERROR state
Product: Red Hat OpenStack Reporter: Alexander Stafeyev <astafeye>
Component: openstack-octaviaAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED DUPLICATE QA Contact: Alexander Stafeyev <astafeye>
Severity: urgent Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: ihrachys, lpeer, majopela, nyechiel
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-29 20:01:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 4 Carlos Goncalves 2018-08-29 20:01:34 UTC 
Closing this one as duplicate of rhbz #1623146.

2018-07-19 09:10:32.300 23 ERROR futurist.periodics [-] Failed to call periodic 'octavia.cmd.health_manager.periodic_health_check' (it runs every 3.00 seconds): DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics Traceback (most recent call last):
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 290, in run
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     work()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 64, in __call__
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return self.callback(*self.args, **self.kwargs)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 178, in decorator
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return f(*args, **kwargs)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/octavia/cmd/health_manager.py", line 54, in periodic_health_check
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     hm.health_check()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/octavia/controller/healthmanager/health_manager.py", line 85, in health_check
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     lock_session.rollback()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     self.force_reraise()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     six.reraise(self.type_, self.value, self.tb)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/octavia/controller/healthmanager/health_manager.py", line 75, in health_check
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     amp = self.amp_health_repo.get_stale_amphora(lock_session)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/octavia/db/repositories.py", line 1095, in get_stale_amphora
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     self.model_class.last_update < expired_time).first()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     ret = list(self[0:1])
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return list(res)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return self._execute_and_instances(context)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     close_with_result=True)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     **kw
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     conn = self.session.connection(**kw)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     execution_options=execution_options)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     engine, execution_options)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     conn = bind.contextual_connect()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     self._wrap_pool_connect(self.pool.connect, None),
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     e, dialect, self)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     util.raise_from_cause(newraise, exc_info)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return fn()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return _ConnectionFairy._checkout(self)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     fairy = _ConnectionRecord.checkout(pool)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     rec.checkin()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     compat.reraise(exc_type, exc_value, exc_tb)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     dbapi_connection = rec.get_connection()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     self.__connect()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     connection = pool._invoke_creator(self)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return dialect.connect(*cargs, **cparams)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return self.dbapi.connect(*cargs, **cparams)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     return Connection(*args, **kwargs)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     self.connect()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     self._get_server_information()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     packet = self._read_packet()
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     packet_header = self._read_bytes(4)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes
2018-07-19 09:10:32.300 23 ERROR futurist.periodics     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-07-19 09:10:32.300 23 ERROR futurist.periodics DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-07-19 09:10:32.300 23 ERROR futurist.periodics 
2018-07-19 09:10:35.314 23 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: f94fca60-dac0-47af-b811-af2d30d558e7
2018-07-19 09:10:35.325 23 INFO octavia.controller.healthmanager.health_manager [-] Waiting for 1 failovers to finish
2018-07-19 09:10:35.432 23 WARNING octavia.controller.worker.controller_worker [-] Failing over amphora with no spares pool may cause delays in failover times while a new amphora instance boots.

*** This bug has been marked as a duplicate of bug 1623146 ***