Bug 1927169

Summary:	DBAPIError exception wrapped from (pymysql.err.InternalError) (1054, u"Unknown column 'pool.tls_certificate_id' in 'field list'")
Product:	Red Hat OpenStack	Reporter:	tmicheli
Component:	openstack-octavia	Assignee:	Carlos Goncalves <cgoncalves>
Status:	CLOSED NOTABUG	QA Contact:	Bruna Bonguardo <bbonguar>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	13.0 (Queens)	CC:	cgoncalves, gthiemon, ihrachys, lpeer, majopela, scohen
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-10 17:17:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1939511

Description tmicheli 2021-02-10 09:28:24 UTC

Description of problem:
During patching of the controller nodes a loadbalancer object was created. The loadbalancer is in pending state and it is not possible to delete the object.

Version-Release number of selected component (if applicable):
- octavia-2.1.2-5 

How reproducible:


Steps to Reproduce:
1. Create a loadbalancer during patching the controller nodes
2.
3.

Actual results:
----
(overcloud) [stack@hostname[I-D-Director] ~]$
(overcloud) [stack@hostname[I-D-Director] ~]$ openstack loadbalancer list
+--------------------------------------+-------------------+----------------------------------+--------------+---------------------+----------+
| id                                   | name              | project_id                       | vip_address  | provisioning_status | provider |
+--------------------------------------+-------------------+----------------------------------+--------------+---------------------+----------+
| e16475c9-b402-427e-8f3d-1c5e3d99d7d1 | test-loadbalancer | fdf59f4425d24ed2b18113a630c879f0 | 192.168.0.19 | PENDING_UPDATE      | amphora  |
+--------------------------------------+-------------------+----------------------------------+--------------+---------------------+----------+
(overcloud) [stack@hostname[I-D-Director] ~]$

(overcloud) [stack@hostname[I-D-Director] ~]$ openstack loadbalancer pool list --loadbalancer e16475c9-b402-427e-8f3d-1c5e3d99d7d1
+--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+
| id                                   | name      | project_id                       | provisioning_status | protocol | lb_algorithm | admin_state_up |
+--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+
| 2c370453-3b08-4eea-8a38-7603c32abac7 | test-pool | fdf59f4425d24ed2b18113a630c879f0 | PENDING_UPDATE      | HTTP     | ROUND_ROBIN  | True           |
+--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+
(overcloud) [stack@hostname[I-D-Director] ~]$

(overcloud) [stack@hostname[I-D-Director] ~]$
(overcloud) [stack@hostname[I-D-Director] ~]$ openstack loadbalancer delete e16475c9-b402-427e-8f3d-1c5e3d99d7d1
Validation failure: Cannot delete Load Balancer e16475c9-b402-427e-8f3d-1c5e3d99d7d1 - it has children (HTTP 400) (Request-ID: req-0c51b339-fbb5-4145-9588-71656c1b5aab)
(overcloud) [stack@hostname[I-D-Director] ~]$

(overcloud) [stack@hostname[I-D-Director] ~]$
(overcloud) [stack@hostname[I-D-Director] ~]$ openstack loadbalancer member delete 2c370453-3b08-4eea-8a38-7603c32abac7 785545e5-dcfb-4904-a3f3-b7cef54edd79
Pool 2c370453-3b08-4eea-8a38-7603c32abac7 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-0dc898e6-52b6-409e-810d-8ec72fb02fe7)
(overcloud) [stack@hostname[I-D-Director] ~]$

(overcloud) [stack@hostname[I-D-Director] ~]$
(overcloud) [stack@hostname[I-D-Director] ~]$ openstack loadbalancer member set --disable 2c370453-3b08-4eea-8a38-7603c32abac7 785545e5-dcfb-4904-a3f3-b7cef54edd79
Pool 2c370453-3b08-4eea-8a38-7603c32abac7 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-95bc8a70-2e56-4886-ada3-129978f6cf5c)
----

Expected results:
- The loadbalancer is deleted

Additional info:
The following errors could be observed in the octavia-logs

----
#./var/log/containers/octavia/health-managers.log
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query ([Errno 104] Connection reset by peer)') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     connection.scalar(select([1]))
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     return self.execute(object, *multiparams, **params).scalar()
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     return meth(self, multiparams, params)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     return connection._execute_clauseelement(self, multiparams, params)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     compiled_sql, distilled_params
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     context)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     util.raise_from_cause(newraise, exc_info)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     context)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     result.read()
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     packet_header = self._read_bytes(4)
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1029, in _read_bytes
2021-01-29 16:07:40.836 27 ERROR oslo_db.sqlalchemy.engines     "Lost connection to MySQL server during query (%s)" % (e,))

# ./var/log/containers/octavia/housekeeping.log
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1054, u"Unknown column 'amphora.created_at' in 'field list'") [SQL: u'SELECT amphora.created_at AS amphora_created_at, amphora.updated_at AS amphora_updated_at, amphora.id AS amphora_id, amphora.load_balancer_id AS amphora_load_balancer_id, amphora.compute_id AS amphora_compute_id, amphora.lb_network_ip AS amphora_lb_network_ip, amphora.vrrp_ip AS amphora_vrrp_ip, amphora.ha_ip AS amphora_ha_ip, amphora.vrrp_port_id AS amphora_vrrp_port_id, amphora.ha_port_id AS amphora_ha_port_id, amphora.cert_expiration AS amphora_cert_expiration, amphora.cert_busy AS amphora_cert_busy, amphora.`role` AS amphora_role, amphora.status AS amphora_status, amphora.vrrp_interface AS amphora_vrrp_interface, amphora.vrrp_id AS amphora_vrrp_id, amphora.vrrp_priority AS amphora_vrrp_priority, amphora.cached_zone AS amphora_cached_zone, amphora.image_id AS amphora_image_id, amphora.compute_flavor AS amphora_compute_flavor \nFROM amphora \nWHERE amphora.cert_busy = false AND amphora.cert_expiration < %(cert_expiration_1)s \n LIMIT %(param_1)s FOR UPDATE'] [parameters: {u'cert_expiration_1': datetime.datetime(2021, 2, 12, 15, 5, 3, 772361), u'param_1': 1}] (Background on this error at: http://sqlalche.me/e/2j85): InternalError: (1054, u"Unknown column 'amphora.created_at' in 'field list'")
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     context)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
2021-01-29 16:05:03.780 8 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1054, u"Unknown column 'amphora.created_at' in 'field list'")

# ./var/log/containers/octavia/octavia.log
2021-02-03 14:33:18.239 33 WARNING octavia.api.v2.controllers.load_balancer [req-c1ac0559-425a-43bc-a0c8-31899e508fd0 - 372056b1705541ffb5ea934171a1e7db - default default] Cannot delete Load Balancer e16475c9-b402-427e-8f3d-1c5e3d99d7d1 - it has children


2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters [req-71bac907-191a-4663-b722-91e175796394 - fdf59f4425d24ed2b18113a630c879f0 - default default] DBAPIError exception wrapped from (pymysql.err.InternalError) (1054, u"Unknown 
column 'pool.tls_certificate_id' in 'field list'") [SQL: u'SELECT pool.created_at AS pool_created_at, pool.updated_at AS pool_updated_at, pool.id AS pool_id, pool.project_id AS pool_project_id, pool.name AS pool_name, pool.description AS p
ool_description, pool.protocol AS pool_protocol, pool.lb_algorithm AS pool_lb_algorithm, pool.provisioning_status AS pool_provisioning_status, pool.operating_status AS pool_operating_status, pool.enabled AS pool_enabled, pool.load_balancer
_id AS pool_load_balancer_id, pool.tls_certificate_id AS pool_tls_certificate_id, pool.ca_tls_certificate_id AS pool_ca_tls_certificate_id, pool.crl_container_id AS pool_crl_container_id, pool.tls_enabled AS pool_tls_enabled \nFROM pool \n
WHERE pool.project_id = %(project_id_1)s AND pool.provisioning_status != %(provisioning_status_1)s ORDER BY pool.created_at ASC, pool.id ASC \n LIMIT %(param_1)s'] [parameters: {u'project_id_1': u'fdf59f4425d24ed2b18113a630c879f0', u'param
_1': 1000, u'provisioning_status_1': 'DELETED'}] (Background on this error at: http://sqlalche.me/e/2j85): InternalError: (1054, u"Unknown column 'pool.tls_certificate_id' in 'field list'")
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     context)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1054, u"Unknown column 'pool.tls_certificate_id' in 'field list'")
2021-01-29 16:05:03.161 26 ERROR oslo_db.sqlalchemy.exc_filters 

# # ./var/log/containers/octavia/worker.log
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1054, u"Unknown column 'member.backup' in 'field list'") [SQL: u'SELECT member.created_at AS member_created_at, member.updated_at AS member_updated_at, member.id AS member_id, member.project_id AS member_project_id, member.name AS member_name, member.pool_id AS member_pool_id, member.subnet_id AS member_subnet_id, member.ip_address AS member_ip_address, member.protocol_port AS member_protocol_port, member.weight AS member_weight, member.backup AS member_backup, member.monitor_address AS member_monitor_address, member.monitor_port AS member_monitor_port, member.provisioning_status AS member_provisioning_status, member.operating_status AS member_operating_status, member.enabled AS member_enabled \nFROM member \nWHERE member.id = %(id_1)s \n LIMIT %(param_1)s'] [parameters: {u'id_1': u'785545e5-dcfb-4904-a3f3-b7cef54edd79', u'param_1': 1}] (Background on this error at: http://sqlalche.me/e/2j85): InternalError: (1054, u"Unknown column 'member.backup' in 'field list'")
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     context)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1054, u"Unknown column 'member.backup' in 'field list'")
2021-01-29 16:05:21.367 26 ERROR oslo_db.sqlalchemy.exc_filters

Comment 1 Carlos Goncalves 2021-02-10 09:38:17 UTC

Do I understand correctly that a new load balancer was created while performing an overcloud update? If so, it appears that you are updating from a release older than OSP 13 zstream 13 (8 October 2020 maintenance release) where some controller nodes have already updated and others have not. If that is the case, please refer to https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/networking_guide/sec-octavia#upgrade-load-balancing-service

Comment 5 Carlos Goncalves 2021-03-08 17:12:09 UTC

For reference, I opened RFE BZ #1936529 soliciting a mechanism to fence service API endpoints off in Director.