Bug 1703019

Summary: Scale out failure cause with "Data too long for column 'instance_info'"
Product: Red Hat OpenStack Reporter: Khomesh Thakre <kthakre>
Component: openstack-ironicAssignee: RHOS Maint <rhos-maint>
Status: CLOSED NOTABUG QA Contact: mlammon
Severity: urgent Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: bfournie, mburns, msufiyan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-17 11:50:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Khomesh Thakre 2019-04-25 10:11:01 UTC
Description of problem:
While performing a scale-out operation it failed with below traceback

~~~
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters [req-a1a3c0bc-7017-45e5-bdc7-3eddf79789a6 - - - - -] DBAPIError exception wrapped from (pymysql.err.DataError) (1406, u"Data too long for column 'instance_info' at row 1") [SQL: u'UPDATE nodes SET updated_at=%(updated_at)s, instance_info=%(instance_info)s WHERE nodes.id = %(nodes_id)s'] [parameters: {'nodes_id': 199, 'updated_at': datetime.datetime(2019, 4, 24, 12, 23, 22, 723638),
.
.
.
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     context)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 841, in query
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1029, in _read_query_result
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1312, in read
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters DataError: (1406, u"Data too long for column 'instance_info' at row 1")
2019-04-24 17:53:22.735 1770 ERROR oslo_db.sqlalchemy.exc_filters
~~~

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform 10 Newton

Comment 3 Bob Fournier 2019-05-07 19:26:34 UTC
From the case it looks like this is no longer an issue.

Because of the database configuration its not possible to backport the fix for 
https://bugzilla.redhat.com/show_bug.cgi?id=1433088 to OSP-10.  The recommended fix is to update the database as described in https://bugzilla.redhat.com/show_bug.cgi?id=1433088#c32.  This has been well tested at multiple customer sites.

As this is no longer a problem with customer recommend closing this bug.

Comment 5 Bob Fournier 2019-05-17 11:50:36 UTC
Closing based on workaround to update database field.