Hide Forgot
Description of problem: Non-Latin characters entered in the MAC address field for a system could be silently truncated by MySQL across UTF-8 character boundaries. The row will then fail to load. Steps to Reproduce: 1. Set a system's MAC address to "aяяяяяяяяяяяяяяяяяяяяяяяяяя" 2. Hit save 3. Boom Actual results: An exception like this: 26/Oct/2011:23:06:34 HTTP INFO Page handler: <bound method Root.view of <bkr.server.controllers.Root object at 0x2aaab59c7850>> Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 121, in _run self.main() File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 264, in main body = page_handler(*virtual_path, **self.params) File "/usr/lib/python2.4/site-packages/bkr/server/controllers.py", line 1044, in view return self._view_system_as_html(fqdn, **kwargs) File "<string>", line 3, in _view_system_as_html File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 358, in expose output = database.run_with_transaction( File "<string>", line 5, in run_with_transaction File "/usr/lib/python2.4/site-packages/turbogears/database.py", line 407, in sa_rwt retval = func(*args, **kw) File "<string>", line 5, in _expose File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 373, in <lambda> mapping, fragment, args, kw))) File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 410, in _execute_func output = errorhandling.try_call(func, *args, **kw) File "/usr/lib/python2.4/site-packages/turbogears/errorhandling.py", line 77, in try_call return func(self, *args, **kw) File "<string>", line 3, in _view_system_as_html File "/usr/lib/python2.4/site-packages/turbogears/paginate.py", line 168, in decorated output = func(*args, **kw) File "/usr/lib/python2.4/site-packages/bkr/server/controllers.py", line 868, in _view_system_as_html system = System.by_fqdn(fqdn,identity.current.user) File "/usr/lib/python2.4/site-packages/bkr/server/model.py", line 1848, in by_fqdn return System.all(user).filter(System.fqdn == fqdn).one() File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 924, in one ret = list(self[0:2]) File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 1003, in iterate_instances process[0](context, row, rows) File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 1581, in main extension=context.extension, only_load_props=context.only_load_props, refresh_instance=context.refresh_instance File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/mapper.py", line 1444, in _instance self.populate_instance(context, instance, row, only_load_props=only_load_props, instancekey=identitykey, isnew=isnew) File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/mapper.py", line 1509, in populate_instance selectcontext.exec_with_path(self, key, populator, instance, row, ispostselect=ispostselect, isnew=isnew, **flags) File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 1703, in exec_with_path return fn(*args, **kwargs) File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/strategies.py", line 79, in new_execute instance.__dict__[self.key] = row[self.columns[0]] File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 1339, in __getitem__ return self.__parent._get_col(self.__row, key) File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 1605, in _get_col return processor(row[index]) File "/usr/lib/python2.4/site-packages/sqlalchemy/types.py", line 407, in process return value.decode(dialect.encoding) File "/usr/lib64/python2.4/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xef in position 17: unexpected end of data Expected results: Value should be truncated on character boundaries, or at least refuse to be saved. Additional info: There are probably heaps of places in Beaker subject to this problem. Same problem as bug 652334 but in a different place.
*** Bug 749353 has been marked as a duplicate of this bug. ***
This bug is actually caused by a misconfiguration with our MySQL connection. Although our database character set and column collations are set correctly (UTF-8) and all values are being treated as UTF-8 on the Python side, we never set the connection character set and so that is defaulting to latin-1. As a result MySQL believes we are sending it latin-1 and re-encoding all the bytes we give it as UTF-8. This encoding is lossless which is why we have not noticed a problem before, apart from these issues with truncation across character boundaries.