Bug 749441 - Invalid UTF-8 char in MAC address field
Summary: Invalid UTF-8 char in MAC address field
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Retired
Component: inventory
Version: 0.7
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: ---
Assignee: Dan Callaghan
QA Contact:
URL:
Whiteboard:
: 749353 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-10-27 04:07 UTC by Dan Callaghan
Modified: 2019-05-22 13:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-03-15 00:16:34 UTC


Attachments (Terms of Use)

Description Dan Callaghan 2011-10-27 04:07:32 UTC
Description of problem:
Non-Latin characters entered in the MAC address field for a system could be silently truncated by MySQL across UTF-8 character boundaries. The row will then fail to load.

Steps to Reproduce:
1. Set a system's MAC address to "aяяяяяяяяяяяяяяяяяяяяяяяяяя"
2. Hit save
3. Boom
  
Actual results:
An exception like this:

    26/Oct/2011:23:06:34 HTTP INFO Page handler: <bound method Root.view of <bkr.server.controllers.Root object at 0x2aaab59c7850>>
    Traceback (most recent call last):
      File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 121, in _run
        self.main()
      File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 264, in main
        body = page_handler(*virtual_path, **self.params)
      File "/usr/lib/python2.4/site-packages/bkr/server/controllers.py", line 1044, in view
        return self._view_system_as_html(fqdn, **kwargs)
      File "<string>", line 3, in _view_system_as_html
      File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 358, in expose
        output = database.run_with_transaction(
      File "<string>", line 5, in run_with_transaction
      File "/usr/lib/python2.4/site-packages/turbogears/database.py", line 407, in sa_rwt
        retval = func(*args, **kw)
      File "<string>", line 5, in _expose
      File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 373, in <lambda>
        mapping, fragment, args, kw)))
      File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 410, in _execute_func
        output = errorhandling.try_call(func, *args, **kw)
      File "/usr/lib/python2.4/site-packages/turbogears/errorhandling.py", line 77, in try_call
        return func(self, *args, **kw)
      File "<string>", line 3, in _view_system_as_html
      File "/usr/lib/python2.4/site-packages/turbogears/paginate.py", line 168, in decorated
        output = func(*args, **kw)
      File "/usr/lib/python2.4/site-packages/bkr/server/controllers.py", line 868, in _view_system_as_html
        system = System.by_fqdn(fqdn,identity.current.user)
      File "/usr/lib/python2.4/site-packages/bkr/server/model.py", line 1848, in by_fqdn
        return System.all(user).filter(System.fqdn == fqdn).one()
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 924, in one
        ret = list(self[0:2])
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 1003, in iterate_instances
        process[0](context, row, rows)
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 1581, in main
        extension=context.extension, only_load_props=context.only_load_props, refresh_instance=context.refresh_instance
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/mapper.py", line 1444, in _instance
        self.populate_instance(context, instance, row, only_load_props=only_load_props, instancekey=identitykey, isnew=isnew)
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/mapper.py", line 1509, in populate_instance
        selectcontext.exec_with_path(self, key, populator, instance, row, ispostselect=ispostselect, isnew=isnew, **flags)
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/query.py", line 1703, in exec_with_path
        return fn(*args, **kwargs)
      File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/strategies.py", line 79, in new_execute
        instance.__dict__[self.key] = row[self.columns[0]]
      File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 1339, in __getitem__
        return self.__parent._get_col(self.__row, key)
      File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 1605, in _get_col
        return processor(row[index])
      File "/usr/lib/python2.4/site-packages/sqlalchemy/types.py", line 407, in process
        return value.decode(dialect.encoding)
      File "/usr/lib64/python2.4/encodings/utf_8.py", line 16, in decode
        return codecs.utf_8_decode(input, errors, True)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xef in position 17: unexpected end of data 

Expected results:
Value should be truncated on character boundaries, or at least refuse to be saved.

Additional info:
There are probably heaps of places in Beaker subject to this problem. Same problem as bug 652334 but in a different place.

Comment 1 Dan Callaghan 2011-10-27 22:44:25 UTC
*** Bug 749353 has been marked as a duplicate of this bug. ***

Comment 2 Dan Callaghan 2011-10-31 00:53:26 UTC
This bug is actually caused by a misconfiguration with our MySQL connection. Although our database character set and column collations are set correctly (UTF-8) and all values are being treated as UTF-8 on the Python side, we never set the connection character set and so that is defaulting to latin-1. As a result MySQL believes we are sending it latin-1 and re-encoding all the bytes we give it as UTF-8. This encoding is lossless which is why we have not noticed a problem before, apart from these issues with truncation across character boundaries.


Note You need to log in before you can comment on or make changes to this bug.