Bug 466011

Summary: Bug in unicode handling between python-sqlobject-0.9.7 and MySQL-python-1.2.1-1
Product: [Fedora] Fedora EPEL Reporter: Toshio Ernie Kuratomi <a.badger>
Component: python-sqlobjectAssignee: Luke Macken <lmacken>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: el5CC: a.badger, andred, fschwarz, lmacken, pfrields
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 0.9.7-2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-10-18 00:39:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Toshio Ernie Kuratomi 2008-10-07 18:04:54 UTC 
Description of problem:
When using SQLObject 0.9.7 on EL-5 (with MySQL-python-1.2.1-1) Unicode errors are thrown when unicode input is given.


How reproducible:
Everytime

Steps to Reproduce:
1. Run transifex-0.2 on an EL-5 box
2. Attempt to submit something while logged in with a username that uses unicode characters.
3.
  
Actual results:
Traceback

Expected results:
upload will be submitted

Additional info:
The root cause of this is that SQLObject-0.9.7 and above is passing the query string to the MySQL bindings as a byte string and the MySQL bindings are expecting a unicode string.  In Fedora-9 this is fixed in the MySQL bindings by checking whether unicode or a byte string were given.

In SQLObject-0.9.2, there is a check in mysql/mysqlconnection.py that checks if the version of Mysql is greater than X.Y and if so transforms the query string to unicode prior to submitting it to the backend.

We have several choices for resolving this:

* open a bug against EL-5's MySQL-python
* return to python-sqlobject-0.9.2
* forward port the detection of MySQL-python version and subsequent transformation to python-sqlobject-0.9.7

Looking at the code, this appears to affect 0.10.2 as well.
Comment 1 Toshio Ernie Kuratomi 2008-10-07 18:12:47 UTC 
Relevant section of: sqlobject/mysql/mysqlconnection.py

class MySQLConnection(DBAPI):
[...]
    def __init__(self, db, user, password='', host='localhost', port=0, **kw):
[...]
        if MySQLdb.version_info[:3] >= (1, 2, 1):
            # need to add: and MySQLdb.version_info[:3] < (1, 2, 2):
            self.need_unicode = True
        else:
            self.need_unicode = False
[...]
    def _executeRetry(self, conn, cursor, query):
[...]
                if self.need_unicode and not isinstance(query, unicode):
                    try:
                        query = unicode(query, self.encoding)
                    except UnicodeError:
                        pass                        
                return cursor.execute(query)
Comment 2 Toshio Ernie Kuratomi 2008-10-08 02:03:31 UTC 
- query = unicode(query, self.encoding)
+ query = unicode(query, self.dbEncoding)
Comment 3 Toshio Ernie Kuratomi 2008-10-08 02:25:56 UTC 
Hotpatched this on app4 in fedora infrastructure.  This seems to allow transifex-0.2 to work.
Comment 4 Toshio Ernie Kuratomi 2008-10-18 00:20:42 UTC 
Patch under discussion on upstream mailing list and applied toour EL-5 package.
Comment 5 andred 2009-02-20 04:15:57 UTC 
(In reply to comment #2)
> - query = unicode(query, self.encoding)
> + query = unicode(query, self.dbEncoding)

This modification doesn't work in all cases. 

class MySQLConnection(DBAPI):
[...]
    def __init__(self, db, user, password='', host='localhost', port=0, **kw):
[...]
        if "charset" in kw:
            self.dbEncoding = self.kw["charset"] = col.popKey(kw, "charset")
        else:
            self.dbEncoding = None

self.dbEncoding will be None if charset isn't passed in with the kw's. This then causes

[...]
    def _executeRetry(self, conn, cursor, query):
[...]
                if self.need_unicode and not isinstance(query, unicode):
                    try:
                        query = unicode(query, self.dbEncoding)
                except UnicodeError:
 

to throw a TypeError which isn't caught by the exception handler.

TypeError: unicode() argument 2 must be string, not None

I have run into this using TurboGears / sqlobject to connect to a MySQL DB while attempting to support RHEL for a custom project.