Bug 467014 - python-qpid: Lacking support for multi-byte charset (UTF-8) (qpid-config,cumin)
python-qpid: Lacking support for multi-byte charset (UTF-8) (qpid-config,cumin)
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: python-qpid (Show other bugs)
1.0
All Linux
medium Severity high
: 1.1
: ---
Assigned To: Rafael H. Schloming
Kim van der Riet
:
: 469385 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-10-15 03:08 EDT by David Sommerseth
Modified: 2016-05-22 19:27 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-02-04 10:37:43 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description David Sommerseth 2008-10-15 03:08:11 EDT
When creating exchanges, routing keys/bindings or queue which contains multi-byte characters, like Kanji characters or simply "æøå" (Norwegian chars)

qpid-config gives a traceback when doing investigating exchanges or queues.


****
**** Traceback - Only queue name with utf-8
****
------------------------------------------------------------------------------
[root@dhcp-lab-204 tests]# qpid-config -b exchanges 
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.5/threading.py", line 460, in __bootstrap
    self.run()
  File "/usr/lib64/python2.5/threading.py", line 440, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.5/site-packages/qpid/queue.py", line 82, in run
    self.listener(o)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 1147, in _replyCb
    elif opcode == 'g': self.session._handleContentInd      (self, codec, seq, prop=True, stat=True)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 460, in _handleContentInd
    object = Object(self, broker, schema, codec, prop, stat)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 818, in __init__
    self.properties.append((property, self.session._decodeValue(codec, property.type)))
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 499, in _decodeValue
    elif typecode == 6:  data = str(codec.read_str8()) # SSTR
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-23: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/bin/qpid-config", line 287, in <module>
    bm.ExchangeListRecurse (modifier)
  File "/usr/bin/qpid-config", line 118, in ExchangeListRecurse
    bindings  = self.qmf.getObjects(cls="binding")
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 306, in getObjects
    raise RuntimeError("No agent responded within timeout period")
RuntimeError: No agent responded within timeout period
------------------------------------------------------------------------------





****
**** Traceback - Only routing key with utf-8
****
------------------------------------------------------------------------------
[root@dhcp-lab-204 tests]# qpid-config -b exchanges 
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.5/threading.py", line 460, in __bootstrap
    self.run()
  File "/usr/lib64/python2.5/threading.py", line 440, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.5/site-packages/qpid/queue.py", line 82, in run
    self.listener(o)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 1147, in _replyCb
    elif opcode == 'g': self.session._handleContentInd      (self, codec, seq, prop=True, stat=True)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 460, in _handleContentInd
    object = Object(self, broker, schema, codec, prop, stat)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 818, in __init__
    self.properties.append((property, self.session._decodeValue(codec, property.type)))
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 499, in _decodeValue
    elif typecode == 6:  data = str(codec.read_str8()) # SSTR
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/bin/qpid-config", line 287, in <module>
    bm.ExchangeListRecurse (modifier)
  File "/usr/bin/qpid-config", line 119, in ExchangeListRecurse
    queues    = self.qmf.getObjects(cls="queue")
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 306, in getObjects
    raise RuntimeError("No agent responded within timeout period")
RuntimeError: No agent responded within timeout period
------------------------------------------------------------------------------





****
**** Traceback - Only exchange name with utf-8
****
------------------------------------------------------------------------------
[root@dhcp-lab-204 tests]# qpid-config -b exchanges
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.5/threading.py", line 460, in __bootstrap
    self.run()
  File "/usr/lib64/python2.5/threading.py", line 440, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.5/site-packages/qpid/queue.py", line 82, in run
    self.listener(o)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 1147, in _replyCb
    elif opcode == 'g': self.session._handleContentInd      (self, codec, seq, prop=True, stat=True)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 460, in _handleContentInd
    object = Object(self, broker, schema, codec, prop, stat)
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 818, in __init__
    self.properties.append((property, self.session._decodeValue(codec, property.type)))
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 499, in _decodeValue
    elif typecode == 6:  data = str(codec.read_str8()) # SSTR
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/bin/qpid-config", line 287, in <module>
    bm.ExchangeListRecurse (modifier)
  File "/usr/bin/qpid-config", line 118, in ExchangeListRecurse
    bindings  = self.qmf.getObjects(cls="binding")
  File "/usr/lib/python2.5/site-packages/qpid/qmfconsole.py", line 306, in getObjects
    raise RuntimeError("No agent responded within timeout period")
RuntimeError: No agent responded within timeout period
------------------------------------------------------------------------------





When creating exchanges or queues via qpid-control, this traceback is given:
------------------------------------------------------------------------------
[root@dhcp-lab-204 tests]# qpid-config add queue "testæøå"
Traceback (most recent call last):
  File "/usr/bin/qpid-config", line 299, in <module>
    bm.AddQueue (cargs[2:])
  File "/usr/bin/qpid-config", line 199, in AddQueue
    self.broker.getAmqpSession().queue_declare (queue=qname, durable=_durable, arguments=declArgs)
  File "/usr/lib/python2.5/site-packages/qpid/invoker.py", line 27, in <lambda>
    method = lambda *args, **kwargs: self.invoke(resolved, args, kwargs)
  File "/usr/lib/python2.5/site-packages/qpid/session.py", line 158, in invoke
    return self.do_invoke(type, args, kwargs)
  File "/usr/lib/python2.5/site-packages/qpid/session.py", line 183, in do_invoke
    sc.write_command(hdr, cmd)
  File "/usr/lib/python2.5/site-packages/qpid/codec010.py", line 260, in write_command
    cmd._type.encode_fields(self, cmd)
  File "/usr/lib/python2.5/site-packages/qpid/spec010.py", line 256, in encode_fields
    f.type.encode(codec, values[f.name])
  File "/usr/lib/python2.5/site-packages/qpid/spec010.py", line 164, in encode
    self.type.encode(codec, value)
  File "/usr/lib/python2.5/site-packages/qpid/spec010.py", line 147, in encode
    getattr(codec, "write_%s" % self.name)(value)
  File "/usr/lib/python2.5/site-packages/qpid/codec010.py", line 124, in write_str8
    self.write_vbin8(s.encode("utf8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)
------------------------------------------------------------------------------
Comment 1 David Sommerseth 2008-10-22 11:09:57 EDT
The same entry point for an exception appears also when using the Cumin admin interface against broker with multi-byte exchanges.

-----------------------------------------------------------------------

Exception in thread Thread-17:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/usr/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.4/site-packages/qpid/queue.py", line 82, in run
    self.listener(o)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 138, in topicCb
    self.tcb (self, msg)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 311, in topicCb
    self.parse (ch, codec, hdr[0], hdr[1])
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 736, in parse
    self.parseContent (ch, 'C', codec)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 708, in parseContent
    data = self.decodeValue (codec, tc)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 415, in decodeValue
    data = str (codec.read_str8 ())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

mint.update  - Processing props    10.64.198.1:5672 qpid.exchange      5
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/usr/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.4/site-packages/qpid/queue.py", line 82, in run
    self.listener(o)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 138, in topicCb
    self.tcb (self, msg)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 311, in topicCb
    self.parse (ch, codec, hdr[0], hdr[1])
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 736, in parse
    self.parseContent (ch, 'C', codec)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 708, in parseContent
    data = self.decodeValue (codec, tc)
  File "/usr/lib/python2.4/site-packages/qpid/management.py", line 415, in decodeValue
    data = str (codec.read_str8 ())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
-----------------------------------------------------------------------
Comment 2 Ted Ross 2008-11-11 12:05:12 EST
To verify, use the qpid-config utility to create a queue (or exchange) with a name that contains multi-byte characters.  The data should be stored without error and should be able to be read back correctly by both qpid-config and qpid-tool as shown below:

$ qpid-config add queue Ted-ƟƱ

$ qpid-config queues
Queue Name                                  Attributes
=================================================================
Ted-ƟƱ                                     
reply-dhcp-100-18-254.bos.redhat.com.13726  auto-del excl
topic-dhcp-100-18-254.bos.redhat.com.13726  auto-del excl

$ qpid-tool
Management Tool for QPID
qpid: list queue
Objects of type org.apache.qpid.broker:queue
    ID   Created   Destroyed  Index
    =========================================================================
    112  15:42:23  -          103.Ted-ƟƱ
    116  15:42:37  -          103.mgmt-dhcp-100-18-254.bos.redhat.com.13742
    118  15:42:37  -          103.repl-dhcp-100-18-254.bos.redhat.com.13742
Comment 3 Ted Ross 2008-11-11 12:10:33 EST
*** Bug 469385 has been marked as a duplicate of this bug. ***
Comment 5 Frantisek Reznicek 2008-11-26 11:30:57 EST
Validated that python-qpid package now supports UTF-8 characters.
Unicode strings u'...' are needed to handle UTF strings.
Validated on package: python-qpid-0.3.719419-2.el5
->VERIFIED
Comment 6 David Sommerseth 2008-12-01 07:58:28 EST
Double checked this bug on Frantisek's request ... and it fails when sending UTF-8 characters in the message body.

---------------------------------------------------------------
Traceback (most recent call last):
  File "./utf8test", line 28, in main
    tester.SendMessage(tdata.GetExchange(), tdata.GetRoutingKey(), tdata.GetMessageBody())
  File "/mnt/tests/distribution/MRG_Messaging/qpid_i18n_multibyte_tests/tmp/python/UTF8test.py", line 53, in SendMessage
    sess.message_transfer(destination=exch, message=Message(prps, msg))
  File "/usr/lib/python2.4/site-packages/qpid/invoker.py", line 27, in <lambda>
    method = lambda *args, **kwargs: self.invoke(resolved, args, kwargs)
  File "/usr/lib/python2.4/site-packages/qpid/session.py", line 158, in invoke
    return self.do_invoke(type, args, kwargs)
  File "/usr/lib/python2.4/site-packages/qpid/session.py", line 208, in do_invoke
    self.send(seg)
  File "/usr/lib/python2.4/site-packages/qpid/session.py", line 262, in send
    self.sender.send(seg)
  File "/usr/lib/python2.4/site-packages/qpid/session.py", line 326, in send
    self.session.channel.connection.write_segment(seg)
  File "/usr/lib/python2.4/site-packages/qpid/assembler.py", line 116, in write_segment
    self.write_frame(frame)
  File "/usr/lib/python2.4/site-packages/qpid/framer.py", line 146, in write_frame
    self.write(frame.payload)
  File "/usr/lib/python2.4/site-packages/qpid/framer.py", line 85, in write
    self._buf += buf
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)
---------------------------------------------------------------

Python just loves to chew on multi-byte characters, it seems ...

I reproducer can be found via CVS ... /tests/distribution/MRG_Messaging/qpid_i18n_multibyte_tests ... Both test data and Python test program is here.

The reproducer is ran against any broker:  

cd python
./utf8test <broker host> <broker port> ../testdata/utf8_ no
./utf8test <broker host> <broker port> ../testdata/utf8_ cn
./utf8test <broker host> <broker port> ../testdata/utf8_ jp

For smoke test, you can use the English based test data
./utf8test <broker host> <broker port> ../testdata/plain en
Comment 7 David Sommerseth 2008-12-01 07:59:57 EST
Version info:

# rpm -q python-qpid
python-qpid-0.3.719419-2.el5
Comment 8 David Sommerseth 2008-12-02 04:19:31 EST
Just some more info reg. Python and Unicode/UTF-8 ... from tech-list

http://post-office.corp.redhat.com/archives/tech-list/2008-August/msg00349.html

To sum it up:
UTF-8/unicode in Python can be a quite tricky, as it defaults to 'ascii'.  It is possible to make it work with unicode, but some changes are really needed to make it work flawlessly.
Comment 9 David Sommerseth 2008-12-05 11:09:15 EST
Created attachment 325869 [details]
Test programs for C++, Java and Python

Unpack the tar.bz2 file.  Go into the qpid_i18n_multibyte_tests/python directory and run make.  This is only needed once, to install needed hashlib library if not available.  You need to be root to do this.

Then start up a broker and run:

  ./utf8test localhost 5672 ../testdata/plain_ en
  ./utf8test localhost 5672 ../testdata/utf8_ no
  ./utf8test localhost 5672 ../testdata/utf8_ cn
  ./utf8test localhost 5672 ../testdata/utf8_ jp
Comment 10 Gordon Sim 2008-12-05 13:58:47 EST
Confirmed that in fact the application should itself be responsible for determining correct encoding and encoding/decoding as appropriate. Modified the (excellent!) test suite and sent changes to David for inclusion.
Comment 12 errata-xmlrpc 2009-02-04 10:37:43 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-0035.html

Note You need to log in before you can comment on or make changes to this bug.