Description of problem: unicode string concatenation in python sucks large granite blocks through small glass pipettes. Sideways. $ cat foo.py #!/usr/bin/python import os import sys import codecs f = open(sys.argv[1]) lines = f.readlines() f.close() res = u'' f = codecs.open('/dev/null', 'w', encoding='utf8') for line in lines: res = res + unicode(line, 'utf8') f.write(res) f.close() $ time ./foo.py <a 1.9MB XML file with 783 lines> real 0m4.155s user 0m2.256s sys 0m1.757s $ cat foo2.py #!/usr/bin/python import os import sys import codecs f = open(sys.argv[1]) lines = f.readlines() f.close() f = codecs.open('/dev/null', 'w', encoding='utf8') for line in lines: f.write(unicode(line,'utf8')) f.close() $ time ./foo2.py <same XML file> real 0m0.060s user 0m0.042s sys 0m0.018s Really? 69 times slower? (For reference, when not using unicode strings, the same programs run in ~0.05 seconds and ~0.04 seconds, respectively.) Version-Release number of selected component (if applicable): python-2.6-7.fc11.x86_64 How reproducible: 100%
% perl -le 'print $_ for (1..1_000)' > /tmp/abcd % time /tmp/t.py /tmp/abcd /tmp/t.py /tmp/abcd 0.02s user 0.00s system 94% cpu 0.023 total % perl -le 'print $_ for (1..10_000)' > /tmp/abcd % time /tmp/t.py /tmp/abcd /tmp/t.py /tmp/abcd 0.20s user 0.00s system 98% cpu 0.213 total % perl -le 'print $_ for (1..100_000)' > /tmp/abcd % time /tmp/t.py /tmp/abcd /tmp/t.py /tmp/abcd 40.54s user 42.27s system 99% cpu 1:23.29 total ...and indeed changing the loop to: res = '' for line in lines: res += line ret = unicode(res, 'utf8') ...takes the time for the last case down to: /tmp/t.py /tmp/abcd 0.07s user 0.01s system 97% cpu 0.082 total
I should note the above tests where done on python-0:2.5.2-1.fc10.x86_64 and python-0:2.4.3-24.el5.x86_64, giving the same results.
The old join hack works too: res = [] for line in lines: res.append(unicode(line, 'utf8')) res = u''.join(res) ...gives: /tmp/t.py /tmp/abcd 0.37s user 0.02s system 99% cpu 0.387 total ...amusingly this is ~4x worse than the "use bytes until the end" code, but ~250x better than the original.
It's also worth pointing out that 2.6.2 gives: /opt/py2.6.2/bin/python /tmp/t.py /tmp/abcd 13.49s user 19.10s system 98% cpu 33.101 total ...for the default loop, which is better by almost ~3x (but still sucks).
This message is a reminder that Fedora 11 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 11. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '11'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 11's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 11 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.