Bug 505220 - unicode string concatenation is just a little slow
unicode string concatenation is just a little slow
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
All Linux
low Severity medium
: ---
: ---
Assigned To: James Antill
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2009-06-11 00:34 EDT by Bill Nottingham
Modified: 2014-03-16 23:18 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2010-06-28 08:53:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Bill Nottingham 2009-06-11 00:34:38 EDT
Description of problem:

unicode string concatenation in python sucks large granite blocks through small glass pipettes. Sideways.

$ cat foo.py

import os
import sys
import codecs

f = open(sys.argv[1])
lines = f.readlines()

res = u''
f = codecs.open('/dev/null', 'w', encoding='utf8')
for line in lines:
    res = res + unicode(line, 'utf8')

$ time ./foo.py <a 1.9MB XML file with 783 lines>

real    0m4.155s
user    0m2.256s
sys     0m1.757s

$ cat foo2.py

import os
import sys
import codecs

f = open(sys.argv[1])
lines = f.readlines()

f = codecs.open('/dev/null', 'w', encoding='utf8')
for line in lines:

$ time ./foo2.py <same XML file>

real    0m0.060s
user    0m0.042s
sys     0m0.018s

Really? 69 times slower? 

(For reference, when not using unicode strings, the same programs
run in ~0.05 seconds and ~0.04 seconds, respectively.)

Version-Release number of selected component (if applicable):


How reproducible:

Comment 1 James Antill 2009-06-11 10:13:28 EDT
% perl -le 'print $_ for (1..1_000)' > /tmp/abcd 
% time /tmp/t.py /tmp/abcd
/tmp/t.py /tmp/abcd  0.02s user 0.00s system 94% cpu 0.023 total
% perl -le 'print $_ for (1..10_000)' > /tmp/abcd
% time /tmp/t.py /tmp/abcd
/tmp/t.py /tmp/abcd  0.20s user 0.00s system 98% cpu 0.213 total
% perl -le 'print $_ for (1..100_000)' > /tmp/abcd
% time /tmp/t.py /tmp/abcd
/tmp/t.py /tmp/abcd  40.54s user 42.27s system 99% cpu 1:23.29 total

...and indeed changing the loop to:

res = ''
for line in lines:
    res += line
ret = unicode(res, 'utf8')

...takes the time for the last case down to:

/tmp/t.py /tmp/abcd  0.07s user 0.01s system 97% cpu 0.082 total
Comment 2 James Antill 2009-06-11 10:17:54 EDT
I should note the above tests where done on python-0:2.5.2-1.fc10.x86_64 and python-0:2.4.3-24.el5.x86_64, giving the same results.
Comment 3 James Antill 2009-06-11 10:35:46 EDT
 The old join hack works too:

    res = []
    for line in lines:
        res.append(unicode(line, 'utf8'))
    res = u''.join(res)


/tmp/t.py /tmp/abcd  0.37s user 0.02s system 99% cpu 0.387 total

...amusingly this is ~4x worse than the "use bytes until the end" code, but ~250x better than the original.
Comment 4 James Antill 2009-06-11 11:31:12 EDT
 It's also worth pointing out that 2.6.2 gives:

/opt/py2.6.2/bin/python /tmp/t.py /tmp/abcd  13.49s user 19.10s system 98% cpu 33.101 total

...for the default loop, which is better by almost ~3x (but still sucks).
Comment 5 Bug Zapper 2010-04-27 10:45:45 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
Comment 6 Bug Zapper 2010-06-28 08:53:21 EDT
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.