Red Hat Bugzilla – Bug 176488
grep SLOW on multibyte LC_CTYPE
Last modified: 2007-11-30 17:07:22 EST
+++ This bug was initially created as a clone of Bug #121313 +++
Description of problem:
grep is painfully slow on multibyte locales. Slowdown factor >30 observed.
Version-Release number of selected component (if applicable):
~ $ time LC_CTYPE=en_US.UTF-8 grep '^//PS ' /tmp/r3.log | wc -l
grep : 97.31s user 0.17s system 87% cpu 1:51.15 total
~ $ time LC_CTYPE=C grep '^//PS ' /tmp/r3.log | wc -l
grep : 0.22s user 0.04s system 83% cpu 0.312 total
Test file attached later on, and also downloadable from:
It's 40KB gzipped, 2.6MB gunzipped.
-- Additional comment from Emmanuel.Thome@inria.fr on 2004-04-20 08:31 EST --
Created an attachment (id=99558)
Test file I used
-- Additional comment from firstname.lastname@example.org on 2004-04-20 12:38 EST --
(2.5.1-26 is a devel package; changing version.)
-- Additional comment from email@example.com on 2004-04-20 12:48 EST --
The longer-term solution is to make grep use the system regex for
multibyte encodings. The GNU libc implementation has quite an
efficient implementation now.
-- Additional comment from firstname.lastname@example.org on 2004-11-08 09:33 EST --
Please try grep-2.5.1-36, available at:
-- Additional comment from Emmanuel.Thome@inria.fr on 2004-11-08 09:58 EST --
I'm happy with it.
with GREP_USE_DFA set, I observe a 2x slowdown.
-- Additional comment from email@example.com on 2004-11-10 06:39 EST --
grep-2.5.1-37 fixes a problem that can cause false matches. It will
be available in the Fedora development tree tomorrow, or at:
-- Additional comment from firstname.lastname@example.org on 2005-12-22 12:35 EST --
Nice to know the problem was fixed in Fedora Core. However it seems that
grep-2.5.1-31 (RHEL4) still suffers from this problem. Any chance of fixing
that one too? Looking at the dates in comments, I kinda expected that there
would be new version of grep released as part of U1 or at latest U2.
One additional thing. I found that grep is slow if there are many matches. If
there are no matches (or just a few of matches), it is fast.
LANG=en_US.UTF-8 # Should be default
while [ $a -lt 30000 ]; do
printf "%.9d0\n" $a; a=$(( $a + 1 ))
done > testfile.txt
echo "Going to be sloooow... Get yourself some coffe"
time grep -c '0$' testfile.txt
echo "However, this one is fast. Sorry, no time for coffe"
time grep -c '1$' testfile.txt
It takes about 25 seconds on 2.8GHz Pentium D to run the first grep (jeeez).
The second grep (that doesn't match any lines from the file) is fast. Of
course, setting LANG to C or en_US solves the problem.
*** This bug has been marked as a duplicate of 179636 ***