Description of problem: If you run grep on a large sparse file (eg. multi-gigabyte) then grep consumes more and more memory. The system as a whole becomes more unusable. Eventually grep dies: grep: memory exhausted Version-Release number of selected component (if applicable): grep-2.9-3.fc16.x86_64 How reproducible: 100% Steps to Reproduce: 1. truncate -s 32G sparse 2. grep foo sparse 3. Actual results: System becomes unusable until grep finally dies with grep: memory exhausted Expected results: grep should work a bit better in this case. After all, this file is completely empty. Additional info: Possibly http://savannah.gnu.org/bugs/?31928 ?
> Possibly http://savannah.gnu.org/bugs/?31928 ? No, it doesn't seem to be related and the exit code has been already fixed in Fedora. Upstream ticket: http://savannah.gnu.org/bugs/index.php?34020
> After all, this file is completely empty. No, it's not, it has a single line consisting of 34 billion NUL characters. But yes, it should work a bit better.
I should say that I don't really care about grepping large sparse files. The problem arises because libtool runs grep (and sed) on all of its parameters. In libguestfs this causes a problem, because we often run commands which come down to: libtool --mode=execute guestfish -a test1.img where 'test1.img' is a huge sparse file. Therefore either libtool or grep need to be fixed to deal with this case.
FYI there is a similar bug concerning 'sed', 'libtool' and huge sparse files: https://bugzilla.redhat.com/show_bug.cgi?id=636045
I think the bug is in libtool, though the observation you made about non-sparse files is interesting. As mentioned in comment 2: to grep and sed, a large sparse file has a huge line consisting of all zeros, and grep/sed need to slurp it all in memory (for sed; in order to edit it; for grep, in case it has to print it). Since the libtool problem is already opened, I'm closing this one.