Hide Forgot
Created attachment 512254 [details] patch; also posted to bug-gnu-utils Description of problem: long line in input causes loss of all input Version-Release number of selected component (if applicable): sed-4.2.1-6.fc15.x86_64 How reproducible: every time Steps to Reproduce: 1. perl -le 'print "v"x(2**31-1)' |sed 's/^/v/' |wc -c 2. observe that sed maps that 2GiB one-line input to empty output 3. Actual results: 0 Expected results: 2147483648 Additional info: patch attached
sed-4.2.1-7.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/sed-4.2.1-7.fc15
sed-4.2.1-6.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/sed-4.2.1-6.fc14
In my F15 x86_64 host, with 4 GiB of RAM, the update fixes the bug - output before and after updating sed: [andre@compaq-pc tmp]$ perl -le 'print "v"x(2**31-1)' |sed 's/^/v/' |wc 0 0 0 [andre@compaq-pc tmp]$ perl -le 'print "v"x(2**31-1)' |sed 's/^/v/' |wc 1 1 2147483648 [andre@compaq-pc tmp]$ However, in my VirtualBox F14 and F15 x86_64 guests with 1 GiB of RAM allocated to each, it does not - the output is still " 0 0 0". I also noticed that in the guests, the command returns earlier. So I'm guessing that with insufficient RAM, the command exits prematurely giving the wrong result? And if so, which component is responsible?
I believe it is perl, but you can test: perl -le 'print "v"x(2**31-1)' > file ls -l file if empty perl is the culprit. if not empty sed is the culprit, you can confirm with: sed 's/^/v/' file | wc Also important is testing that sed -i does not destroy a file. In your 1GiB guests, try: perl -e 'print "v"x(2**29)' > small # note no -l here cat small small small small > file wc file # should be 0 1 2147483648 sed -i 's/^/v/' file # should fail wc file # should still be 0 1 2147483648 You will need a few GiB of free disk space for this test. Thanks!
In my F15 x86_64 1 GiB guest: [andre@localhost ~]$ perl -le 'print "v"x(2**31-1)' > file [andre@localhost ~]$ ls -l file -rw-rw-r--. 1 andre andre 2147483648 Jul 13 06:48 file [andre@localhost ~]$ sed 's/^/v/' file | wc sed: couldn't re-allocate memory 0 0 0 [andre@localhost ~]$ and [andre@localhost ~]$ perl -e 'print "v"x(2**29)' > small [andre@localhost ~]$ cat small small small small > file [andre@localhost ~]$ wc file 0 1 2147483648 file [andre@localhost ~]$ sed -i 's/^/v/' file sed: couldn't re-allocate memory [andre@localhost ~]$ wc file 0 1 2147483648 file [andre@localhost ~]$ I'll assume that the F14 behavior is the same, and leave my karma at -1 for both (although if this issue is hard to fix, and existed before, I could change them to +1 anyway since it works for large amounts of RAM).
1) Yes, the issue existed before too. There's nothing really to fix, perhaps it's just extra swap that you need. 2) The second test is the important one, i.e. sed doesn't destroy data when it exits with an error. As long as the exit code of sed is 1 after the "couldn't re-allocate memory" error I think the patch is fine. Before, the failure was silent and happened independent of the amount of memory.
On the same guest, the exit code is 0, not 1: [andre@localhost ~]$ sed 's/^/v/' file | wc sed: couldn't re-allocate memory 0 0 0 [andre@localhost ~]$ echo $? 0 [andre@localhost ~]$
Hold on, that would be the exit code of wc, not sed. Let me figure out how to test sed itself and report back.
[andre@localhost ~]$ sed 's/^/v/' file > file2 sed: couldn't re-allocate memory [andre@localhost ~]$ echo $? 4 [andre@localhost ~]$ I'll change my F15 karma to +1, and retest F14 as well.
Thanks!
sed-4.2.1-7.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.
sed-4.2.1-6.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.