Bug 720438 - long line in input causes loss of all input
Summary: long line in input causes loss of all input
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: sed
Version: 15
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Vojtech Vitek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-07-11 16:01 UTC by Jim Meyering
Modified: 2015-03-04 23:57 UTC (History)
6 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2011-07-13 19:26:36 UTC


Attachments (Terms of Use)
patch; also posted to bug-gnu-utils (2.47 KB, patch)
2011-07-11 16:01 UTC, Jim Meyering
no flags Details | Diff

Description Jim Meyering 2011-07-11 16:01:36 UTC
Created attachment 512254 [details]
patch; also posted to bug-gnu-utils

Description of problem: long line in input causes loss of all input


Version-Release number of selected component (if applicable): sed-4.2.1-6.fc15.x86_64


How reproducible: every time


Steps to Reproduce:
1. perl -le 'print "v"x(2**31-1)' |sed 's/^/v/' |wc -c
2. observe that sed maps that 2GiB one-line input to empty output
3.
  
Actual results: 0


Expected results: 2147483648


Additional info: patch attached

Comment 1 Fedora Update System 2011-07-12 14:31:12 UTC
sed-4.2.1-7.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/sed-4.2.1-7.fc15

Comment 2 Fedora Update System 2011-07-12 14:32:48 UTC
sed-4.2.1-6.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/sed-4.2.1-6.fc14

Comment 3 Andre Robatino 2011-07-13 10:17:30 UTC
In my F15 x86_64 host, with 4 GiB of RAM, the update fixes the bug - output before and after updating sed:

[andre@compaq-pc tmp]$ perl -le 'print "v"x(2**31-1)' |sed 's/^/v/' |wc
      0       0       0
[andre@compaq-pc tmp]$ perl -le 'print "v"x(2**31-1)' |sed 's/^/v/' |wc
      1       1 2147483648
[andre@compaq-pc tmp]$

However, in my VirtualBox F14 and F15 x86_64 guests with 1 GiB of RAM allocated to each, it does not - the output is still "      0       0       0". I also noticed that in the guests, the command returns earlier. So I'm guessing that with insufficient RAM, the command exits prematurely giving the wrong result? And if so, which component is responsible?

Comment 4 Paolo Bonzini 2011-07-13 10:35:47 UTC
I believe it is perl, but you can test:

perl -le 'print "v"x(2**31-1)' > file
ls -l file

if empty perl is the culprit.
if not empty sed is the culprit, you can confirm with:

sed 's/^/v/' file | wc

Also important is testing that sed -i does not destroy a file.  In your 1GiB guests, try:

perl -e 'print "v"x(2**29)' > small  # note no -l here
cat small small small small > file
wc file                              # should be 0 1 2147483648
sed -i 's/^/v/' file                 # should fail
wc file                              # should still be 0 1 2147483648

You will need a few GiB of free disk space for this test.  Thanks!

Comment 5 Andre Robatino 2011-07-13 11:04:54 UTC
In my F15 x86_64 1 GiB guest:

[andre@localhost ~]$ perl -le 'print "v"x(2**31-1)' > file
[andre@localhost ~]$ ls -l file
-rw-rw-r--. 1 andre andre 2147483648 Jul 13 06:48 file
[andre@localhost ~]$ sed 's/^/v/' file | wc
sed: couldn't re-allocate memory
      0       0       0
[andre@localhost ~]$

and

[andre@localhost ~]$ perl -e 'print "v"x(2**29)' > small
[andre@localhost ~]$ cat small small small small > file
[andre@localhost ~]$ wc file
         0          1 2147483648 file
[andre@localhost ~]$ sed -i 's/^/v/' file
sed: couldn't re-allocate memory
[andre@localhost ~]$ wc file
         0          1 2147483648 file
[andre@localhost ~]$

I'll assume that the F14 behavior is the same, and leave my karma at -1 for both (although if this issue is hard to fix, and existed before, I could change them to +1 anyway since it works for large amounts of RAM).

Comment 6 Paolo Bonzini 2011-07-13 11:53:31 UTC
1) Yes, the issue existed before too.  There's nothing really to fix, perhaps it's just extra swap that you need.

2) The second test is the important one, i.e. sed doesn't destroy data when it exits with an error.

As long as the exit code of sed is 1 after the "couldn't re-allocate memory" error I think the patch is fine.  Before, the failure was silent and happened independent of the amount of memory.

Comment 7 Andre Robatino 2011-07-13 12:15:50 UTC
On the same guest, the exit code is 0, not 1:

[andre@localhost ~]$ sed 's/^/v/' file | wc
sed: couldn't re-allocate memory
      0       0       0
[andre@localhost ~]$ echo $?
0
[andre@localhost ~]$

Comment 8 Andre Robatino 2011-07-13 12:17:17 UTC
Hold on, that would be the exit code of wc, not sed. Let me figure out how to test sed itself and report back.

Comment 9 Andre Robatino 2011-07-13 12:29:49 UTC
[andre@localhost ~]$ sed 's/^/v/' file > file2
sed: couldn't re-allocate memory
[andre@localhost ~]$ echo $?
4
[andre@localhost ~]$

I'll change my F15 karma to +1, and retest F14 as well.

Comment 10 Paolo Bonzini 2011-07-13 12:31:42 UTC
Thanks!

Comment 11 Fedora Update System 2011-07-13 19:26:30 UTC
sed-4.2.1-7.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2011-08-09 01:24:46 UTC
sed-4.2.1-6.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.