Bug 143963

Summary: unbuffer utility in expect package does not work.
Product: [Fedora] Fedora Reporter: Charles Sullivan <cwsulliv01>
Component: expectAssignee: Jens Petersen <petersen>
Status: CLOSED RAWHIDE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3CC: mattdm
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: 5.43.0-2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-06-08 05:43:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
unbuffer none

Description Charles Sullivan 2005-01-02 18:43:23 UTC
Description of problem:
The unbuffer utility does not work - the "unbuffer"ed command 
provides no output, just hangs.

Version-Release number of selected component (if applicable):
expect-5.39.0-96.1

How reproducible:
Always.

Steps to Reproduce:
Example using 'grep', first with grep's --line-buffered option:

$ /bin/grep --line-buffered Now - | /bin/grep time
Now is the time  <-- typed
Now is the time  <-- echoed
Is Now the time  <-- typed
Is Now the time  <-- echoed

<break>

Again, but with unbuffer:
$ unbuffer /bin/grep Now - | /bin/grep time
Now is the time  <-- typed
Is Now the time  <-- typed

Expected results:
The output should be the same as with the --line-buffered option.

Another example with grep'ing from a file containing the lines:
  Now is the time for
  all good men to come
  to the aid of Linux

$ grep all filename.txt | unbuffer /bin/grep good | grep come

Actual results:
The command line just hangs.

Expected results:
Command should terminate and display the second line of the above 
paragraph.

Additional info:
unbuffer doesn't work with any command I've tried, not just grep.
My newer system runs Fedora Core 2 upgraded to current date with 
up2date, but I can't get unbuffer to work on an older system under 
Red Hat 9 either.

Comment 1 Matthew Miller 2005-04-26 16:07:09 UTC
Fedora Core 2 is now maintained by the Fedora Legacy project for
security updates only. If this problem is a security issue, please
reopen and reassign to the Fedora Legacy product. If it is not a
security issue and hasn't been resolved in the current FC3 updates or
in the FC4 test release, reopen and change the version to match.

Comment 2 Jens Petersen 2005-05-06 09:03:50 UTC
Reproduced.

Comment 4 Jens Petersen 2005-05-19 21:05:19 UTC
Apparently this should be fixed in the latest expect.

Could you please try expect-5.43.0-1.src.rpm from Fedora Core development
since it should be fixed there.

Comment 5 Charles Sullivan 2005-05-22 14:42:17 UTC
Sorry, I can't get 5.43.0-1 to work on my FC2 system.  From the source rpm,
./configure exits with:
checking for Tcl configuration... configure: warning: Can't find Tcl    
configuration definitions
./configure: line 1106: #: No such file or directory

The binary expect-5.43.0-1.i386.rpm from rpmfind.net installs, but attempting
to run my example fails, i.e.,
$ unbuffer /bin/grep Now - | /bin/grep time
expect: relocation error: /usr/lib/libexpect5.43.so: symbol __sprintf_chk, 
version GLIBC_2.3.4 not defined in file libc.so.6 with link time reference

(I have glibc-2.3.3-27.1.  I'm afraid upgrading to 2.3.4 will break a hundred 
other things.)






Comment 6 Charles Sullivan 2005-05-22 15:20:31 UTC
Forgot to add: Thanks for your attention to this problem Jens.  I've since 
hacked the utility I was attempting to 'unbuffer' so my immediate need for a fix 
has passed. 

Comment 7 Jens Petersen 2005-05-23 03:47:18 UTC
I can reproduce the problem in comment 5 with FC3 too.  It seems expect
doesn't like current autotools: thank you for pointing that out.

Comment 8 Jens Petersen 2005-05-23 05:20:49 UTC
Hmm I think my comment 2 and comment 4 are in contradiction though.


Comment 9 Jens Petersen 2005-05-23 06:31:23 UTC
expect-5.43.0 doesn't seem to like automake-1.8...
If you replace aclocal in the spec file with aclocal-1.7
it builds ok for me on FC2.

Comment 10 Jens Petersen 2005-05-27 11:33:50 UTC
Ok, I received clarification from the maintainer: comment 4 is misleading.

| 1) unbuffer does not support hand-entered input.  That's why the first example
| don't work.  Frankly I've never seen a real example where this is important
| but if you can justify a case in which this is useful, I think I can add a flag
| to provide it.
|
| 2) In the second example, you need to use the -p flag for unbuffer to work in
| a pipeline (see the man page for more info) but even this won't help the fact
| that your example still exposes a latent bug in unbuffer.  I've attached a new
| version of unbuffer which should work better.  It should be called his way:
|
|  grep all filename.txt | unbuffer -p /bin/grep good | grep come 
|


Comment 11 Jens Petersen 2005-05-27 11:35:46 UTC
Created attachment 114905 [details]
unbuffer

New version of unbuffer.

Could you test this please? :)

Comment 12 Charles Sullivan 2005-05-28 02:11:44 UTC
I changed the first line of the new unbuffer script to:
#!/usr/bin/expect --

But when I try it with expect-5.39.0-96.1, I get the following error:
$ grep all goodmen.txt | ./unbuffer -p /bin/grep good | grep come
invalid command name "close_on_eof"
    while executing
"close_on_eof -i $user_spawn_id 0"
    invoked from within
"if {[string compare [lindex $argv 0] "-p"] == 0} {
    # pipeline
    set stty_init "-echo"
    eval spawn -noecho [lrange $argv 1 end]
    close_on_e..."
    (file "./unbuffer" line 5)
$

So I again tried to compile expect-5.43 from source.
I'm not sure how to change the spec file per your comment #9, so I made a
link in /usr/bin:  aclocal --> aclocal-1.7

But configure still generates an error:

checking for Tcl configuration... configure: warning: Can't find Tcl 
configuration definitions
./configure: line 1106: #: No such file or directory

   == == ==

For the record, the (failed) command line which prompted this bug report was:
  $ cat /dev/ttyS0 | unbuffer -p /usr/bin/xxd -g1 -c5 | my_timestamper

Where a peripheral device is sending 5-byte binary messages to the serial port
at irregular and infrequent intervals.  The binary messages are translated into 
hex by 'xxd' and the current system time is appended to each line by 
'my_timestamper'.

(When this failed to work I hacked 'xxd' to add the timestamp to it's output.
Luckily for me, 'xxd' is not very complex.)


Comment 13 Jens Petersen 2005-05-30 08:10:41 UTC
Comment 9 should be taken literally: s/aclocal/aclocal-1.7/, ie:

- aclocal
+ aclocal-1.7

But anyway I don't think this is fixed in expect-5.43.0...

Comment 14 Jens Petersen 2005-05-31 01:02:41 UTC
In reply to comment 12, I suspected as much but I confirmed with
author that the new unbuffer script needs expect-5.43.0 to run.

Comment 15 Jens Petersen 2005-05-31 02:28:09 UTC
The new unbuffer seems to fix the second example in comment 0.
I added a patch to that effect in expect-5.43.0-2.
Could you please test that when it appears in FC devel?  Thanks.

Comment 16 Charles Sullivan 2005-06-07 21:07:11 UTC
Sorry for the delay in responding.

I'm still getting the same errors as in comment #5 with either the i386 or src 
rpms.  Am I doing the right things with the src rpm?  I.e.,
  # rpm -Uvh expect-5.43.0-2.src.rpm
  # cd /usr/src/redhat/SOURCES
  # patch -p0 <expect-5.43.0-cfg-setpgrp.patch
  # patch -p0 <unbuffer-child-flush-143963.patch
  # ./configure

I don't understand what it's telling me about the Tcl configuration definitions
(and in fact don't understand what Tcl is all about anyway).
  # rpm -q tcl
  tcl-8.4.9-1_3.rhfc2.at


    

Comment 17 Jens Petersen 2005-06-07 23:05:18 UTC
How about "rpmbuild --rebuild expect-5.43.0-2.src.rpm"? :)

Comment 18 Jens Petersen 2005-06-07 23:33:05 UTC
Or rather this for fc2:

# rpm -i expect-5.43.0-2.src.rpm
# cd /usr/src/redhat/SPECS
# sed -i -e "s/aclocal/aclocal-1.7/" expect.spec
$ rpmbuild -bb expect.spec


Comment 19 Charles Sullivan 2005-06-08 02:37:50 UTC
Ahah!  I followed your instructions in comment #18 and everything now works - 
the build, the installation of the resulting i386 rpm, and finally unbuffer 
itself!  Thanks very much for your persistance in solving this problem Jens.

The fact that 'unbuffer' doesn't seem to have worked from at least as early as 
Red Hat 9 leads me to believe it isn't that important for most users.  But I'm 
surprised that something as basic as user control of buffering in pipelines 
isn't a shell built-in.

Comment 20 Jens Petersen 2005-06-08 05:43:39 UTC
Thanks for your persistence in testing too and alerting to the build problem. :)