Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 618910 - /usr/bin/lesspipe.sh performance on huge files
/usr/bin/lesspipe.sh performance on huge files
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: file (Show other bugs)
5.4
All Linux
low Severity medium
: rc
: ---
Assigned To: Jan Kaluža
BaseOS QE Security Team
: Reopened, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-27 23:29 EDT by Jason Haar
Modified: 2012-10-29 09:00 EDT (History)
4 users (show)

See Also:
Fixed In Version: file-4.17-17
Doc Type: Bug Fix
Doc Text:
Cause: When detecting file from stdin using "file -", File tried to read whole file from stdin to be able to get additional information about ELF binary even when the file on stdin wasn't ELF binary. Consequence: File read the whole file from stdin into memory. This wasted memory for bigger files and took lot of time. Fix: File detects if the file is ELF binary using the ELF magic bytes and reads whole file into memory only when it is sure it is real ELF binary. Result: Detection of bigger files is faster and does not waste the memory.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-02-21 00:46:11 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch to speed lesspipe up for large files (1.56 KB, patch)
2010-07-27 23:29 EDT, Jason Haar
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0201 normal SHIPPED_LIVE file bug fix update 2012-02-20 09:53:52 EST

  None (edit)
Description Jason Haar 2010-07-27 23:29:24 EDT
Created attachment 434897 [details]
patch to speed lesspipe up for large files

Description of problem:

When I try to run less on previous /var/log/ files, lesspipe.sh mistakes .1, .2, etc for evidence for manpages, and therefore pipes the file through "file -" to detect the type.

As logfiles can be multi-gigabyte in size, this means less hangs for many minutes (and swallows RAM ,etc) before file returns

Version-Release number of selected component (if applicable):


How reproducible:

always.

Steps to Reproduce:
1. less /path/really.big.file.1
2. wait
3. wait, etc
  
Actual results:

hangs for ages - longer for larger files

Expected results:

should be much faster

Additional info:

Here's a suggested patch. Seems to work

BTW, this is a general issue - lesspipe.sh is the same on FC, RHE4 and RHE5
Comment 1 Jeff Raber 2010-07-27 23:52:41 EDT
Thank you for your bug report.
We are sorry, but the Fedora Project no longer maintains this version of Fedora. Please upgrade to the latest version and reopen this bug against that version if this bug exists there.
As a result we are setting this bug to CLOSED:WONTFIX



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 2 Jason Haar 2010-07-27 23:58:43 EDT
Umm, alright, I'll open it against RHE4 or RHE5 then?
Comment 3 RHEL Product and Program Management 2011-01-11 14:57:17 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 4 RHEL Product and Program Management 2011-01-11 17:57:23 EST
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 5 RHEL Product and Program Management 2011-05-31 09:10:16 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 6 Vojtech Vitek 2011-09-14 16:04:10 EDT
Thank you for the patch, it is actually nice work-around.

However, I don't think this is a bug in lesspipe.sh script. Imho, this issue should be fixed straight in file(1) program, as it is hanging up your system.

I have these time stats with text files in current Fedora:
> $ du -h /tmp/big /tmp/small
> 5.0G    /tmp/big
> 72K     /tmp/small
> 
> $ time file /tmp/big 
> /tmp/big: ASCII English text, with very long lines
> real    0m0.187s
> user    0m0.087s
> sys     0m0.005s
> 
> $ time file /tmp/small 
> /tmp/small: ASCII English text, with very long lines
> real    0m0.035s
> user    0m0.023s
> sys     0m0.005s
I can't see any long delay anymore..

---

Let's see, what can `file' component owner say..
Comment 7 Jan Kaluža 2011-09-15 04:30:15 EDT
It's file bug in RHEL5. I'm able to reproduce it.

Problem is that if File tries to check file from stdin, its ELF reader reads whole file to be able to seek over it. It does it without checking if it's really ELF binary. It's fixed upstream and also in RHEL6/Fedora.

Following readelf.c patch can be backported to address this issue:
https://github.com/glensc/file/commit/071367c845c02f194f8374713a34eb6bd86fad1b#diff-21
Comment 13 Jan Kaluža 2012-01-05 05:07:12 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: When detecting file from stdin using "file -", File tried to read whole file from stdin to be able to get additional information about ELF binary even when the file on stdin wasn't ELF binary.

Consequence: File read the whole file from stdin into memory. This wasted memory for bigger files and took lot of time.

Fix: File detects if the file is ELF binary using the ELF magic bytes and reads whole file into memory only when it is sure it is real ELF binary.

Result: Detection of bigger files is faster and does not waste the memory.
Comment 14 errata-xmlrpc 2012-02-21 00:46:11 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0201.html

Note You need to log in before you can comment on or make changes to this bug.