Bug 1013134 - Improve memory management in logconv.pl
Improve memory management in logconv.pl
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base (Show other bugs)
6.5
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Rich Megginson
Sankar Ramalingam
:
: 1111164 (view as bug list)
Depends On:
Blocks: 1061410
  Show dependency treegraph
 
Reported: 2013-09-27 17:50 EDT by Rich Megginson
Modified: 2014-10-14 03:50 EDT (History)
5 users (show)

See Also:
Fixed In Version: 389-ds-base-1.2.11.15-34.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-10-14 03:50:54 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rich Megginson 2013-09-27 17:50:07 EDT
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/419

https://bugzilla.redhat.com/show_bug.cgi?id=823882 (''Red Hat Directory Server'')

{{{
RFE Request

0. Proposed title of this feature request
Feature Request: better memory management in logconv.pl

2. What is the nature and description of the request?
Feature Request: better memory management in logconv.pl

3. Why does the customer need this? (List the business requirements here)
Since with Directory Server Red Hat offer an enterprise solution, a company
should be able to expect enterprise behaviour, in this case that is that the
product won't break from regular use. This is what's happening with the use of
logconv.pl: we use it to handle an undefined amount of logs, and we expect it
to do anyhing BUT break the system and cause unavailability. Whether it be an
error message (too many logs), a better handling of the logs (i.e. in chunks)
or a better memory management, all is acceptable. For a script like this to
crash a server is unacceptable in an enterprise environmemt.

4. How would the customer like to achieve this? (List the functional
requirements here)

Customer is expecting better memory Management in logconv.pl in below ways.
- refusing to start processing logs altogether when logconv.pl calculates the
logs are too big to be processed in memory
- processing logs in sizeable chunks and later adding these processed chunks
together to do a final report
- fine-tune the memory usage of logconv.pl
- finding temporary diskspace to flush parts of memory (of course with the
option of still one of the previous three when disk space is too low)

5. For each functional requirement listed in question 4, specify how Red Hat
and the customer can test to confirm the requirement is successfully
implemented.
Create a replica setup which generate 5 GB of daily logs & try to run
logconv.pl.

6. Is there already an existing RFE upstream or in Red Hat bugzilla?
No

7. How quickly does this need resolved? (desired target release)
No time-line

8. Does this request meet the RHEL Bug and Feature Inclusion Criteria (please
review)
Yes

9. List the affected packages
389-ds-base

10. Would the customer be able to assist in testing this functionality if
implemented?
Yes
}}}

USCBP would like this and all logconv.pl enhancements not currently in 6.4 to be available in 6.5
Comment 3 Noriko Hosoi 2014-06-19 16:35:50 EDT
*** Bug 1111164 has been marked as a duplicate of this bug. ***
Comment 10 srkrishn@redhat.com 2014-08-18 07:10:37 EDT
using logconv.pl -efcibaltnxrgjuyp accesslog
     
I got the following errors:
    Use of uninitialized value $1 in string ne at /usr/bin/logconv.pl line 2135, <LOG> line 113254.
    Use of uninitialized value $1 in array element at /usr/bin/logconv.pl line 2134, <LOG> line 113256.
    Use of uninitialized value $1 in string ne at /usr/bin/logconv.pl line 2135, <LOG> line 113256.
    Use of uninitialized value $1 in hash element at /usr/bin/logconv.pl line 2146, <LOG> line 113256.
    Use of uninitialized value $1 in array element at /usr/bin/logconv.pl line 2134, <LOG> line 113258.

tested on version:1.2.11.15 38.el6
Comment 11 Rich Megginson 2014-08-18 11:53:04 EDT
The problem is with the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1103287 - moving this bug back to ON_QA, and moving 1103287 to ASSIGNED
Comment 12 srkrishn@redhat.com 2014-08-19 08:10:27 EDT
this bug has been verified:
using logconv.pl -efcibaltnxrgjuyp accesslog


----- Top 20 Most returned nentries -----

224952          nentries=1    
9254            nentries=0    
980             nentries=2    
672             nentries=3    

----- Top 20 Longest etimes -----

etime=40        14        
etime=39        42        
etime=38        14        
etime=37        28

----- Top 20 Search Filters -----

Number of Unique Search Filters: 11944

22400           (objectclass=raboperson)                
20622           (&(objectclass=raboperson)(uid=topaz_am)(activeentry=1))
6776            (objectclass=*)                         
5054            (&(objectclass=person)(cn=te*))         
2240            (&(objectclass=platformfunction)(!(activeentry=0)))

real	19m20.916s
user	17m46.122s
sys	1m30.416s

this was tested on 1.2.11.15.40
Comment 13 Sankar Ramalingam 2014-08-19 08:29:45 EDT
Based on previous comment from Sriram, I am marking the bug as Verified.
Comment 14 errata-xmlrpc 2014-10-14 03:50:54 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1385.html

Note You need to log in before you can comment on or make changes to this bug.