Bug 1145948 - allow logconv.pl -S/-E switches to work even when timestamps not present in access log
Summary: allow logconv.pl -S/-E switches to work even when timestamps not present in a...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: 389-ds-base
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Rich Megginson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1160243
TreeView+ depends on / blocked
 
Reported: 2014-09-24 07:43 UTC by Marko Myllynen
Modified: 2020-09-13 21:12 UTC (History)
5 users (show)

Fixed In Version: 389-ds-base-1.3.4.0-1.fc22
Doc Type: Bug Fix
Doc Text:
Cause: Using dates/times that are not present in the logs Consequence: The log report is empty Fix: Improve the logic for checking if a log's timestamp is within the start/end times. Result: The log report is not empty if the date range does not match a timestamp in the logs
Clone Of:
: 1160243 (view as bug list)
Environment:
Last Closed: 2015-12-02 03:55:40 UTC
Type: Bug


Attachments (Terms of Use)
access log (11.97 KB, application/x-gz)
2014-09-24 07:43 UTC, Marko Myllynen
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 1241 0 None None None 2020-09-13 21:12:38 UTC

Description Marko Myllynen 2014-09-24 07:43:35 UTC
Created attachment 940690 [details]
access log

Description of problem:
Attached is a small example access log which can be used to reproduce this, I'm seeing the same on F20 & RHEL6.

$ logconv.pl ./access - reports meaningful statistics

$ logconv.pl -S foo ./access - reports all zeroes but does not complain about the invalid time string - should probably error out if the time string is invalid as otherwise the results will be unexpected and it's hard to spot such mistakes

$ logconv.pl -S '[24/Sep/2014:10:10:00 +0300]' ./access - reports all zeroes and the printed "Start of Logs" does not match the provided time string

Using -E does not seem to have any effect.

Version-Release number of selected component (if applicable):
F20 / RHEL6

Comment 1 Marko Myllynen 2014-09-25 14:22:18 UTC
This issue prevent the ds389log PMDA in PCP working properly, see http://oss.sgi.com/pipermail/pcp/2014-September/005683.html.

Comment 2 Rich Megginson 2014-09-25 22:03:25 UTC
The given start and end time _must exist in the access log_.  For example:

$ awk '{print $1}' access | sort -u

[24/Sep/2014:10:08:09
[24/Sep/2014:10:08:10
[24/Sep/2014:10:09:06
[24/Sep/2014:10:09:07
[24/Sep/2014:10:09:08
[24/Sep/2014:10:11:07
389-Directory/1.3.2.23
f20-test.example.com:389

Using any of these times works:

$ logconv.pl -S '[24/Sep/2014:10:09:07' -E '[24/Sep/2014:10:09:08' access
----------- Access Log Output ------------

Start of Logs:    24/Sep/2014:10:09:07
End of Logs:      24/Sep/2014:10:09:08

Processed Log Time:  0 Hours, 0 Minutes, 1 Seconds

Restarts:                     0
Total Connections:            122

$ logconv.pl -E '[24/Sep/2014:10:08:10' access.bad
----------- Access Log Output ------------

Start of Logs:    24/Sep/2014:10:08:09
End of Logs:      24/Sep/2014:10:08:10

Processed Log Time:  0 Hours, 0 Minutes, 1 Seconds

Restarts:                     1
Total Connections:            235

$ logconv.pl -S '[24/Sep/2014:10:09:08'  access
----------- Access Log Output ------------

Start of Logs:    24/Sep/2014:10:09:08
End of Logs:      24/Sep/2014:10:11:07

Processed Log Time:  0 Hours, 1 Minutes, 59 Seconds

Restarts:                     0
Total Connections:            117

So I guess this should be changed into an RFE - logconv.pl -E and -S arguments: timestamp should work even when date/time not present in log

Does the PMDA functionality require being able to use dates/times not present in the log?

Comment 3 Marko Myllynen 2014-09-26 09:14:09 UTC
(In reply to Rich Megginson from comment #2)
> The given start and end time _must exist in the access log_.

Ok, I didn't see this mentioned in the man page.

> Using any of these times works:
> 
> $ logconv.pl -S '[24/Sep/2014:10:09:07' -E '[24/Sep/2014:10:09:08' access

Yes, this works, but logconv.pl(1) states:

       -S, --startTime <time to begin analyzing logfile from>
              Time  to begin analyzing logfile from E.g. [28/Mar/2002:13:14:22
              -0800]

       -E, --endTime <time to stop analyzing logfile>
              Time to stop analyzing logfile from  E.g.  [28/Mar/2002:13:24:62
              -0800]

So it is strange that

$ logconv.pl -S '[24/Sep/2014:10:09:07]' -E '[24/Sep/2014:10:09:08]' access

does not work (note the trailing ]). I think this underlines the need for logconv.pl to abort on timestamps it does not recognise.

> So I guess this should be changed into an RFE - logconv.pl -E and -S
> arguments: timestamp should work even when date/time not present in log

Ok.

> Does the PMDA functionality require being able to use dates/times not
> present in the log?

Yes, but I think this would be useful functionality also in general (i.e. request for statistics between 10:00:00 and 11:00:00 with actually checking was there those exact timestamps present in the log.

Thanks.

Comment 4 Marko Myllynen 2014-09-26 10:12:42 UTC
(In reply to Marko Myllynen from comment #3)
> 
> Yes, this works, but logconv.pl(1) states:
> 
>        -S, --startTime <time to begin analyzing logfile from>
>               Time  to begin analyzing logfile from E.g.
> [28/Mar/2002:13:14:22
>               -0800]
> 
>        -E, --endTime <time to stop analyzing logfile>
>               Time to stop analyzing logfile from  E.g. 
> [28/Mar/2002:13:24:62
>               -0800]
> 
> So it is strange that
> 
> $ logconv.pl -S '[24/Sep/2014:10:09:07]' -E '[24/Sep/2014:10:09:08]' access
> 
> does not work (note the trailing ]). I think this underlines the need for
> logconv.pl to abort on timestamps it does not recognise.

Sigh, I missed the timezone part above, that was the reason the above didn't work.

$ logconv.pl -S '[24/Sep/2014:10:09:07 +0300]' -E '[24/Sep/2014:10:09:08 +0300]' access

works as expected.

Comment 5 Rich Megginson 2014-09-26 13:30:17 UTC
(In reply to Marko Myllynen from comment #3)
> (In reply to Rich Megginson from comment #2)
> > The given start and end time _must exist in the access log_.
> 
> Ok, I didn't see this mentioned in the man page.

Because it's not there - we need to add it - the -S and -E arguments are definitely not intuitive.  Or we just need to fix -S and -E handling (see below).

> 
> > Using any of these times works:
> > 
> > $ logconv.pl -S '[24/Sep/2014:10:09:07' -E '[24/Sep/2014:10:09:08' access
> 
> Yes, this works, but logconv.pl(1) states:
> 
>        -S, --startTime <time to begin analyzing logfile from>
>               Time  to begin analyzing logfile from E.g.
> [28/Mar/2002:13:14:22
>               -0800]
> 
>        -E, --endTime <time to stop analyzing logfile>
>               Time to stop analyzing logfile from  E.g. 
> [28/Mar/2002:13:24:62
>               -0800]
> 
> So it is strange that
> 
> $ logconv.pl -S '[24/Sep/2014:10:09:07]' -E '[24/Sep/2014:10:09:08]' access
> 
> does not work (note the trailing ]). I think this underlines the need for
> logconv.pl to abort on timestamps it does not recognise.

It should be improved in this case - if there is a trailing ], it should just assume that the local current time zone is being used.

> 
> > So I guess this should be changed into an RFE - logconv.pl -E and -S
> > arguments: timestamp should work even when date/time not present in log
> 
> Ok.
> 
> > Does the PMDA functionality require being able to use dates/times not
> > present in the log?
> 
> Yes, but I think this would be useful functionality also in general (i.e.
> request for statistics between 10:00:00 and 11:00:00 with actually checking
> was there those exact timestamps present in the log.

Agreed.  The problem is that the comparison is just a simple string equality comparison - there is currently no concept of < or > in this comparison - to do so would probably require an actual date -> time_t conversion.


> 
> Thanks.

Comment 6 Rich Megginson 2014-09-26 13:38:19 UTC
Upstream ticket:
https://fedorahosted.org/389/ticket/47910

Comment 7 Fedora End Of Life 2015-05-29 12:57:08 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 8 mreynolds 2015-06-11 00:16:49 UTC
Fixed upstream.

Comment 13 Fedora End Of Life 2015-11-04 15:52:22 UTC
This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 14 Fedora End Of Life 2015-12-02 03:55:46 UTC
Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.