Bug 846082

Summary: Log4j log parsing not thread safe
Product: [Other] RHQ Project Reporter: Lukas Krejci <lkrejci>
Component: EventsAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.5CC: hrupp
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-11 07:26:50 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1006619    

Description Lukas Krejci 2012-08-06 15:00:18 EDT
Description of problem:
If there are multiple resources on the same agent parsing the log4j log files and producing events from them, some of them would randomly stop collecting the events for no apparent reason.

The reason why they fail is that Log4JLogEntryProcessor class in the plugin-api module uses shared (static) DateFormat instances to parse different supported date strings in the log files. The DateFormat is not thread-safe and would randomly fail if multiple simultaneous parse() calls were made on them.

Version-Release number of selected component (if applicable):
4.5.0-SNAPSHOT

How reproducible:
always

Steps to Reproduce:
1. Setup multiple resources on the same agent for parsing log4j logs
2. Make the managed resources write something into the log files
3. Wait a bit for the agent to generate events from the log file entries.
  
Actual results:
Randomly, some of the resources will no longer report new events even though the underlying log files change.

Expected results:
The event generation works as expected.

Additional info:
This bug has been fixed in a separate branch with the work on the hadoop plugin by the following commits:

feature/hadoop-plugin http://git.fedorahosted.org/cgit/rhq/rhq.git/diff/?id=f719aa14b69abda856fe51e1d45db3f9b7588c85
Author: Lukas Krejci <lkrejci@redhat.com>
Date:   Mon Aug 6 20:48:13 2012 +0200

    Make the date formats used by the log4j log processor instance fields so
    that they are not shared by multiple instances that would use those formats
    from different threads.
    The DateFormat is not a thread safe class.

feature/hadoop-plugin http://git.fedorahosted.org/cgit/rhq/rhq.git/diff/?id=c82eb40efe105f8d719cbc16592add9e259dbd10
Author: Lukas Krejci <lkrejci@redhat.com>
Date:   Mon Aug 6 19:56:30 2012 +0200

    More rigorous error handling while processing the log files to prevent
    a runtime exception cancel the scheduled job for log processing for given
    resource.
Comment 1 Lukas Krejci 2013-09-11 07:26:50 EDT
This has been incorporated into mainline quite a while ago. Closing.

$ git tag --contains f719aa14b69abda856fe51e1d45db3f9b7588c85
RHQ_4_6_0                                                                       RHQ_4_6_0_JONCI                                                                 RHQ_4_7_0                                                                       RHQ_4_7_0_JONCI                                                                 RHQ_4_8_0                                                                       RHQ_4_9_0                                                                                                                                                       
$ git tag --contains c82eb40efe105f8d719cbc16592add9e259dbd10
RHQ_4_6_0                                                                       RHQ_4_6_0_JONCI                                                                 RHQ_4_7_0                                                                       RHQ_4_7_0_JONCI                                                                  RHQ_4_8_0                                                                        RHQ_4_9_0