Created attachment 809896 [details]
Screenshot showing bigger number of events on the web than in the log files. It was gained byt the same way as it is in the description.

Description of problem:
On the web is reported more events from the source event log files than they really contain.

Version-Release number of selected component (if applicable):
Version: 3.2.0.ER2
Build Number: 9bf6f76:371eac0

How reproducible:
always

Steps to Reproduce:
1. Start JBoss ON 3.2 system.
2. Import RHQ Agent and RHQ Server into inventory.
3. Modify RHQ Agent resource configuration Event Report Max Per Source and Event Report Max Total to 1000 and 10000.
4. Add 11 event logging sources to RHQ Server resource using generated log files /tmp/file-00.log, /tmp/file-01.log, /tmp/file-02.log, ..., /tmp/file-10.log
5. Restart agent with DEBUG logging enabled.
6. Create /tmp/generateLog using the script below.
7. Then run /tmp/generateLog to generate our random log data:
    
    a) chmod +x /tmp/generateLog
    b) for i in $(seq -w 0 10); do /tmp/generateLog /tmp/file-${i}.log 2>/dev/null & done

8. Then click on Inventory, choose Servers - Top Level Imports under Resources. Then click on EAP, choose its tab Events and have this page opened.
9. Go to /tmp folder of your machine and after some time can be found out that number of events on the web is bigger than in the log files.
For checking it use in /tmp folder command:

egrep -c "(DEBUG|INFO|WARN|ERROR|FATAL)" file-* | cut -d ":" -f2 | awk '{ sum += $1 } END { print sum }'

and immediately after that click on Refresh button on displayed web page (from step 8). This can be seen in the attached screenshot.


File: /tmp/generateLog
------------------------------------------------------------------------
#!/bin/sh
#

maxSleepTimePerSleep=2
maxEntriesPerSleep=5
maxEntries=1000000
maxSleepTime=3600





logLevels=("FATAL"
"ERROR"
"WARN "
"INFO "
"DEBUG"
"TRACE")

logThreads=(main
$(echo Thread-{0..99})
$(echo Fake-{0..9}))

logCategories=(apache.cassandra.db.commitlog.CommitLogReplayer
org.jboss.as.server.deployment 
org.jboss.as.messaging 
org.some.other.category)

logMessages=("A log message"
"Another log message"
"A log message with some meaning"
"A log message without any meaning"
"To be a log message or not to be"
"To infinity and beyond"
"Every rose has its throne"
"Come out and pay"
"That thing wasn't supposed to do that."
"Now that's what I'm talking about"'!'
"Friends don't let friends eat grapefruit")

function printRandomLogLevel() {
    numLogLevels=${#logLevels[*]}
    echo -n ${logLevels[$((RANDOM%numLogLevels))]}
}

function printRandomLogThread() {
    numLogThreads=${#logThreads[*]}
    echo -n ${logThreads[$((RANDOM%numLogThreads))]}
}

function printRandomLogCategory() {
    numLogCategories=${#logCategories[*]}
    echo -n ${logCategories[$((RANDOM%numLogCategories))]}
}

function printRandomLogMessage() {
    numLogMessages=${#logMessages[*]}
    echo -n ${logMessages[$((RANDOM%numLogMessages))]}
}


function printLogTimestamp() {
    timestamp=$(date +"%F %T|%N")
    ms="$(printf "%03d" $(( 10#${timestamp#*|}/1000000 )))"
    echo -n "${timestamp%|*},${ms}"
}

function printRandomLogEntry() {
    echo "$(printLogTimestamp) $(printRandomLogLevel) [$(printRandomLogThread)] ($(printRandomLogCategory)) $(printRandomLogMessage)"
}

function printRandomLogEntries() {
    totalSleepTime=0
    totalEntries=0

    while [ true ]; do
        randomSleep=$((RANDOM%${maxSleepTimePerSleep}))
        [ $((${maxSleepTime}-(${totalSleepTime}+${randomSleep}))) -lt 0 ] && randomSleep=$((${randomSleep} +(${maxSleepTime}-(${totalSleepTime}+${randomSleep}))))
        echo >&2 Sleeping for ${randomSleep} seconds...
        sleep ${randomSleep}s
        totalSleepTime=$((totalSleepTime+$randomSleep))
        randomNumEntries=$((RANDOM%${maxEntriesPerSleep}))
        [ $((${maxEntries}-(${totalEntries}+${randomNumEntries}))) -lt 0 ] && randomNumEntries=$((${randomNumEntries} +(${maxEntries}-(${totalEntries}+${randomNumEntries}))))
        echo >&2 Printing ${randomNumEntries} log entries...
        for ((i=0; i<randomNumEntries; i++)); do
            [ $# -gt 0 ] && {
                echo $(printRandomLogEntry) >>"$1"
            } || {
                echo $(printRandomLogEntry)
            }
        done
        totalEntries=$((totalEntries+${randomNumEntries}))
        [ ${totalEntries} -ge ${maxEntries} -o ${totalSleepTime} -ge ${maxSleepTime} ] && break
    done
    echo >&2 Total entries is ${totalEntries} and total sleep time was ${totalSleepTime} seconds.
}

printRandomLogEntries $@
------------------------------------------------------------------------

Actual results:
From certain time there is more events on the web than in the log files.

Expected results:
Number of events on the web should be always <= than in the log files.

Additional info:
For imagine the number of events on the web in my case was 157 370 and in all the the log files 131 074. This was found out when all the processes generating events into the log files finished by themselves.