Bug 1009695

Summary: On agent/plugin-container shutdown an error is reported indicating Failed to remove poller with PollerKey
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: Plugin ContainerAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Sunil Kondkar <skondkar>
Severity: low Docs Contact:
Priority: unspecified    
Version: JON 3.1.2, JON 3.2CC: jshaughn, skondkar
Target Milestone: ER03   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 13:59:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Larry O'Leary 2013-09-18 21:40:10 UTC
Description of problem:
When the agent is shutdown or the agent's plug-in container is restarted while there are event sources being monitored (polled) the removal of registered event pollers fails.

Version-Release number of selected component (if applicable):
4.7.0.JON [adad71f]

How reproducible:
Always

Steps to Reproduce:
1. Start JBoss ON system.
2. Import RHQ Server resource into inventory.
3. Add an event source to the RHQ Server resource.
4. Verify events are being captured.
5. Shutdown agent.

Actual results:
Agent will log ERRORs for each event source that is active:
ERROR [ResourceContainer.invoker.daemon-14] (rhq.core.pc.event.EventManager)- Failed to remove poller with PollerKey[resourceId=10003, eventType=logEntry] from thread pool.

Expected results:
No errors appears on shutdown of agent or restart of plug-in container.

Additional info:
This issue appears to have no functional impact to the system.


(04:17:10 PM) jsanda: *now* i think i know the cause
(04:19:22 PM) jsanda: during the PC shutdown
(04:19:43 PM) jsanda: EventManager.shutdown is invoked before the resource components are stopped
(04:20:07 PM) loleary: Interesting.
(04:20:11 PM) loleary: I see where that is going.
(04:20:27 PM) jsanda: and EventManager.shutdown shuts down the thread pool
(04:20:45 PM) jsanda: and then subsequently, the resource components try to unregister their pollers
(04:21:09 PM) jsanda: while the error msg is misleading, i think it is actually harmless
(04:21:16 PM) jsanda: with that said, it should be fixed
(04:21:39 PM) loleary: Agreed.
(04:22:07 PM) jsanda: EventManager.unregisterEventPoller() should be a no-op if pollerThreadPool.isTerminated() returns true
(04:22:31 PM) jsanda: that call to isTerminated needs to be added and then if anything maybe just log a debug msg
(04:22:49 PM) jsanda: sorry, isShutdown(), not isTerminated
(04:23:37 PM) jsanda: i verified that this is in fact happening by setting breakpoints

Comment 1 Jay Shaughnessy 2014-09-04 18:55:25 UTC
Master commit 4d2ec5e773ff94879d11a7e2fb2e395220b2183e
Author: Jay Shaughnessy <jshaughn>
Date:   Thu Sep 4 14:52:06 2014 -0400

    Don't try to unregister the poller shutdown if the threadpool is shutdown.



Release/jon3.3.x commit c17e15132f0e49b745e2d9d3383f76c9a68bbebd
Author: Jay Shaughnessy <jshaughn>
Date:   Thu Sep 4 14:52:06 2014 -0400

    (cherry picked from commit 4d2ec5e773ff94879d11a7e2fb2e395220b2183e)
    Signed-off-by: Jay Shaughnessy <jshaughn>

Comment 2 Simeon Pinder 2014-09-17 02:49:38 UTC
Moving to ON_QA as available for test with the following brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=385149

Comment 3 Sunil Kondkar 2014-09-18 15:24:09 UTC
Verified on version : 3.3.0.ER03 build Number :4aefe39:44e33a4