744803 – NoopProfile when Inventory EAP 5.1.1

Bug 744803 - NoopProfile when Inventory EAP 5.1.1

Summary: NoopProfile when Inventory EAP 5.1.1

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	RHQ Project
Classification:	Other
Component:	Agent
Sub Component:
Version:	3.0.0 Beta1
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Robert Buck
QA Contact:	Mike Foley
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	jon30-sprint8 751873
TreeView+	depends on / blocked

Reported:	2011-10-10 14:21 UTC by Douglas Woods
Modified:	2012-02-07 19:32 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	751873 (view as bug list)
Environment:
Last Closed:	2011-10-13 14:56:43 UTC
Embargoed:

Attachments	(Terms of Use)

Description Douglas Woods 2011-10-10 14:21:22 UTC

Description of problem:
Environment: EAP v5.1.1 default profile
             RHEL v6.1 Server
             SUN JDK 1.6.0_26
             JON v3.0.0 Beta
             Oracle Express Edition 11g R2

Version-Release number of selected component (if applicable):
v3.0.0 Beta

How reproducible:
Inventory an EAP v5.1.1 server

Steps to Reproduce:
1. Boot up EAP default profile
2. Inventory Server in #1 with JON GUI
3. 
  
Actual results:
server.log attached for ERRORS.
2011-10-10 10:07:56,813 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[127.0.0.1:39420]) WorkerThread#1[127.0.0.1:39420] failed
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile

Expected results:
No Errors in server.log

Additional info:

Comment 1 Mike Foley 2011-10-12 14:43:14 UTC

qa please test with EAP 5.1.0

Comment 2 Mike Foley 2011-10-13 14:56:43 UTC

cannot reproduce with EAP 5.1 (the supported version)

Comment 3 Douglas Woods 2011-10-13 15:23:33 UTC

Tested with EAP v5.1.0 default profile, same results with Workerthread errors in the server.log. This message does not appear in the server.log until after the server has been inventoried into the JON repository.

011-10-13 11:12:31,099 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (main) JBoss (Microcontainer) [5.1.0 (build: SVNTag=JBPAPP_5_1_0 date=201006282158)] Started in 17s:95ms
2011-10-13 11:15:58,718 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[127.0.0.1:54611]) WorkerThread#1[127.0.0.1:54611] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile

Comment 4 Douglas Woods 2011-10-13 15:28:43 UTC

Tested with same EAP v5.1.0 default profile and JON v2.4.1. No issues.

Comment 5 Mike Foley 2011-10-13 18:30:36 UTC

correction ... i am seeing this behavior with EAP v5.1 (observed while verifying https://bugzilla.redhat.com/show_bug.cgi?id=711502)

Comment 6 Robert Buck 2011-11-07 11:57:59 UTC

I am seeing this too while trying to reproduce other JBoss bugs; clustered setup on a single machine, EAP5.1.

java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile

Comment 7 Robert Buck 2011-11-07 13:20:48 UTC

As a note, the minute JON (apparently) contacts EAP the errors start spewing out by the reams. Also, undeployment of ANY wars that point forward is not possible. Even if you delete the wars from the file system, JBoss will not unload them from memory. This is my experience, and I wonder if the two are related. This may want to be classified as URGENT, and a show-stopper for JON 3.0.

Comment 8 Charles Crouch 2011-11-13 03:46:46 UTC

Bob, whats the status of this? I thought a change had already gone in to remove 
an unnecessary check which tried to lookup a profile name?

Comment 9 Robert Buck 2011-11-14 14:45:57 UTC

commit 8d5f78b194c2e957f2d5778e3a4ead1fdd98713c
Author: Robert Buck <rbuck>
Date:   2011-11-07 16:14:45 -0500

    [BZ 751873] Prevent JBoss log spew and undeploy issues; w/o this fix after a bundle is deployed to an EAP cluster and a war is updated, it is no longer possible to undeploy via rm -fr on the command line.

Comment 10 Robert Buck 2011-11-14 17:30:45 UTC

This still happens, here is the test scenario.

Start up an RHQ server, a local agent (prompt).
Start up JBoss (any profile, I duped all to n01 and used that -c n01)
At the agent prompt run 'discovery --full'
Notice the JBoss logs and prompt - about 109 log errors with full stack traces.

********************************

12:23:58,230 ERROR [ServerThread] WorkerThread#1[127.0.0.1:54748] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

12:24:04,255 ERROR [ServerThread] WorkerThread#1[127.0.0.1:55142] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)

******************************************************

Comment 11 Robert Buck 2011-11-15 02:37:30 UTC

Its down to these lines of code in InventoryManager:

ResourceDiscoveryComponent proxy = this.discoveryComponentProxyFactory
   .getDiscoveryComponentProxy(
      context.getResourceType(),
      component, timeout, parentResourceContainer);
Set<DiscoveredResourceDetails> results = proxy.discoverResources(context);

Comment 12 Robert Buck 2011-11-15 02:52:30 UTC

This call is problematic:

  Set<DiscoveredResourceDetails> results = proxy.discoverResources(context);

Comment 13 Robert Buck 2011-11-15 03:33:05 UTC

And more specifically it is somewhere in the following line of code in AbstractManagedDeploymentDiscoveryComponent.discoverResources:

  String resourceKey = buildResourceKey(managedDeployment, profileService);

Comment 14 Robert Buck 2011-11-15 03:51:43 UTC

This call:

   profile = profileService.getActiveProfile(profileKey);

produces:

22:39:54,003 ERROR [ServerThread] WorkerThread#0[127.0.0.1:21600] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile

N.b. following that, from this call:

   deploymentManager.loadProfile(profileKey);

I see lots of these:

   java.lang.SecurityException: Authentication exception, principal=null

Of course, what does the code do? It eats the exceptions, not good:

   ...
   } catch (Exception e1) {
     continue;
   }

The profileKey, above, that this all chokes on in my case:

ProfileKey@12f6ce8[domain=default, server=default, name=n01]

On subsequent calls we see this key:

   ProfileKey@ecc6f4[domain=default, server=default, name=farm]

which results in all of the other stack traces filling the logs:

22:48:49,391 ERROR [ServerThread] WorkerThread#0[127.0.0.1:22195] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile

on the following line of code:

   profile = profileService.getActiveProfile(profileKey);

and the cause:

   java.lang.reflect.UndeclaredThrowableException

Comment 15 Robert Buck 2011-11-15 04:02:09 UTC

From what I am reading online, this is at least one JBoss issue, perhaps two issues. But I have concluded these are certainly not in RHQ.

Comment 16 Robert Buck 2011-11-15 04:05:18 UTC

n.b. the UndeclaredThrowableException is probably the crux to the whole problem; the JBoss folks are throwing checked exceptions across a proxy interface for which there are no declared throws (real bad).

Comment 17 Robert Buck 2011-11-16 14:39:46 UTC

I tried this on the release_jon3.x branch and the calls above are no longer there as the code has been removed; however, the RHQ_4_2_0_JON300_CR1 tag occurs prior to the change so CR1 will not have it.

Comment 18 Mike Foley 2011-11-16 19:41:20 UTC

verified jon3 branch

Comment 19 Robert Buck 2011-11-17 21:07:54 UTC

See https://issues.jboss.org/browse/JBPAPP-7545

Comment 20 Alan Evans 2011-12-14 07:41:36 UTC

I can further confirm this bug, I am experiencing the problem exactly as described.

How to reproduce:
  Start JBoss
  Start rhq-agent or force a discovery "discovery -f" from the console

I am using:
JBOSS
JBoss [EAP] 5.1.0 (build: SVNTag=JBPAPP_5_1_0 date=201009150028) from zip

RHQ AGENT
RHQ 4.2.0 [6746535] (Sat Oct 29 18:54:07 GMT+00:00 2011)
> version
      Build-Date: Sat Oct 29 18:54:07 GMT+00:00 2011
       Build-Jdk: 1.6.0_24
Build-Jdk-Vendor: Sun Microsystems Inc.
    Build-Number: 6746535
   Build-OS-Name: Linux
Build-OS-Version: 2.6.32-71.el6.i686
     Module-Name: RHQ Enterprise Agent
  Module-Version: 4.2.0
    Product-Name: RHQ
 Product-Version: 4.2.0

Relevant logs: (I'll post full logs to JBAPP-7545)

22:36:33,790 ERROR [ServerThread] WorkerThread#1[10.10.10.22:53381] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:33,817 ERROR [ServerThread] WorkerThread#0[10.10.10.22:53384] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:33,882 INFO  [DeployHandler] Begin start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/admin-console.war/]
22:36:33,884 INFO  [TomcatDeployment] undeploy, ctxPath=/admin-console
22:36:34,211 INFO  [TomcatDeployment] deploy, ctxPath=/admin-console
Dec 13, 2011 10:36:34 PM com.sun.faces.config.ConfigureListener contextInitialized
INFO: Initializing Mojarra (1.2_13-b01-FCS) for context '/admin-console'
22:36:34,284 INFO  [config] Initializing Mojarra (1.2_13-b01-FCS) for context '/admin-console'
22:36:35,729 INFO  [DeployHandler] End start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/admin-console.war/]
22:36:35,878 ERROR [ServerThread] WorkerThread#2[10.10.10.22:53395] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:35,932 ERROR [ServerThread] WorkerThread#0[10.10.10.22:53399] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:35,972 INFO  [DeployHandler] Begin start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/jmx-console.war/]
22:36:35,974 INFO  [TomcatDeployment] undeploy, ctxPath=/jmx-console
22:36:36,049 INFO  [TomcatDeployment] deploy, ctxPath=/jmx-console
22:36:36,073 INFO  [DeployHandler] End start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/jmx-console.war/]

Regards,
-Alan

Comment 21 Mike Foley 2012-02-07 19:32:13 UTC

changing status of VERIFIED BZs for JON 2.4.2 and JON 3.0 to CLOSED/CURRENTRELEASE

Comment 22 Mike Foley 2012-02-07 19:32:14 UTC

marking VERIFIED BZs to CLOSED/CURRENTRELEASE

Note You need to log in before you can comment on or make changes to this bug.