Bug 744803 - NoopProfile when Inventory EAP 5.1.1
NoopProfile when Inventory EAP 5.1.1
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Agent (Show other bugs)
3.0.0 Beta1
Unspecified Unspecified
urgent Severity unspecified (vote)
: ---
: ---
Assigned To: Robert Buck
Mike Foley
: Reopened
Depends On:
Blocks: jon30-sprint8 751873
  Show dependency treegraph
 
Reported: 2011-10-10 10:21 EDT by Douglas Woods
Modified: 2012-02-07 14:32 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 751873 (view as bug list)
Environment:
Last Closed: 2011-10-13 10:56:43 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Douglas Woods 2011-10-10 10:21:22 EDT
Description of problem:
Environment: EAP v5.1.1 default profile
             RHEL v6.1 Server
             SUN JDK 1.6.0_26
             JON v3.0.0 Beta
             Oracle Express Edition 11g R2

Version-Release number of selected component (if applicable):
v3.0.0 Beta

How reproducible:
Inventory an EAP v5.1.1 server

Steps to Reproduce:
1. Boot up EAP default profile
2. Inventory Server in #1 with JON GUI
3. 
  
Actual results:
server.log attached for ERRORS.
2011-10-10 10:07:56,813 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[127.0.0.1:39420]) WorkerThread#1[127.0.0.1:39420] failed
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile

Expected results:
No Errors in server.log

Additional info:
Comment 1 Mike Foley 2011-10-12 10:43:14 EDT
qa please test with EAP 5.1.0
Comment 2 Mike Foley 2011-10-13 10:56:43 EDT
cannot reproduce with EAP 5.1 (the supported version)
Comment 3 Douglas Woods 2011-10-13 11:23:33 EDT
Tested with EAP v5.1.0 default profile, same results with Workerthread errors in the server.log. This message does not appear in the server.log until after the server has been inventoried into the JON repository.

011-10-13 11:12:31,099 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (main) JBoss (Microcontainer) [5.1.0 (build: SVNTag=JBPAPP_5_1_0 date=201006282158)] Started in 17s:95ms
2011-10-13 11:15:58,718 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[127.0.0.1:54611]) WorkerThread#1[127.0.0.1:54611] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
Comment 4 Douglas Woods 2011-10-13 11:28:43 EDT
Tested with same EAP v5.1.0 default profile and JON v2.4.1. No issues.
Comment 5 Mike Foley 2011-10-13 14:30:36 EDT
correction ... i am seeing this behavior with EAP v5.1 (observed while verifying https://bugzilla.redhat.com/show_bug.cgi?id=711502)
Comment 6 Robert Buck 2011-11-07 06:57:59 EST
I am seeing this too while trying to reproduce other JBoss bugs; clustered setup on a single machine, EAP5.1.

java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
Comment 7 Robert Buck 2011-11-07 08:20:48 EST
As a note, the minute JON (apparently) contacts EAP the errors start spewing out by the reams. Also, undeployment of ANY wars that point forward is not possible. Even if you delete the wars from the file system, JBoss will not unload them from memory. This is my experience, and I wonder if the two are related. This may want to be classified as URGENT, and a show-stopper for JON 3.0.
Comment 8 Charles Crouch 2011-11-12 22:46:46 EST
Bob, whats the status of this? I thought a change had already gone in to remove 
an unnecessary check which tried to lookup a profile name?
Comment 9 Robert Buck 2011-11-14 09:45:57 EST
commit 8d5f78b194c2e957f2d5778e3a4ead1fdd98713c
Author: Robert Buck <rbuck@redhat.com>
Date:   2011-11-07 16:14:45 -0500

    [BZ 751873] Prevent JBoss log spew and undeploy issues; w/o this fix after a bundle is deployed to an EAP cluster and a war is updated, it is no longer possible to undeploy via rm -fr on the command line.
Comment 10 Robert Buck 2011-11-14 12:30:45 EST
This still happens, here is the test scenario.

Start up an RHQ server, a local agent (prompt).
Start up JBoss (any profile, I duped all to n01 and used that -c n01)
At the agent prompt run 'discovery --full'
Notice the JBoss logs and prompt - about 109 log errors with full stack traces.

********************************

12:23:58,230 ERROR [ServerThread] WorkerThread#1[127.0.0.1:54748] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

12:24:04,255 ERROR [ServerThread] WorkerThread#1[127.0.0.1:55142] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)

******************************************************
Comment 11 Robert Buck 2011-11-14 21:37:30 EST
Its down to these lines of code in InventoryManager:

ResourceDiscoveryComponent proxy = this.discoveryComponentProxyFactory
   .getDiscoveryComponentProxy(
      context.getResourceType(),
      component, timeout, parentResourceContainer);
Set<DiscoveredResourceDetails> results = proxy.discoverResources(context);
Comment 12 Robert Buck 2011-11-14 21:52:30 EST
This call is problematic:

  Set<DiscoveredResourceDetails> results = proxy.discoverResources(context);
Comment 13 Robert Buck 2011-11-14 22:33:05 EST
And more specifically it is somewhere in the following line of code in AbstractManagedDeploymentDiscoveryComponent.discoverResources:

  String resourceKey = buildResourceKey(managedDeployment, profileService);
Comment 14 Robert Buck 2011-11-14 22:51:43 EST
This call:

   profile = profileService.getActiveProfile(profileKey);

produces:

22:39:54,003 ERROR [ServerThread] WorkerThread#0[127.0.0.1:21600] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile

N.b. following that, from this call:

   deploymentManager.loadProfile(profileKey);

I see lots of these:

   java.lang.SecurityException: Authentication exception, principal=null

Of course, what does the code do? It eats the exceptions, not good:

   ...
   } catch (Exception e1) {
     continue;
   }

The profileKey, above, that this all chokes on in my case:

ProfileKey@12f6ce8[domain=default, server=default, name=n01]

On subsequent calls we see this key:

   ProfileKey@ecc6f4[domain=default, server=default, name=farm]

which results in all of the other stack traces filling the logs:

22:48:49,391 ERROR [ServerThread] WorkerThread#0[127.0.0.1:22195] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile

on the following line of code:

   profile = profileService.getActiveProfile(profileKey);

and the cause:

   java.lang.reflect.UndeclaredThrowableException
Comment 15 Robert Buck 2011-11-14 23:02:09 EST
From what I am reading online, this is at least one JBoss issue, perhaps two issues. But I have concluded these are certainly not in RHQ.
Comment 16 Robert Buck 2011-11-14 23:05:18 EST
n.b. the UndeclaredThrowableException is probably the crux to the whole problem; the JBoss folks are throwing checked exceptions across a proxy interface for which there are no declared throws (real bad).
Comment 17 Robert Buck 2011-11-16 09:39:46 EST
I tried this on the release_jon3.x branch and the calls above are no longer there as the code has been removed; however, the RHQ_4_2_0_JON300_CR1 tag occurs prior to the change so CR1 will not have it.
Comment 18 Mike Foley 2011-11-16 14:41:20 EST
verified jon3 branch
Comment 19 Robert Buck 2011-11-17 16:07:54 EST
See https://issues.jboss.org/browse/JBPAPP-7545
Comment 20 Alan Evans 2011-12-14 02:41:36 EST
I can further confirm this bug, I am experiencing the problem exactly as described.

How to reproduce:
  Start JBoss
  Start rhq-agent or force a discovery "discovery -f" from the console

I am using:
JBOSS
JBoss [EAP] 5.1.0 (build: SVNTag=JBPAPP_5_1_0 date=201009150028) from zip

RHQ AGENT
RHQ 4.2.0 [6746535] (Sat Oct 29 18:54:07 GMT+00:00 2011)
> version
      Build-Date: Sat Oct 29 18:54:07 GMT+00:00 2011
       Build-Jdk: 1.6.0_24
Build-Jdk-Vendor: Sun Microsystems Inc.
    Build-Number: 6746535
   Build-OS-Name: Linux
Build-OS-Version: 2.6.32-71.el6.i686
     Module-Name: RHQ Enterprise Agent
  Module-Version: 4.2.0
    Product-Name: RHQ
 Product-Version: 4.2.0

Relevant logs: (I'll post full logs to JBAPP-7545)

22:36:33,790 ERROR [ServerThread] WorkerThread#1[10.10.10.22:53381] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:33,817 ERROR [ServerThread] WorkerThread#0[10.10.10.22:53384] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:33,882 INFO  [DeployHandler] Begin start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/admin-console.war/]
22:36:33,884 INFO  [TomcatDeployment] undeploy, ctxPath=/admin-console
22:36:34,211 INFO  [TomcatDeployment] deploy, ctxPath=/admin-console
Dec 13, 2011 10:36:34 PM com.sun.faces.config.ConfigureListener contextInitialized
INFO: Initializing Mojarra (1.2_13-b01-FCS) for context '/admin-console'
22:36:34,284 INFO  [config] Initializing Mojarra (1.2_13-b01-FCS) for context '/admin-console'
22:36:35,729 INFO  [DeployHandler] End start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/admin-console.war/]
22:36:35,878 ERROR [ServerThread] WorkerThread#2[10.10.10.22:53395] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.NoopProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:35,932 ERROR [ServerThread] WorkerThread#0[10.10.10.22:53399] exception occurred during first invocation
java.io.NotSerializableException: org.jboss.system.server.profile.repository.AbstractProfile
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.aop.joinpoint.InvocationResponse.writeExternal(InvocationResponse.java:100)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
        at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
        at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
        at org.jboss.invocation.unified.marshall.InvocationMarshaller.write(InvocationMarshaller.java:75)
        at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:1072)
        at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:830)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
22:36:35,972 INFO  [DeployHandler] Begin start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/jmx-console.war/]
22:36:35,974 INFO  [TomcatDeployment] undeploy, ctxPath=/jmx-console
22:36:36,049 INFO  [TomcatDeployment] deploy, ctxPath=/jmx-console
22:36:36,073 INFO  [DeployHandler] End start, [vfsfile:/opt/jboss-eap-5.1.0/jboss-as/server/default/deploy/jmx-console.war/]

Regards,
-Alan
Comment 21 Mike Foley 2012-02-07 14:32:13 EST
changing status of VERIFIED BZs for JON 2.4.2 and JON 3.0 to CLOSED/CURRENTRELEASE
Comment 22 Mike Foley 2012-02-07 14:32:14 EST
marking VERIFIED BZs to CLOSED/CURRENTRELEASE

Note You need to log in before you can comment on or make changes to this bug.