Bug 1293377 - Agent throw ERRORS on RHEL 7.1 openJDK 1.6.0_34
Agent throw ERRORS on RHEL 7.1 openJDK 1.6.0_34
Status: CLOSED CANTFIX
Product: JBoss Operations Network
Classification: JBoss
Component: Agent (Show other bugs)
JON 3.3.5
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Michael Burman
Prachi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-21 10:31 EST by Prachi
Modified: 2016-08-12 09:40 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-02 10:49:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
agent.log after applying CP05 (28.11 KB, text/plain)
2015-12-21 10:31 EST, Prachi
no flags Details
agent.log-CP05-ER01 (62.20 KB, text/plain)
2016-01-25 06:36 EST, Prachi
no flags Details

  None (edit)
Description Prachi 2015-12-21 10:31:07 EST
Created attachment 1108330 [details]
agent.log after applying CP05

Description of problem:

For this 1.6.0_34 specific version of openJDK the agent is not started correctly throwing in log:ERROR [main] (rhq.core.pc.plugin.PluginManager)- Plugin [JDG]



Version-Release number of selected component (if applicable):

JON3.3.5.DR01


How reproducible:


Steps to Reproduce:
1. use RHEL7.1 with openJDK1.6.0_34
2. install JON3.3.3.5
3. start it (rhqctl start)

Actual results:

*************************
ERROR [main] (rhq.core.pc.plugin.PluginManager)- Plugin [JDG] at [file:/home/hudson/rhq-agent/plugins/infinispan-server-rhq-plugin-6.1.0.Final-redhat-3.jar] could not be loaded and will therefore not be deployed.
java.lang.NullPointerException
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parseSimpleProperty(ConfigurationMetadataParser.java:255)
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parseProperty(ConfigurationMetadataParser.java:209)
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parseProperties(ConfigurationMetadataParser.java:180)
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parse(ConfigurationMetadataParser.java:109)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseResourceDescriptor(PluginMetadataParser.java:650)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseServiceDescriptor(PluginMetadataParser.java:486)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseResourceDescriptor(PluginMetadataParser.java:817)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseServiceDescriptor(PluginMetadataParser.java:486)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseDescriptor(PluginMetadataParser.java:172)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.<init>(PluginMetadataParser.java:109)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataManager.loadPlugin(PluginMetadataManager.java:186)
        at org.rhq.core.pc.plugin.PluginManager.loadPlugin(PluginManager.java:322)
        at org.rhq.core.pc.plugin.PluginManager.<init>(PluginManager.java:178)
        at org.rhq.core.pc.PluginContainer.initialize(PluginContainer.java:290)
        at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:2066)
        at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:757)
        at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:461)


 ERROR [main] (rhq.core.pc.upgrade.ResourceUpgradeDelegate)- ResourceUpgradeFacet threw an exception while upgrading resource [Resource[id=10005, uuid=05772e65-c164-4546-8ee1-b6a5db057419, type={RHQStorage}RHQ Storage Node, key=RHQ Storage Node(prachijon335.bc.jonqe.lab.eng.bos.redhat.com), name=RHQ Storage Node(prachijon335.bc.jonqe.lab.eng.bos.redhat.com), parent=prachijon335.bc.jonqe.lab.eng.bos.redhat.com, version=4.12.0.JON330GA]]
java.lang.reflect.UndeclaredThrowableException
        at com.sun.proxy.$Proxy47.upgrade(Unknown Source)
        at org.rhq.core.pc.inventory.InventoryManager.invokeDiscoveryComponentResourceUpgradeFacet(InventoryManager.java:575)
        at org.rhq.core.pc.upgrade.ResourceUpgradeDelegate.executeResourceUpgradeFacetAndStoreRequest(ResourceUpgradeDelegate.java:290)
        at org.rhq.core.pc.upgrade.ResourceUpgradeDelegate.processAndQueue(ResourceUpgradeDelegate.java:107)
        at org.rhq.core.pc.inventory.InventoryManager.activateAndUpgradeResourceRecursively(InventoryManager.java:3729)
        at org.rhq.core.pc.inventory.InventoryManager.activateAndUpgradeResourceRecursively(InventoryManager.java:3740)
        at org.rhq.core.pc.inventory.InventoryManager.activateAndUpgradeResources(InventoryManager.java:3699)
        at org.rhq.core.pc.inventory.InventoryManager.initialize(InventoryManager.java:298)
        at org.rhq.core.pc.PluginContainer.initialize(PluginContainer.java:292)
        at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:2066)
        at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:757)
        at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:461)
Caused by: java.lang.Exception: Discovery component invocation failed.
        at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:309)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:701)
Caused by: org.rhq.cassandra.util.ConfigEditorException: Failed to load ./../conf/cassandra.yaml
        at org.rhq.cassandra.util.ConfigEditor.load(ConfigEditor.java:44)
        at org.rhq.plugins.cassandra.CassandraNodeDiscoveryComponent.getDetails(CassandraNodeDiscoveryComponent.java:154)
        at org.rhq.plugins.cassandra.CassandraNodeDiscoveryComponent.scanForResources(CassandraNodeDiscoveryComponent.java:84)
        at org.rhq.plugins.storage.StorageNodeDiscoveryComponent.discoverResources(StorageNodeDiscoveryComponent.java:54)
        at org.rhq.plugins.storage.StorageNodeDiscoveryComponent.upgrade(StorageNodeDiscoveryComponent.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:305)
        ... 5 more
Caused by: java.io.FileNotFoundException: ./../conf/cassandra.yaml (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:140)
        at org.rhq.cassandra.util.ConfigEditor.load(ConfigEditor.java:40)
        ... 14 more

*************************

Expected results:
No errors

Additional info:
see full agent.log in attachment.
Comment 1 Jay Shaughnessy 2016-01-04 09:23:24 EST
release/jon3.3.x commit 52396e69b4f9b69adf5a26292237b1cee54335f8
Author: Jay Shaughnessy <jshaughn@redhat.com>
Date:   Mon Jan 4 09:15:20 2016 -0500

    Don't rely on toString() impl when setting optionsSource target.




I have also cherry-picked this to master as master commit 27bbc5cb1a599e0cd551dca54fe3c050687d5822
Comment 2 Simeon Pinder 2016-01-07 15:16:57 EST
Moving to MODIFIED as already in release branch.
Comment 3 Simeon Pinder 2016-01-16 03:17:33 EST
Moving to ON_QA for testing with the following build:

https://brewweb.devel.redhat.com//buildinfo?buildID=474795

http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/75/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip
 *Note: jon-server-patch-3.3.0.GA.zip maps to ER01 build of
 jon-server-3.3.0.GA-update-05.zip.
Comment 4 Prachi 2016-01-25 06:36 EST
Created attachment 1117943 [details]
agent.log-CP05-ER01
Comment 5 Prachi 2016-01-25 06:41:15 EST
Attached agent.log after applying CP05-ER01 (agent.log-CP05-ER01)

Description of problem:

For this 1.6.0_34 specific version of openJDK the agent is not started correctly throwing in log:ERROR Plugin [JDG] at [file:/home/hudson/rhq-agent/plugins/infinispan-server-rhq-plugin-6.1.0.Final-redhat-3.jar] could not be loaded and will therefore not be deployed.



Version-Release number of selected component (if applicable):

JON3.3.5.ER01


How reproducible:


Steps to Reproduce:
1. use RHEL7.1 with openJDK1.6.0_34
2. install JON3.3.3.5- ER01
3. start it (rhqctl start)


Actual Results:

**************************
 ERROR [main] (rhq.core.pc.plugin.PluginManager)- Plugin [JDG] at [file:/home/hudson/rhq-agent/plugins/infinispan-server-rhq-plugin-6.1.0.Final-redhat-3.jar] could not be loaded and will therefore not be deployed.
java.lang.NullPointerException
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parseSimpleProperty(ConfigurationMetadataParser.java:255)
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parseProperty(ConfigurationMetadataParser.java:209)
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parseProperties(ConfigurationMetadataParser.java:180)
        at org.rhq.core.clientapi.agent.metadata.ConfigurationMetadataParser.parse(ConfigurationMetadataParser.java:109)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseResourceDescriptor(PluginMetadataParser.java:650)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseServiceDescriptor(PluginMetadataParser.java:486)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseResourceDescriptor(PluginMetadataParser.java:817)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseServiceDescriptor(PluginMetadataParser.java:486)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.parseDescriptor(PluginMetadataParser.java:172)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataParser.<init>(PluginMetadataParser.java:109)
        at org.rhq.core.clientapi.agent.metadata.PluginMetadataManager.loadPlugin(PluginMetadataManager.java:186)
        at org.rhq.core.pc.plugin.PluginManager.loadPlugin(PluginManager.java:322)
        at org.rhq.core.pc.plugin.PluginManager.<init>(PluginManager.java:178)
        at org.rhq.core.pc.PluginContainer.initialize(PluginContainer.java:290)
        at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:2066)
        at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:757)
        at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:461)	
**************************

Expected results:
No errors

Additional info:
see full agent.log in attachment.
Comment 6 Jay Shaughnessy 2016-01-25 09:14:21 EST
Prachi and I had already tested this informally and the fix was working.  Simeon, please make sure the fix is in the update and that the update was applied as expected on the agent.
Comment 8 Michael Burman 2016-01-26 05:06:39 EST
I think this is caused by the fix in BZ 1264395 as it removes schema-awareness with broken OpenJDK versions. This causes JAXB to behave differently and fail with the plugin-descriptor reading.

I'll commit a change to the fix of that BZ.
Comment 9 Michael Burman 2016-01-27 10:57:57 EST
Well it seems the problem isn't even going to away by keeping the schema-awareness (to allow proper parsing of OptionSource enum) and instead it keeps failing. I'll have to investigate even more and add additional fixes (assuming this problem is fixable with this broken JDK).
Comment 10 Filip Brychta 2016-01-28 11:07:07 EST
Based on our discussion (Larry, Simeon, Filip), the decision is to push this issue to 3.3.6 and not to block the 3.3.5 release.

The plan is to exhaust all possibilities of a workaround for the 3.3.6 stream and capture this issue as an OpenJDK/Oracle bug. Once this issue has been properly captured, we may just end up closing the JON bug as CANTFIX with a reference to the OpenJDK bug.
Comment 11 Michael Burman 2016-01-29 08:06:58 EST
I debugged little bit more on this one.. it fails here:

   <!ENTITY addConnectorProtocolCommandParameters '
      &addConnectorCommandParameters;
      <c:simple-property name="socket-binding" required="true" description="The socket binding to use for this connector">
         <c:option-source target="configuration" expression="*/socket-binding=name:type=SocketBindingGroup"/>
      </c:simple-property>

And "target" being null. This is extended part that is used in many parts of the rhq-plugin.xml for JDG. It does not fail on every invocation as each time the target is loaded correctly for many many occasions until failing. So the target parsing (enum) does not fail on every occasion, only with the Infinispan server & hotrod connector.

2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- Parsing: Infinispan Server
2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- Parsing: Hotrod Connector
2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- Parsing: Hotrod Connector
2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- Expression: type='Cache Container' plugin=JDG
2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- getTarget is null: false
2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- Expression: */socket-binding=name:type=SocketBindingGroup
2016-01-29 08:00:00,376 INFO  [main] (ConfigurationMetadataParser)- getTarget is null: true
2016-01-29 08:00:00,377 ERROR [main] (rhq.core.pc.plugin.PluginManager)- Plugin [JDG] at [file:/home/hudson/rhq-agent/plugins/infinispan-server-rhq-plugin-6.1.0.Final-redhat-3.jar] could not be loaded and will therefore not be deployed.
java.lang.NullPointerException

This means two lines above this one works:

<c:option-source target="resource" expression="type=&apos;Cache Container&apos; plugin=&pluginName;"/>

But just below it

<c:option-source target="configuration" expression="*/socket-binding=name:type=SocketBindingGroup"/>

Can not be parsed (the expression is parsed correctly, but target is not) - the "configuration" value is correct. @XmlEnumValue("configuration")
Comment 13 Filip Brychta 2016-05-17 06:11:25 EDT
Retested with openjdk _34 -- _39 and all versions are affected.
Oracle 1.6.0_45 is not.
Comment 17 Larry O'Leary 2016-06-02 10:49:36 EDT
Closing this because we can not fix it. 

A fix has been attempted that seemed like it would workaround the OpenJDK 1.6 issue however, it did not fix this issue.

Due to the amount of effort already invested in this issue without a path to resolution, we are going to cut our losses. To date, this does not appear to affect users and was only identified during QE. The recommendation to get around this bug is to use Oracle JVM or upgrade to JVM 1.7.

This issue is most likely a bug in OpenJDK and if this should be perused, please work with OpenJDK team to identify how this issue can be resolved.

Note You need to log in before you can comment on or make changes to this bug.