Created attachment 800513 [details] eap6 component errors dialog Description of problem: I upgraded JON 3.1.2.GA with a JBossAS7 Standalone Server resource in inventory to JON 3.2.er1 and this resource was not available any more. Relevant upgraded agent is throwing following exception: 2013-09-20 04:15:26,464 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to start component for Resource[id=10006, uuid=add0dde0-9394-4ed4-baba-080f92a0e38b, type={JBossAS7}JBossAS7 Standalone Server, key=hostConfig: /home/hudson/jboss-eap6-standalone/standalone/configuration/standalone.xml, name=EAP (0.0.0.0:9990), parent=jman-fbrychta-ag2.bc.jonqe.lab.eng.bos.redhat.com, version=EAP 6.0.0.GA] from synchronized merge. org.rhq.core.clientapi.agent.PluginContainerException: Failed to start component for resource Resource[id=10006, uuid=add0dde0-9394-4ed4-baba-080f92a0e38b, type={JBossAS7}JBossAS7 Standalone Server, key=hostConfig: /home/hudson/jboss-eap6-standalone/standalone/configuration/standalone.xml, name=EAP (0.0.0.0:9990), parent=jman-fbrychta-ag2.bc.jonqe.lab.eng.bos.redhat.com, version=EAP 6.0.0.GA]. at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1941) at org.rhq.core.pc.inventory.InventoryManager.refreshResourceComponentState(InventoryManager.java:3371) at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2939) at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2946) at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1242) at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1211) at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:1193) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:130) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:92) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Caused by: java.lang.NullPointerException at org.rhq.modules.plugins.jbossas7.ASConnection.newInstanceForServerPluginConfiguration(ASConnection.java:230) at org.rhq.modules.plugins.jbossas7.BaseServerComponent.start(BaseServerComponent.java:98) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) ... 3 more Plugin configuration of eap6 standalone via CLI: rhqadmin.23.81:7080$ ConfigurationManager.getPluginConfiguration(10006); Configuration [12602] - null port = 9990 homeDir = /home/hudson/jboss-eap6-standalone hostname = 0.0.0.0 configDir = /home/hudson/jboss-eap6-standalone/standalone/configuration domainHost = password = _._._[MaSKeD]_._._ productType = EAP startScriptPrefix = /usr/bin/nohup startScript = /home/hudson/jboss-eap6-standalone/bin/standalone.sh startScriptEnv = JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre PATH=/bin:/usr/bin baseDir = /home/hudson/jboss-eap6-standalone/standalone logDir = /home/hudson/jboss-eap6-standalone/standalone/log realm = ManagementRealm hostConfigFile = /home/hudson/jboss-eap6-standalone/standalone/configuration/standalone.xml startScriptArgs = null hostXmlFileName = standalone.xml user = rhqadmin logEventSources [1] { enabled logFilePath includesPattern minimumSeverity dateFormat -------------------------------------------------------------------------------------------------------------------- false /home/hudson/jboss-eap6-standalone/standalone/log/server.log 1 rows } Version-Release number of selected component (if applicable): Version: 3.2.0.ER1 Build Number: 54dd29c:464a643 How reproducible: 2/2 Additional info: This happened after upgrade from jon3.1.0.GA to jon3.2.er1 as well. JBossAS7 Host Controller resource is down after upgrade as well. Neither a restart of RHQ server nor RHQ agent fixed this issue. Complete server and agent logs attached.
Created attachment 800514 [details] complete logs
It's the second time somebody is reporting the issue, see BZ998842. There might be a problem in the agent resource upgrade code, we're trying to activate a resource which has valid values on the server side, but empty values on the agent side. Workaround is to execute Prompt Command operation 'inventory --sync'. It can be executed on all agents if you create a compatible group of agents.
Fixed in master commit 8cfdad217914a513f98f2a474e367b1374eb23ea Author: Thomas Segismont <tsegismo> Date: Thu Sep 26 18:37:49 2013 +0200 The bug was introduced with the implementation of BZ974876 "[RFE] ResourceUpgradeFacet should support upgrade of plugin config properties" In DiscoveryBossBean, upgrade responses were filled with current resource plugin configuration even if the upgrade request had nothing to update in it. But plugin config was loaded lazily and when going through the server services marshalling layer, it was turned into null. As a consequence, the server still had the right configuration, but the agent data was wrong.
Moving to ON_QA for testing.
Verified on Version: 3.2.0.ER3 Build Number: c0742ed:90dd474