Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1144998

Summary: List of errors and warnings in server.log after upgrade to 3.3.0.ER03
Product: [JBoss] JBoss Operations Network Reporter: Filip Brychta <fbrychta>
Component: Core ServerAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Filip Brychta <fbrychta>
Severity: medium Docs Contact:
Priority: unspecified    
Version: JON 3.3.0CC: fbrychta, jshaughn, mmahoney
Target Milestone: ER04   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 14:01:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
server log none

Description Filip Brychta 2014-09-22 08:00:22 UTC
Created attachment 939910 [details]
server log

Description of problem:
This is a list of warnings and errors logged during upgrade to 3.3.0.ER03.
Idea of this bz is to identify which should be fixed and which can be safely ignored.

07:08:02,138 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016009: Caught:: java.lang.IllegalStateException

07:08:02,176 WARN  [org.jboss.security] (Periodic Recovery) PBOX000231: End loadConfig, failed to load config: file:/home/hudson/jon-server-3.3.0.ER03/jbossas/bin/login-config.xml: java.io.FileNotFoundException: /home/hudson/jon-server-3.3.0.ER03/jbossas/bin/login-config.xml (No such file or directory)

This is already reported in bz 1127272 -
07:08:02,185 ERROR [org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl] (Periodic Recovery) IJ000906: Error during crash recovery: java:jboss/datasources/RHQDS (Could not create connection): javax.resource.ResourceException: Could not create connection

07:08:02,719 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS014612: Operation ("read-resource") failed - address: ([("subsystem" => "web")]) - failure description: "JBAS014807: Management resource '[(\"subsystem\" => \"web\")]' not found"


07:10:42,526 WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [Servicemix3 Server] from a plugin named [Servicemix3]. This is probably because that plugin is missing. Resource Type [{ActiveMQ}Broker] will not have that missing type as a possible parent.


07:11:40,498 ERROR [org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet] (http-/0.0.0.0:7080-3) Failed to stream agent jar to remote client [10.16.23.172 (10.16.23.172)]: java.lang.RuntimeException:Stream data cannot be copied -> org.apache.catalina.connector.ClientAbortException:null -> java.net.SocketException:Connection reset
07:11:41,027 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/agentupdate].[org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet]] (http-/0.0.0.0:7080-3) JBWEB000236: Servlet.service() for servlet org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet threw exception: java.lang.IllegalStateException


07:15:44,862 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: BindDN
07:15:44,863 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: Filter
07:15:44,863 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: java.naming.factory.initial
07:15:44,863 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: LoginProperty
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: java.naming.referral
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: BaseDN
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: GroupFilter
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: java.naming.provider.url
07:15:44,865 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: BindPW


Complete serve.log is attached

Comment 1 Jay Shaughnessy 2014-09-24 14:19:48 UTC
*** Bug 1127361 has been marked as a duplicate of this bug. ***

Comment 2 Jay Shaughnessy 2014-09-24 14:40:23 UTC

These are reported in Bug 1026799 and should be tracked there.

07:15:44,862 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: BindDN
07:15:44,863 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: Filter
07:15:44,863 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: java.naming.factory.initial
07:15:44,863 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: LoginProperty
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: java.naming.referral
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: BaseDN
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: GroupFilter
07:15:44,864 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: java.naming.provider.url
07:15:44,865 WARN  [org.jboss.security] (http-/0.0.0.0:7080-8) PBOX000234: Invalid or misspelled module option: BindPW

Comment 3 Jay Shaughnessy 2014-09-24 15:25:58 UTC

This looks to be a Wildfly/EAP issue to-be-fixed in the future.  https://issues.jboss.org/browse/WFLY-2828.   Adding a filter for this.

07:08:02,138 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016009: Caught:: java.lang.IllegalStateException



This one is a little strange, I'm not sure why it would be looking for a login-config.xml because that file is not used anymore.  Perhaps a declaration for it is being carried forward in the upgrade process?  I'm not sure.  Maybe Filip can attach his standalone-full.xml from the upgraded environment.  I'll likely just filter this message as the file isn't used or needed.

07:08:02,176 WARN  [org.jboss.security] (Periodic Recovery) PBOX000231: End loadConfig, failed to load config: file:/home/hudson/jon-server-3.3.0.ER03/jbossas/bin/login-config.xml: java.io.FileNotFoundException: /home/hudson/jon-server-3.3.0.ER03/jbossas/bin/login-config.xml (No such file or directory)

Comment 4 Jay Shaughnessy 2014-09-24 16:14:07 UTC
This one is to be expected, we check for existence of the web subsystem and if it doesn't exist you get:

07:08:02,719 ERROR [org.jboss.as.controller.management-operation] (management- handler-thread - 1) JBAS014612: Operation ("read-resource") failed - address: ([("subsystem" => "web")]) - failure description: "JBAS014807: Management resource '[(\"subsystem\" => \"web\")]' not found"

This is generated by EAP so I'll add yet another filter.

Comment 5 Jay Shaughnessy 2014-09-24 17:28:53 UTC

This one may be a real issue in the plugin packs or plugin processing. I've created Bug 1146183 to track it individually.

07:10:42,526 WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [Servicemix3 Server] from a plugin named [Servicemix3]. This is probably because that plugin is missing. Resource Type [{ActiveMQ}Broker] will not have that missing type as a possible parent.

Comment 6 Jay Shaughnessy 2014-09-24 17:37:26 UTC
Filip,

This looks like a connectivity issue or something of that nature.  It looks like an agent is requesting an update due to a version mismatch but we can't stream the data back to the agent. It may be that the server can not resolve or reach the agent but the agent can reach the server.  We won't pursue this issue in this BZ.  If this can not be explained by environment and is a persistent issue, then please report it it a separate BZ.

07:11:40,498 ERROR [org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet] (http-/0.0.0.0:7080-3) Failed to stream agent jar to remote client [10.16.23.172 (10.16.23.172)]: java.lang.RuntimeException:Stream data cannot be copied -> org.apache.catalina.connector.ClientAbortException:null -> java.net.SocketException:Connection reset
07:11:41,027 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/agentupdate].[org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet]] (http-/0.0.0.0:7080-3) JBWEB000236: Servlet.service() for servlet org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet threw exception: java.lang.IllegalStateException

Comment 7 Jay Shaughnessy 2014-09-24 18:51:10 UTC
This handles the relevant parts of the BZ, handled by filtering.  See the comments to determine which are filtering issues and which are being tracked in seperate BZs.


commit 2c44cde5c5001edf5cf8b1ebcbc1fa98d59cbd91
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Sep 24 13:43:32 2014 -0400

    Add EAP-level log filters for messages we can't avoid and don't want to see.



release/jon3.3.x commit 1b241d7a28f65737762e98250cf8b18f18c1377c
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Sep 24 13:43:32 2014 -0400

    (cherry picked from commit 2c44cde5c5001edf5cf8b1ebcbc1fa98d59cbd91)
    Signed-off-by: Jay Shaughnessy <jshaughn>

Comment 8 Filip Brychta 2014-09-26 07:18:31 UTC
Thanks Jay for great feedback.

Comment 3 - PBOX000231, setup used for this upgrade run is gone so I don't have relevant standalone-full.xml. I tried another quick upgrade run and this issue was not visible. I guess we can ignore this issue for now and I will file a separate bz with relevant standalone-full.xml if it happens again. 


Comment 6 - the error is thrown just a few times when the agents are being upgraded so it's not persistent. I will just ignore that like other connection issues reported by agent when the server is down or in upgrade process.

Comment 9 Simeon Pinder 2014-10-01 21:33:16 UTC
Moving to ON_QA as available for test with build:
https://brewweb.devel.redhat.com/buildinfo?buildID=388959

Comment 10 Filip Brychta 2014-10-08 10:43:47 UTC
Verified on
Version :	
3.3.0.ER04
Build Number :	
99d2107:d7c537e