Bug 1292367

Summary: [GSS](6.4.z) Race condition during server start-up - RA not deployed
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Hisanobu Okuda <hokuda>
Component: JCAAssignee: Lin Gao <lgao>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Simka <msimka>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.4CC: bbaranow, bmaxwell, dereed, dpospisi, egonzale, jbilek, jkudrnac, jpederse, jtruhlar, kkhan, msimka, myarboro, onagano, pslavice, sgilda, smaestri, smumford, tom.ross
Target Milestone: CR1Flags: onagano: needinfo+
Target Release: EAP 6.4.8   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 984855
: 1326951 (view as bug list) Environment:
Last Closed: 2017-01-17 12:38:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 984855    
Bug Blocks: 1047961, 1279553, 1322600, 1326951    
Attachments:
Description Flags
standalone.xml
none
ojdbc6_g.jar
none
startup_shutdown_100times none

Description Hisanobu Okuda 2015-12-17 08:30:13 UTC 
+++ This bug was initially created as a clone of Bug #984855 +++

Description of problem:
EAP 6.4.4 rarely fails to deploy the teiid RA and raises an exception:-

13:15:31,324 ERROR [org.jboss.msc.service.fail] (ResourceAdapterDeploymentService Thread Pool -- 1) MSC000001: Failed to start service jboss.ra.deployment."org.jboss.teiid.resource-adapter.file:main_fileQS39": org.jboss.msc.service.StartException in service jboss.ra.deplo
yment."org.jboss.teiid.resource-adapter.file:main_fileQS39": JBAS010446: Failed to start RA deployment [fileQS39]
        at org.jboss.as.connector.services.resourceadapters.deployment.AbstractResourceAdapterDeploymentService$1.run(AbstractResourceAdapterDeploymentService.java:279) [jboss-as-connector-7.5.4.Final-redhat-4.jar:7.5.4.Final-redhat-4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]
Caused by: org.jboss.jca.deployers.common.DeployException: IJ020056: Deployment failed: file:/home/opt/dv62x/modules/system/layers/dv/org/jboss/teiid/resource-adapter/file/main/
        at org.jboss.jca.deployers.common.AbstractResourceAdapterDeployer.createObjectsAndInjectValue(AbstractResourceAdapterDeployer.java:2518) [ironjacamar-deployers-common-1.0.32.Final-redhat-1.jar:1.0.32.Final-redhat-1]
        at org.jboss.as.connector.services.resourceadapters.deployment.ResourceAdapterXmlDeploymentService$AS7RaXmlDeployer.doDeploy(ResourceAdapterXmlDeploymentService.java:185) [jboss-as-connector-7.5.4.Final-redhat-4.jar:7.5.4.Final-redhat-4]
        at org.jboss.as.connector.services.resourceadapters.deployment.ResourceAdapterXmlDeploymentService.start(ResourceAdapterXmlDeploymentService.java:106) [jboss-as-connector-7.5.4.Final-redhat-4.jar:7.5.4.Final-redhat-4]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
Caused by: javax.resource.spi.InvalidPropertyException: ParentDirectory is not set
        at org.teiid.resource.adapter.file.FileManagedConnectionFactory.createConnectionFactory(FileManagedConnectionFactory.java:47)
        at org.teiid.resource.spi.BasicManagedConnectionFactory.createConnectionFactory(BasicManagedConnectionFactory.java:56)
        at org.jboss.jca.deployers.common.AbstractResourceAdapterDeployer.createObjectsAndInjectValue(AbstractResourceAdapterDeployer.java:2253) [ironjacamar-deployers-common-1.0.32.Final-redhat-1.jar:1.0.32.Final-redhat-1]
        ... 7 more



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. install EAP 6.4.4
2. install JDV 6.2.0 on the top of EAP 6.4.4
3. copy the attached standalone.xml
4. repeat start/stop server
5. You will see the exception a few times in every 100 startup.

Actual results:


Expected results:


Additional info:

I don't observe the issue if I remove the 4 oracle datasources in the standalone.xml.
Comment 1 Hisanobu Okuda 2015-12-17 09:03:23 UTC 
Created attachment 1106635 [details]
standalone.xml
Comment 2 Hisanobu Okuda 2015-12-17 09:05:08 UTC 
ERRATA

Steps to Reproduce:
1. install EAP 6.4.4
2. install JDV 6.2.0 on the top of EAP 6.4.4
3. copy the attached standalone.xml
4. copy the attached ojdbc6_g.jar
5. repeat start/stop server
6. You will see the exception a few times in every 100 startup.
Comment 3 Hisanobu Okuda 2015-12-17 09:05:59 UTC 
Created attachment 1106636 [details]
ojdbc6_g.jar
Comment 4 Hisanobu Okuda 2015-12-22 03:18:33 UTC 
Created attachment 1108554 [details]
startup_shutdown_100times
Comment 5 Enrique Gonzalez Martinez 2016-01-05 14:51:04 UTC 
Taking this. It does not seem connected to WFLY-1701 as the fix is already in 6.4.x branch. 

https://github.com/jbossas/jboss-eap/commit/8145eb54234120ecc5d2ff55f4720984bb125443
Comment 7 Enrique Gonzalez Martinez 2016-01-08 15:38:12 UTC 
This is a race condition. During the deployment there are a few services involved in the boot up:

a) 
  1) ConnectionDefinitionService
  2) CDConfigPropertiesService

b) AS7RaXmlDeployer (doDeploy)

There are two different race conditions (not just the one found in this BZ) but they are related.

1) if a) is executed after b), the RA deployer won't create the factory. (the ModifiableResourceAdapter has not been set at that point because the ConnectionDefinitionService has no been started yet; therefore the AS7RaXMLDeployer cannot deploy the connection definition during its execution).

2) if it is executed in the right order but a.2 (set the connector property) is executed after b then you get the stacktrace in the description. The problem is that the ModifiableConnDef has not registered the property by the CDConfigPropertiesService; It means that As7RaXmlDeployer cannot set the property before creating the factory during Ra deployment (org.teiid.resource.adapter.file.FileManagedConnectionFactory).
Comment 11 Osamu Nagano 2016-01-27 01:21:07 UTC 
@lgao, Right, my mistake.  Fixed these fields.  Thanks!
Comment 13 JBoss JIRA Server 2016-03-07 03:33:42 UTC 
Lin Gao <lgao> updated the status of jira JBEAP-2899 to Resolved
Comment 15 Mike McCune 2016-03-28 22:55:34 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 18 Jiří Bílek 2016-05-10 06:07:06 UTC 
Verified with EAP 6.4.8.CP.CR2
Comment 19 JBoss JIRA Server 2016-06-14 11:38:14 UTC 
Jiri Pallich <jpallich> updated the status of jira JBEAP-2899 to Closed
Comment 21 Petr Penicka 2017-01-17 12:38:37 UTC 
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.