Bug 1164391

Summary: Timing issues with WAIT_FOR_BEFORE_START can result in connectors never starting
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Aaron Ogburn <aogburn>
Component: WebAssignee: Aaron Ogburn <aogburn>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Cada <mcada>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.0CC: aogburn, rmaucher
Target Milestone: DR11   
Target Release: EAP 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-02 07:30:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1165859, 1166308    
Attachments:
Description Flags
bz1164391.btm none

Description Aaron Ogburn 2014-11-14 22:16:30 UTC 
Description of problem:

Sometimes, you may see that connectors never start if you use the WAIT_FOR_BEFORE_START property.  This is because of a timing issue allowed by the concurrency of WebConnectorService.start and WebVirtualHostService.start.  If the host hasn't started by the WebConnectorService.start time, it doesn't get the WAIT_FOR_BEFORE_START listener, and so we end up missing notifications on the app start up and then the connector can never start.


Version-Release number of selected component (if applicable):

JBossWeb 7.5


How reproducible:

Somewhat difficult to reproduce the timing issue.


Steps to Reproduce:
1. ./standalone.sh-Dorg.apache.catalina.connector.WAIT_FOR_BEFORE_START=/app
2. Deploy /app
3. Check for the connectors not starting when /app deploys.  The issue can be reproduced 100% of the time if you somehow force a a delay of a few seconds at the start of org.apache.catalina.core.StandardEngine.addChild().

Actual results:

Connectors may not start with WAIT_FOR_BEFORE_START.


Expected results:

Connectors always start with WAIT_FOR_BEFORE_START when required apps are deployed.

Additional info:
Comment 2 Aaron Ogburn 2014-11-19 20:32:18 UTC 
And there's another related concurrency issue that breaks this.  The app context could be added tot he host before the listener is added to the host.  When that happens, the property listener is not placed on the context.  If the context is not started when it's checked during the WebConnectorService.start call, then we'll never notify the connector to start.
Comment 3 Aaron Ogburn 2014-11-19 20:47:29 UTC 
Created attachment 959153 [details]
bz1164391.btm
Comment 4 Aaron Ogburn 2014-11-19 20:51:46 UTC 
I attached a byteman rule that so far has always reproduced the issue for me.  Launch JBoss with the following in your JVM options to use it:

-javaagent:/path/to/byteman/lib/byteman.jar=script:/path/to/bz1164391.btm,sys:/path/to/byteman/lib/byteman.jar

And to be able to reproduce the scenario from c#2, start JBoss with the application already in deployments with a .dodeploy.
Comment 5 Aaron Ogburn 2014-11-19 21:37:04 UTC 
PR: https://github.com/jbossas/jboss-eap/pull/1993
Comment 6 Radim Hatlapatka 2014-11-27 11:49:52 UTC 
Verified with EAP 6.4.0.DR11