Bug 1084009 - The connection timed out exception when applying a patch on slow environments
Summary: The connection timed out exception when applying a patch on slow environments
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Upgrade, Launch Scripts
Version: JON 3.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: CR01
: JON 3.3.1
Assignee: Simeon Pinder
QA Contact: Filip Brychta
URL:
Whiteboard:
: 1120090 1132020 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-03 11:55 UTC by Filip Brychta
Modified: 2015-02-27 19:58 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-27 19:58:19 UTC
Type: Bug


Attachments (Terms of Use)
applicable to 3.2.2 windows (18.60 KB, text/plain)
2014-07-31 19:07 UTC, Jay Shaughnessy
no flags Details
update.log JON 3.2.0 GA -> CP2 (704.70 KB, text/x-log)
2014-08-06 14:42 UTC, Jan Bednarik
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1120090 None None None Never
Red Hat Bugzilla 1132020 None None None Never

Internal Links: 1120090 1132020

Description Filip Brychta 2014-04-03 11:55:26 UTC
Description of problem:
Following error is sometimes thrown when applying a patch on slow environment:

Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:6999. The connection timed out

which probably causes that configuration is not completed. This affect e.g. bz 1063480 - no-tx-separate-pools was not set to true when previous exception occurred.

Version-Release number of selected component (if applicable):
Version :	
3.2.0.GA update-1
Build Number :	
13f0eea:c372b58

How reproducible:
2/3 on slow environment

Steps to Reproduce:
1. install JON3.2.0.GA
2. ./rhqctl stop
3. apply the patch

Actual results:
3) Applying the binary update contents.
Current update is :  		 jon-server-3.2.0.GA-update-01
  Applying additional JON Server updates...
  Starting JON Server...
06:49:29,903 INFO  [org.jboss.modules] JBoss Modules version 1.2.2.Final-redhat-1
Trying to start the RHQ Server...
RHQ Server                     (pid 15506  ) IS starting
  Starting JBoss CLI and applying patches...
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
	at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:280)
	at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:240)
	at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.jboss.modules.Module.run(Module.java:270)
	at org.jboss.modules.Main.main(Main.java:411)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at localhost:6999
	at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:951)
	at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:790)
	at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:766)
	at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:278)
	... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:6999. The connection timed out
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
	at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:929)
	... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:6999. The connection timed out
	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:131)
	at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
	at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:176)
	at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:156)
	at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:118)
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:115)
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:90)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
	... 13 more
  Stopping the JON Server...
06:49:57,814 INFO  [org.jboss.modules] JBoss Modules version 1.2.2.Final-redhat-1
Trying to stop the RHQ Server...
RHQ Server (pid=15506) is stopping...
RHQ Server has stopped.

You have successfully extracted patch contents into (JON server) and/or (JON agent) installation folder(s).
You should do the following as is necessary: 
	 i)Closely follow the remainder of the instructions delivered with this update.
	 ii)restart your server(s)* 
	 iii)restart agents and request agent updates
*NOTE: for HA environments you may need to repeat the steps here for each server before patch application is complete.

Expected results:
No errors

Additional info:
I tried the same scenario on more powerful machine and it worked correctly.

Comment 1 Filip Brychta 2014-04-03 13:20:18 UTC
Workaround:
edit apply-updates.sh, find a line  
sleep 10; 

and increase this sleep to 
sleep 30;

Comment 2 Armine Hovsepyan 2014-04-03 16:26:04 UTC
visible in windows as well -> http://pastebin.test.redhat.com/201293

Comment 3 Simeon Pinder 2014-04-03 17:01:01 UTC
The apply-updates.sh/bat scripts were modified to wait 10s for EAP instance to start up before trying to launch the CLI configuration updates.  This seems pessimistic enough for any moderate/large sized JON Server box that the 'update' is being applied to.  To be clear this is just the EAP server startup not necessarily all of the N applications that may be deployed on the same. 

I think this should be fixed for 3.2.2

Possible fixes:
i)Applying a longer wait is possible by extending the wait time(requires script modification).
ii)Modify the wait time to be changeable via environment variable. Take default otherwise whatever been set by customer.
iii)Modify both scripts to test eap instance for fully started up, before attempting to run the script. 
iv)Fix eap instance to support admin-only startup, then do iii)
v)other options. 

Productization would like this process to remain as automated as possible. Automated application should minimize patch application issues for large and small environments alike.

Comment 4 Filip Brychta 2014-04-10 14:56:44 UTC
Windows:
Open apply-updates.bat and see following lines

rem use a ping loop to wait ~10s, enough start time to bring up the jboss-cli
   ping -n 11 127.0.0.1 > nul
   "%SERVER_PATCH_HOME%\bin\rhqctl.bat" start --server >> "%LOG_FILE%" 2>>&1
   echo   Starting JBoss CLI and applying patches...
   set NOPAUSE=true
   "%SERVER_PATCH_HOME%\jbossas\bin\jboss-cli.bat" --controller=localhost:6999 --connect --file="%JBOSS_CLI_COMMANDS_FILE%" >> "%LOG_FILE%" 2>>&1
   set NOPAUSE=


Waiting (command ping) is on incorrect line.

Comment 8 mark yarborough 2014-07-25 14:40:00 UTC
*** Bug 1120090 has been marked as a duplicate of this bug. ***

Comment 9 Jay Shaughnessy 2014-07-31 19:07:10 UTC
Created attachment 923024 [details]
applicable to 3.2.2 windows

Comment 10 Jay Shaughnessy 2014-07-31 19:14:46 UTC
I've attached an updated version of apply-updates.bat that should solve issues in 3.2.2 and has been checked into JON master as commit e403df337ee.  It can be tested on windows (see Bug 1120090) by unzipping the 3.2.2 patch and overlaying this version of the .bat script.

Work on the .sh script is still pending (see spinder for that)

Comment 11 Jan Bednarik 2014-08-06 14:38:01 UTC
After using Jay's improved apply-updates.bat file from Comment 9 the upgrade process (JON 3.2.0 -> CP2) finishes correctly with no errors and/or exceptions being logged in 'update.log' file (I am attaching the update.log file).

Tested on 64bit Windows 2008 Server.

Comment 12 Jan Bednarik 2014-08-06 14:42:22 UTC
Created attachment 924511 [details]
update.log JON 3.2.0 GA -> CP2

Comment 14 Larry O'Leary 2015-01-09 23:27:42 UTC
*** Bug 1132020 has been marked as a duplicate of this bug. ***

Comment 15 Simeon Pinder 2015-01-19 20:52:49 UTC
Moving into CR01 target milestone as missed ER01 cutoff.

Comment 16 Simeon Pinder 2015-01-29 06:23:50 UTC
This is mostly fixed with the following commits:
9df4fc4ad9a
382ec89885f

Still needs some more testing on Windows after merging with latest changes.

Comment 17 Simeon Pinder 2015-02-11 14:40:08 UTC
One more commit: 97d67b6e286a14d. Moving to MODIFIED to test with next brew build.

Comment 18 Simeon Pinder 2015-02-16 04:49:31 UTC
Moving to ON_QA as available to test with latest CP build:
http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/16/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip
*Note: jon-server-patch-3.3.0.GA.zip maps to CR01 build of jon-server-3.3.0.GA-update-01.zip.

Comment 19 Filip Brychta 2015-02-16 06:20:32 UTC
Verified on
Version :	
3.3.0.GA Update 01
Build Number :	
ab9ebbd:884011b


Note You need to log in before you can comment on or make changes to this bug.