Bug 973825 - jboss cartridge restart error with "Timed out waiting for http listening port"
jboss cartridge restart error with "Timed out waiting for http listening port"
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Dan McPherson
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-12 17:08 EDT by Thomas Wiest
Modified: 2015-05-14 19:21 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-24 10:53:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas Wiest 2013-06-12 17:08:23 EDT
Description of problem:
We're seeing quite a few jboss restarts error with "Timed out waiting for http listening port"

I talked to Dan, and he said that there is a loop that checks this roughly 24 times with a 2 second sleep in between.

This may need to be increased.


Broker Log:
2013-06-11 05:15:44.342 [ERROR] Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (pid:8527)
2013-06-11 05:15:44.343 [ERROR] Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (pid:8527)
2013-06-11 05:15:44.354 [ERROR] Reference ID: 0ff2a0c43c134f6c73b51142ea4a66c2 - Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.

2013-06-11 05:14:45.121    {:cartridge=>"jbossas-7", :action=>"restart", :args=>{"--with-app-uuid"=>"51b5a05e50044691ac0004c3", "--with-app-name"=>"REDACTED", "--with-container-uuid"=>"51b5a05e50044691ac0004c3", "--with-container-name"=>"REDACTED", "--with-namespace"=>"REDACTED", "--with-uid"=>6917, "--with-request-id"=>"0ff2a0c43c134f6c73b51142ea4a66c2", "--cart-name"=>"jbossas-7", "--component-name"=>"jbossas-7", "--with-software-version"=>"7", "--cartridge-vendor"=>"redhat"}}, ex-std-node185.prod.rhcloud.com, {'identity' => ex-std-node185.prod.rhcloud.com}) (Request ID: 0ff2a0c43c134f6c73b51142ea4a66c2) (pid:8527)


mcollective log on the ex-node:
I, [2013-06-11T05:14:34.754013 #7972]  INFO -- : openshift.rb:73:in `cartridge_do_action' cartridge_do_action failed
 (1)
------
Failed to execute: 'control restart' for /var/lib/openshift/51b5a05e50044691ac0004c3/jbossas
Restarting jbossas cart
Stopping jbossas cart
Starting jbossas cart
Timed out waiting for http listening port


------)



Version-Release number of selected component (if applicable):
openshift-origin-cartridge-jbossas-1.2.10-1.el6oso.noarch


How reproducible:
unknown, just happens in PROD quite a bit.

Steps to Reproduce:
1. unknown


Actual results:
Jboss gears error when restarting with "Timed out waiting for http listening port"


Expected results:
no error
Comment 1 Dan McPherson 2013-06-12 18:25:23 EDT
https://github.com/openshift/origin-server/pull/2834
Comment 3 Meng Bo 2013-06-14 02:24:33 EDT
Tested on devenv_3360

Add "echo 1TIME" to the ishttpup function for control script.

SSH login to jboss app, and do a restart ops. 
Kill the java process during the app restart.
It will try more times then before.


\> /usr/libexec/openshift/cartridges/v2/jbossas/bin/control restart
Restarting jbossas cart
Stopping jbossas cart
Sending SIGTERM to jboss:8117 ...
Starting jbossas cart
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
1TIME
Timed out waiting for http listening port

Note You need to log in before you can comment on or make changes to this bug.