Bug 971262

Summary:	Gear does not start and restart with ssh shell results in error
Product:	OpenShift Online	Reporter:	Alexander Bätz <baetz.alexander>
Component:	Containers	Assignee:	Dan Mace <dmace>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	baetz.alexander, bhatiam, dmace, dmcphers, jhonce, maerco
Target Milestone:	---	Keywords:	SupportQuestion
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-30 14:49:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Bätz 2013-06-06 07:02:42 UTC

Description of problem:

Currently my jbossws 2 application does not respond. i only get the 503 apache error page. as far as i can tell the jbossws 2 is not running.

when i try to call ctrl_all restart or something like that i get:

Timed out waiting for http listening port
An error occurred executing 'gear start'
 
For more details about the problem, try running the command again with the '--trace' option.


when i do gear start --trace i get:

Starting gear...
[Wed Jun 05 12:24:19 2013] [warn] module php5_module is already loaded, skipping
httpd (pid 31323) already running
MySQL already running
[Wed Jun 05 12:24:21 2013] [warn] module php5_module is already loaded, skipping
httpd (pid 31521) already running
Timed out waiting for http listening port
An error occurred executing 'gear start'
 
Failed to execute: 'control start' for /var/lib/openshift/51ab89e0e0b8cd11dc000203/jbossews: rc(1)
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:1024:in `block in do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:832:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:993:in `do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:854:in `do_control'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:1198:in `start_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:1140:in `block in start_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:838:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:836:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:836:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:76:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/v2_cart_model.rb:1136:in `start_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/application_container.rb:381:in `start_gear'
/usr/bin/gear:199:in `block (3 levels) in <main>'
/usr/bin/gear:80:in `do_command'
/usr/bin/gear:194:in `block (2 levels) in <main>'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:74:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
for now i will create a second app and see if the same happens there.

Version-Release number of selected component (if applicable):

the application worked before the updates on june 2.
It does not work anymore.

How reproducible:

can be done

Steps to Reproduce:
1. create new jbossws 2 (tomcat7) gear
2. add mysql, phpmyadmin, metrics
3. push content from https://github.com/Laures/openshift-spring-testapp to openshift

The application will try to start at this point and fail for unknown reasons

4. try to restart the application with ctl_all restart (any other ctl_ will do, too)

Actual results:

app does not start, a call to the chosen url will result in an 503 apache error page.

Expected results:

successfull restart OR meaningfull error message

Additional info:

it looks like the jbossws/conf/server.xml on my gears is empty which results in a stacktrace during the start of tomcat.

All of this might be the fault of my application code, but i am not aware of any change during the latest updates that could have broken my app.

my original gear is springtest-laures.rhcloud.com

Comment 1 Alexander Bätz 2013-06-08 12:35:39 UTC

i rechecked a lot. as far as i can tell the ALL files in the jbossws/conf folder are empty. when i create a new gear (clean) they are not empty. when i push my git repo i get a gear postreceive error, after that server.xml is empty and i get all the errors. also the jbossws is never started anymore because there are never more then two startup trials in the log.

Comment 2 maerco 2013-06-11 07:30:46 UTC

I have exactly the same problem on my environment, 
the only difference is that I'm pushing commafeed git

Comment 3 manoj 2013-06-19 18:19:15 UTC

Adding FutureFeature list as this is not a release blocker.

Comment 4 Dan Mace 2013-07-30 14:49:31 UTC

I created a new jbossews-2.0/mysql-5.1 application and pushed the code from https://github.com/Laures/openshift-spring-testapp to it; the application started without error, but I'm not sure what the nature of the code is and how else to verify it.

I'm going to mark this one closed as fixed for the upcoming release.