Bug 1040745

Summary: Error message: Failed to execute: 'control stop' for node.js
Product: OpenShift Online Reporter: Rich Hildred <rhildred>
Component: ImageAssignee: Paul Morie <pmorie>
Status: CLOSED WORKSFORME QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.xCC: bparees, rhildred
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-10 14:39:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Rich Hildred 2013-12-12 03:35:13 UTC
Description of problem:

When I try to do a git push I get the following message:

master: master [remote rejected]

Repository ssh://52544d70e0b8cd9fba00000e.com/~/git/rhfp.git/

pre-receive hook declined
Stopping NodeJS cartridge
Warning: Application 'rhfp' nodejs exists without a pid file.  Use force-stop to kill.
An error occurred executing 'gear prereceive' (exit code: 141)
Error message: Failed to execute: 'control stop' for /var/lib/openshift/52544d70e0b8cd9fba00000e/nodejs

For more details about the problem, try running the command again with the '--trace' option.

If I ssh in to my gear and do:

[rhfp-syndicateme.rhcloud.com 52544d70e0b8cd9fba00000e]\> ctl_app stop --trace
Stopping gear...
Stopping NodeJS cartridge
Warning: Application 'rhfp' nodejs exists without a pid file.  Use force-stop to kill.
An error occurred executing 'gear stop' (exit code: 141)
Error message: Failed to execute: 'control stop' for /var/lib/openshift/52544d70e0b8cd9fba00000e/nodejs

/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1284:in `block in do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1085:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1247:in `do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1107:in `do_control'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1514:in `stop_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1365:in `block in stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1091:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1089:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1089:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/v2_cart_model.rb:1362:in `stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.18.2/lib/openshift-origin-node/model/application_container.rb:468:in `stop_gear'
/usr/bin/gear:492:in `block (3 levels) in <main>'
/usr/bin/gear:65:in `do_command'
/usr/bin/gear:483:in `block (2 levels) in <main>'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:74:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'

Thankfully my server still seems to be up, but I can't update the content.


Version-Release number of selected component (if applicable):
????

How reproducible:
This happens every time on my gear, and on one of my student's gears (http://kince-mikeonthebike.rhcloud.com/)

Steps to Reproduce:
1. Do a git commit in eclipse with timeout set to 120s
2. Do a git push also in eclipse or at command line


Actual results:
the changes don't make it to the repo on the server

Expected results:
the server is updated with the changes

Additional info:
Looks like the last time I made a successful update to my content was Dec. 6

Comment 1 Paul Morie 2013-12-12 22:53:04 UTC
Hi Rich-

Please try force-stopping your app (rhc app force-stop <name of app>) and restarting.

Comment 2 Rich Hildred 2013-12-13 01:23:41 UTC
Hi Paul:

I will have to try in a couple of hours, as I am in my office right now, away from rhc. I can ssh to my gear only.

thanks

Rich

Comment 3 Rich Hildred 2013-12-13 04:14:00 UTC
The force stop worked. My student, who's gear is in the same condition as mine doesn't have ruby or rhc. I see that there is a curl equivalent to do the force stop.

curl -k -X POST https://openshift.redhat.com/broker/rest/domains/[Domain_ID]/applications/[App_Name]/events --user "[UserName]:[Password]" --data "event=force-stop"

It seems unfortunate that I couldn't update my gear. Can you learn anything about the state of my machine, or that of my student. Is this just something that happens?

Comment 4 Paul Morie 2013-12-13 04:15:36 UTC
Rich-

Have you tried to git-push after force-stopping and restarting?

Comment 5 Rich Hildred 2013-12-13 13:13:34 UTC
Hi Again Paul:

Yes I was able to successfully git push after I did the force start. I will try the curl on my student's machine this morning, and see if that gets him going.

thanks

Rich

Comment 6 Ben Parees 2014-01-06 23:13:35 UTC
the pid file in question is nodejs/run/cartridge.pid on your gear.

Is it possible you or your application deleted that file or directory?

As an alternative to force stop, you can also ssh into the gear and recreate the pid file with the appropriate pid#.  (rhc ssh <appname>)

Comment 7 Ben Parees 2014-01-09 22:25:13 UTC
Rich, do you have any update on this? i'd be interested in seeing your app repository if you are still hitting this.

Comment 8 Rich Hildred 2014-01-10 00:35:58 UTC
Hi Ben:

Sorry I didn't reply on the 6th. My gear is working well now. The curl  -k -X POST https://openshift.redhat.com/broker/rest/domains/[Domain_ID]/applications/[App_Name]/events --user "[UserName]:[Password]" --data "event=force-stop"
allowed me to stop it. An interesting thing that I noticed about the pid in nodejs/run/cartridge.pid is that it is the pid for node /opt/rh/nodejs010/root/us..... The actual pid for my server.js has that pid as it's parent. 

thanks

Rich

Comment 9 Ben Parees 2014-01-10 14:39:25 UTC
yes, we launch node under supervisor, so cartridge.pid contains the supervisor pid, not the node process pid, so that makes sense.

I'm going to close this for now, but if you're able to reliably recreate it, please send along the repository for the app you're recreating it with and we'll investigate further.

Thanks!