Description of problem: https://openshift.redhat.com/community/forums/openshift/git-push-error#comment-27693 git push to server results in the following: MacBook-Pro:life office$ git push Warning: Permanently added the RSA host key for IP address '0.0.0.18' to the list of known hosts. Counting objects: 22, done. Delta compression using up to 8 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 6.06 KiB, done. Total 12 (delta 6), reused 0 (delta 0) remote: restart_on_add=false remote: Sending SIGTERM to jboss:25193 25933 ... remote: kill -9 25193 remote: /usr/libexec/openshift/cartridges/jbossews-1.0/info/bin/app_ctl_impl.sh: line 34: kill: (25193) - No such process remote: Failed to stop jbossews-1.0 To ssh://a69b7ca3069b46329eba7572c1704b23.com/~/git/life.git/ ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'ssh://a69b7ca3069b46329eba7572c1704b23.com/~/git/life.git/' MacBook-Pro:life office$ SSH into app gear and ctl_all restart also results in error: [life-tony1016.rhcloud.com ~]\> ctl_app restart Sending SIGTERM to jboss:25193 25933 ... kill -9 25193 /usr/libexec/openshift/cartridges/jbossews-1.0/info/bin/app_ctl_impl.sh: line 34: kill: (25193) - No such process To get user up and running, the following had to be done: ssh into your app gear, cd into jbossews-1.0/run, remove the pid file and run ctl_all restart. Not sure what led to this but definitely not nice. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I wasn't able to reproduce this. Please re-open if it happens again.
Took me a bit but I've got the special sauce. step 1) do as above step 2) ctl_app stop step 3) confirm jboss.pid still exists and has pids in it step 4) run ctl_app stop again confirm it exits with 141 (instead of 0) Then log out, make a git edit, then git push. The git push will be rejected because it can't stop the app. I think this is one of those things where the stop should be silently failing if the app is already stopped.
I wasn't able to reproduce this again today. Here's my shell from following Mike's steps: [jtest-pmorie101.dev.rhcloud.com ~]\> ctl_app stop Sending SIGTERM to jboss:9418 ... kill -TERM 9613 kill -TERM 9418 [jtest-pmorie101.dev.rhcloud.com ~]\> ls jbosseap-6.0/run/jboss.pid jbosseap-6.0/run/jboss.pid [jtest-pmorie101.dev.rhcloud.com ~]\> ctl_app stop Application is already stopped [jtest-pmorie101.dev.rhcloud.com ~]\> echo $? 0 As you can see, ctl_app exited with 0 instead of 141. After that: $ cd jtest; echo 'foo' >> README; git commit -am 'test'; git push; cd - [master b9190ae] test 1 file changed, 1 insertion(+) Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 282 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: restart_on_add=false remote: Application is already stopped remote: Done remote: restart_on_add=false
Ok, so I'm not quite sure how this would happen naturally but you can see the behavior if you replace jboss.pid with some other running pid like your bash pid. I was in a scenario earlier where my app was completely wedged, I couldn't stop it, start it, git push, anything. might be as simple as a ||: on the kill command but we should still verify after the kill that it's not running.
Bill, could you take a look on this one? There's noise being added being added due to the policy change hosted is still waiting for. It doesn't look like the jboss pid file is ever deleted by any hook or script in the abstract-jboss or specific jboss cart.
This ties into https://bugzilla.redhat.com/show_bug.cgi?id=884684. I think the stop logic for all of the JBoss apps should be: 1) Attempt to shutdown gracefully 2) If 1) does not not stop the process after some timeout then we should kill -9 the process. The tomcat6 and tomcat7 scripts are not removing the pid file (neither are the AS/EAP scripts).
'ctl_app stop' doesn't create the stop_lock file since it doesn't call the stop hook which creates the file. This is different than 'rhc app stop' which goes through the stop hook and creates the stop_lock file. A git push will start the app if it was stopped with ctl_app but not if it was stopped with rhc.
Fixed PID logic
Commits pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/c45f924bb3d9e6c4fa9b57565191dbd35335859e Bug 913217 https://github.com/openshift/origin-server/commit/df8f7de302a3266fe3beb7bde185a4f36925b1a5 Bug 913217 https://github.com/openshift/origin-server/commit/17526d3866bcdfaadb17ab70253f30df621df366 Merge pull request #1474 from bdecoste/master Bug 913217 895507 [merge]
After bug 912215 is verified ,we'll verify this bug
Checked on devenv_2890, 1. Create jbossews app 2. ctl_app stop 3. ctl_app stop again 4. Modify git repo and push [master 4e5d950] test 0 files changed create mode 100644 123 Counting objects: 3, done. Delta compression using up to 8 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (2/2), 241 bytes, done. Total 2 (delta 1), reused 0 (delta 0) remote: restart_on_add=false remote: Application(16231) is already stopped remote: Done remote: restart_on_add=false remote: Running .openshift/action_hooks/pre_build It will judge the app stopped or not from the pid. Mark bug as VERIFIED.