Red Hat Bugzilla – Bug 1016917
System Ruby segfaults, git push and ctl_app are both impossible
Last modified: 2013-11-03 20:51:30 EST
Description of problem:
Our client has a Silver subscription to OpenShift Online. On application 5238c2aa5973cabd1d0001d3 / utilities-allyou.rhcloud.com, something is very broker in the system Ruby stack, which is preventing both "ctl_app" from working on the gear, and preventing git pushes from working (since ruby is used for the postreceive hook.)
Version-Release number of selected component (if applicable):
OpenShift Online Silver
Every invocation of OpenShift's ruby binary.
Steps to Reproduce:
1. rhc ssh utilities
3. <get stacktrace>
I can run ctl_app or git push.
I can do neither without a backtrack.
The stacktrace I am getting is available on GitHub: https://gist.github.com/smerrill/a9398db646dffc2996b9. The application is ID 5238c2aa5973cabd1d0001d3, or http://utilities-allyou.rhcloud.com/.
This problem is definitely related to the gems installed in the gear's .gem directory. Ideally, commands like ctl_app and the postreceive hook should be unsetting GEM_HOME, so that this interference does not happen.
I am able to take that .gem directory and reproduce the issue outside of the gear, so I should be able to file a bug against the json gem for that after I debug a little.
I'll talk to engineering about what the best workaround is here. I could change the commit hook to unset GEM_HOME. for other actions, the user can ssh in, unset GEM_HOME, and then run ctl_app. (They would then need to reset GEM_HOME elsewhere, such as in a pre_start script, I think.)
When I first looked at this, I failed to notice that this app was not set up using the ruby cartridge, and thus was defaulting to ruby-1.8 executables. So what happened here is that the user ran "gem install", and the json 1.8.0 gem was downloaded and compiled against ruby-1.8. So this is the reason for the segfault, rather than some bug in json or our ruby-1.9 interpreter.
I've talked to Clayton, and he agrees that we should be ignoring the gear's GEM_HOME for gear/ctl_app operations.
For now, I have added "unset GEM_HOME" in both the git pre-receive and post-receive hooks for this gear. Please try this and see if it allows you to do git pushes. Then we'll deal with any remaining issues.
Reproducer for this:
rhc app create bz1016917 --from-code https://github.com/a13m/bz1016917
Thanks for the sleuthing on this.
This is indeed primarily a PHP app, but we're taking advantage of the fact that Ruby is available to do a phing build but also install some gems so that we can compile CSS with Compass. I may see if we can make sure that bundler installs to a local directory so that it shouldn't change the json gem that the gear script needs.
A temporary fix is to require 'json' at the correct time. This is "fixed" on the master branch, so this issue does not happen on devenv.
The stage branch PR is https://github.com/openshift/origin-server/pull/3863
Checked on STG(devenv-stage_496) with the app in comment#3, the ctl_app/gear command works well in rhcsh.
[bz1016917-bmengstg.stg.rhcloud.com 52590769dbd93ca387000073]\> gear restart
Cart to restart?
Restarting PHP cartridge
The fix will be pushed to PROD on about Oct. 16th.