Here's a new entry for the troubleshooting guide: When debugging a rails application developers will frequently run the rails console. This can be done with the broker as such: cd /var/www/openshift/broker ./script/rails console production The problem with doing this is that some OpenShift API calls will be cached under /var/www/openshift/broker/tmp/cache and will be owned by whatever user ran the console. In some cases this can be 'root'. If that happens hours later when the cache expires the broker will attempt to invalidate the cache. Since it's running as the 'apache' user it will not be able to do so and will return 500 errors. If you check /var/www/openshift/broker/httpd/logs/error_log you will see permission denied errors. Here's the best way to clear the cache: pushd /var/www/openshift/broker && bundle exec rake tmp:clear && popd
More of a developer issue since we don't tell anybody about using the rails console in any formal docs. Would probably be good to actually do that, though, especially for troubleshooting. Experienced developers will probably be poking via the rails console and hit this. I'll get a sample of some of these errors and add something to the wiki troubleshooting guide, to be added to formal docs when possible.
I think it might be more than the rails console; it's possible some admin oo-* commands (which use the rails env) could conceivably create cache entries, and they would generally be operating as root or equivalent. I don't know of any that do this now, but I'm planning one. I think we could solve the problem simply in the RPM spec by setting setgid and group write bits on the directory (chmod g+ws). I don't know if that might raise some flags though, and need to see if it actually works. If not... well, make sure to always runuser apache eh? I wonder if there isn't a way to bake that into the rails env.
Good call. We should definitely make sure the admin scripts don't have a change to break things in this way.
From libra-devel: >1. Who is it for? Intended audience? System Administrators of OpenShift Enterprise >2. What do we actually need to document? What is the issue? On an OpenShift Broker/Web Console host administrators should never run the Rails console as root: cd /var/www/openshift/broker ./script/rails console production The same would be for the console: cd /var/www/openshift/console ./script/rails console production >3. Why is this important? The problem with doing this is that some OpenShift API calls will be cached under /var/www/openshift/broker/tmp/cache and will be owned by whatever user ran the console. In some cases this can be 'root'. If that happens hours later when the cache expires the broker will attempt to invalidate the cache. Since it's running as the 'apache' user it will not be able to do so and will return 500 errors. >4. How would you use this? What is the impact on users? See #3 >5. Any other information you deem important that I need to include in >the doc? The doc should specify that if an Administrator wants to run the Rails console they should do so as the Apache user: su --shell=/bin/bash -l apache cd /var/www/openshift/console ./script/rails console production >6. Priority? Severity? What release should we target this for? I would say that documenting this is medium/medium. You can target this for 1.1
A new section on running a Rails console has been added to the Troubleshooting Guide and will be available in the next release.