Red Hat Bugzilla – Bug 969098
Cartridge control scripts should not overwrite default httpd_nolog.conf with PassEnv directives
Last modified: 2015-05-14 19:20:18 EDT
Description of problem: Cartridge control scripts appends PassEnv directives to the default httpd_nolog.conf file. This could cause a potential troubles when we want update the httpd_nolog.conf file in the future. I believe that file should be even locked, by the way. Steps to Reproduce: 1. rhc app-restart <app> or 1. rhc app-start <app> Actual results: Cartridge control scripts call update_httpd_passenv $HTTPD_CFG_FILE and overwrites the default httpd_nolog.conf config file. Expected results: Cartridge control script should call update_httpd_passenv $HTTPD_PASSENV_CFG_FILE instead and thus output the PassEnv directives into the separate config file, eg. ./configuration/etc/conf.d/PassEnv.conf
During the update, we should recreate the httpd_nolog.conf to be sure it's not malformed.
We've seen at least one instance where an OpenShift Online user's httpd_nolog.conf was truncated and populated with only PassEnv lines, and there's a thread in the forums which looks related: https://www.openshift.com/forums/openshift/after-your-update-my-application-is-still-not-available The most concerning thing for me is that the problem manifests in a way that completely confuses the user. They make no changes to their app, maybe add a cartridge, restart their app, and suddenly apache is broken, with errors like: Syntax error on line 7 of /var/lib/openshift//php/configuration/etc/conf.d/openshift.conf: Invalid command 'CustomLog', perhaps misspelled or defined by a module not included in the server configuration It takes a good bit of knowledge and investigation for a user to realize that their httpd configuration has been destroyed by the control scripts, and once they've determined that, they don't know how to fix it. ( For those reading here who are experiencing the issue, the workaround is to get a copy of httpd_nolog.conf from the appropriate cartridge, e.g., https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-php/versions/shared/configuration/etc/conf/httpd_nolog.conf , and insert those contents into the top of httpd_nolog.conf in your gear's config directory for that cartridge, e.g., ~/php/configuration/etc/conf.d/ )
Also related: https://www.openshift.com/forums/openshift/my-node-js-app-suddenly-stopped-working-mongodb-error-related-i-think
A quick follow-up on this. Whatever caused this problem may have been isolated to one particular node. About 10 different gears (~15 cartridges total) had this problem on one particular node, which makes me think some thing happened during the release/upgrade process on that node. I've fixed the affected cartridges, and will watch for this to crop up again. Looking forward to the implementation of vvitek's suggestion, which would be much safer.
I found 20 more apps with this problem on various OpenShift Online nodes in a couple of cases, I found zero-length files (httpd.conf, httpd_nolog.conf, and php.ini) in phpmyadmin/versions/shared/conf, which sounds like another issue entirely. I'll open another bug when I have a better understanding of that, but for now, just wanted to note that here, since it results in the same end-user experience.
Help! pesobility.com is down again when I woke up, it happens everytime my node is restarted by the system (probably maintenance?). Also posted it in https://www.openshift.com/forums/openshift/my-node-js-app-suddenly-stopped-working-mongodb-error-related-i-think#comment-33110
@Andy Grimm, I agree that the malformed files described in comment 6 are different issue, probably caused during the migration. The only files that could be malfored because of this bug are: $OPENSHIFT_PERL_DIR/etc/conf/httpd_nolog.conf $OPENSHIFT_PHP_DIR/configuration/etc/conf/httpd_nolog.conf $OPENSHIFT_PYTHON_DIR/etc/conf/httpd_nolog.conf $OPENSHIFT_RUBY_DIR/etc/conf/httpd_nolog.conf
https://github.com/openshift/origin-server/pull/3552
test on devenv_3776 with php/perl/ruby/python cartridge steps: 1. create an app 2. ssh into app to check httpd_nolog.conf and passenv.conf files 3. rhc app restart $appname 4. check httpd_nolog.conf and passenv.conf files again after restart the app, passenv.conf will be updated, and the httpd_nolog.conf will not be changed. move the bug to verified.
Hi, Vojtech Vitek test php/perl/python app after the upgrade. 1. for old perl/php app, both httpd_nolog.conf and passenv.conf are existed in app, after restart, httpd_nolog.conf will not update, just passenv.conf updated 2. for python-2.6/python-2.7 old app, only httpd_nolog.conf existed in app (no passenv.conf file existed), after restart, httpd_nolog.conf will not update, no new passenv.conf file created. 3. for python-3.3 old and new app, httpd_nolog.conf and passenv.conf are not existed in app, after restart, no change. 4. for ruby app, it is blocked by the #bug 1007730, will test after #bug 1007730 fixed.
*** Bug 1006665 has been marked as a duplicate of this bug. ***