Red Hat Bugzilla – Bug 1007730
Ruby apps return 503 when accessing and cannot be started/restarted after migration
Last modified: 2015-05-14 19:28:32 EDT
Description of problem: Create test data include ruby apps on devenv-stage_465, do mongo migration and node upgrade. All the started ruby apps cannot be accessed and cannot be started manually. The app state shows the app is running. # rhc app show ruby18 --state -pxx -k -lopenshift+migration1@redhat.com Cartridge ruby-1.8, postgresql-9.2 is started # curl -I http://ruby18-migration1.dev.rhcloud.com/ HTTP/1.1 503 Service Temporarily Unavailable Date: Fri, 13 Sep 2013 07:39:21 GMT Vary: Accept-Encoding Connection: close Content-Type: text/html; charset=iso-8859-1 # rhc app restart ruby18 -p123123 -k -lopenshift+migration1@redhat.com Unable to complete the requested operation due to: Failed to correctly execute all parallel operations. Reference ID: bb8f0fe5a7945a4ba5d36adcf501f29e Ruby apps on INT are also broken. INT version: devenv_3779 Version-Release number of selected component (if applicable): devenv-stage_465 to devenv-stage_471 How reproducible: always Steps to Reproduce: 1. Launch devenv-stage_465 ami 2. Prepare test data 3. yum clean all && yum update -y 4. Clear broker cache 5. Restart service rhc-broker mcollective rhc-datastore activemq 6. Clear pending ops 7. Migrate datastore # rhc admin-migrate-datastore --version 2.0.33 --non-compatible 8. Migrate gears # oo-admin-upgrade upgrade-node --version 2.0.33 --ignore-cartridge-version 9. Clear broker cache Actual results: Ruby apps are broken as description. Expected results: Apps should work well after migration. Additional info: Found some maybe useful info in the development log: 2013-09-13 03:51:16.964 [DEBUG] DEBUG: server results: restarting Ruby cart (pid:30060) 2013-09-13 03:51:16.966 [DEBUG] DEBUG: server results: (pid:30060) 2013-09-13 03:51:16.971 [DEBUG] DEBUG: server results: /usr/lib/openshift/cartridge_sdk/bash/sdk: line 419: /var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf: Permission denied (pid:30060) 2013-09-13 03:51:16.131 [ERROR] Failed to correctly execute all parallel operations (pid:30060)
Seems to me like the migration didn't run the 'setup' - I mean this line: https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-ruby/bin/setup#L20
Meng, could you please confirm that this file exists on the filesystem? /var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf If no, then it might be a migration problem (see Vojtech comment).
According to the http://openshift.github.io/documentation/oo_cartridge_developers_guide.html#bin-setup, the setup is run only on non-backward compatible upgrade. I think we didn't mark the new cartridges incompatible in the manifest.yml, is that correct, Jhon?
The above statement would be related to the following cartridges: perl php phpmyadmin python ruby
Michal Fojtik, the file is not existing as Comment 2 mentioned.
Michal Fojtik, the file is existing on STG as Comment 2 mentioned, and in Comment 6 it means on INT.
Michal Fojtik, i think Comment 6&7 may make you confuse, sorry for that, please ignore them and refer to the following information: After checking it on STG today, found that the ruby applications can be started/restarted and accessed successfully via browser. So, this issue just occurred on INT/devenv and the passenv.conf file is not on devenv/INT's filesystem. According to above messages, lower the severity to medium. Thanks!
This should have been fixed in current STG(devenv-stage_472), please move it to ON_QA when you think it's ready.
Existing ruby apps work well on STG after migration. Move bug to verified.