Bug 1007730

Summary: Ruby apps return 503 when accessing and cannot be started/restarted after migration
Product: OpenShift Online Reporter: Meng Bo <bmeng>
Component: ContainersAssignee: Michal Fojtik <mfojtik>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bmeng, chunchen, jhonce, vvitek, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-19 16:51:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meng Bo 2013-09-13 08:50:29 UTC
Description of problem:
Create test data include ruby apps on devenv-stage_465, do mongo migration and node upgrade. All the started ruby apps cannot be accessed and cannot be started manually.

The app state shows the app is running.
# rhc app show ruby18 --state -pxx -k -lopenshift+migration1 
Cartridge ruby-1.8, postgresql-9.2 is started

# curl -I http://ruby18-migration1.dev.rhcloud.com/
HTTP/1.1 503 Service Temporarily Unavailable
Date: Fri, 13 Sep 2013 07:39:21 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

# rhc app restart ruby18 -p123123 -k -lopenshift+migration1 
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations.
Reference ID: bb8f0fe5a7945a4ba5d36adcf501f29e


Ruby apps on INT are also broken.
INT version: devenv_3779

Version-Release number of selected component (if applicable):
devenv-stage_465 to devenv-stage_471

How reproducible:
always

Steps to Reproduce:
1. Launch devenv-stage_465 ami
2. Prepare test data
3. yum clean all && yum update -y
4. Clear broker cache
5. Restart service 
rhc-broker
mcollective 
rhc-datastore
activemq
6. Clear pending ops
7. Migrate datastore
# rhc admin-migrate-datastore --version 2.0.33 --non-compatible
8. Migrate gears
# oo-admin-upgrade upgrade-node --version 2.0.33 --ignore-cartridge-version
9. Clear broker cache

Actual results:
Ruby apps are broken as description.

Expected results:
Apps should work well after migration.

Additional info:
Found some maybe useful info in the development log:

2013-09-13 03:51:16.964 [DEBUG] DEBUG: server results: restarting Ruby cart (pid:30060)
2013-09-13 03:51:16.966 [DEBUG] DEBUG: server results: (pid:30060)
2013-09-13 03:51:16.971 [DEBUG] DEBUG: server results: /usr/lib/openshift/cartridge_sdk/bash/sdk: line 419: /var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf: Permission denied (pid:30060)
2013-09-13 03:51:16.131 [ERROR] Failed to correctly execute all parallel operations (pid:30060)

Comment 1 Vojtech Vitek 2013-09-13 15:00:54 UTC
Seems to me like the migration didn't run the 'setup' - I mean this line:
https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-ruby/bin/setup#L20

Comment 2 Michal Fojtik 2013-09-13 15:09:41 UTC
Meng, could you please confirm that this file exists on the filesystem?

/var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf

If no, then it might be a migration problem (see Vojtech comment).

Comment 3 Vojtech Vitek 2013-09-13 15:15:53 UTC
According to the http://openshift.github.io/documentation/oo_cartridge_developers_guide.html#bin-setup, the setup is run only on non-backward compatible upgrade.

I think we didn't mark the new cartridges incompatible in the manifest.yml, is that correct, Jhon?

Comment 4 Vojtech Vitek 2013-09-13 15:18:55 UTC
The above statement would be related to the following cartridges:
perl
php
phpmyadmin
python
ruby

Comment 6 chunchen 2013-09-16 01:56:55 UTC
Michal Fojtik, the file is not existing as Comment 2 mentioned.

Comment 7 chunchen 2013-09-16 03:25:43 UTC
Michal Fojtik, the file is existing on STG as Comment 2 mentioned, and in Comment 6 it means on INT.

Comment 8 chunchen 2013-09-16 06:19:10 UTC
Michal Fojtik, i think Comment 6&7 may make you confuse, sorry for that, please ignore them and refer to the following information:

After checking it on STG today, found that the ruby applications can be started/restarted and accessed successfully via browser. So, this issue just occurred on INT/devenv and the passenv.conf file is not on devenv/INT's filesystem. 

According to above messages, lower the severity to medium. Thanks!

Comment 9 Xiaoli Tian 2013-09-16 06:50:00 UTC
This should have been fixed in current STG(devenv-stage_472), please move it to ON_QA when you think it's ready.

Comment 10 Meng Bo 2013-09-16 07:38:26 UTC
Existing ruby apps work well on STG after migration.

Move bug to verified.