Bug 1007730 - Ruby apps return 503 when accessing and cannot be started/restarted after migration
Summary: Ruby apps return 503 when accessing and cannot be started/restarted after mig...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Michal Fojtik
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-13 08:50 UTC by Meng Bo
Modified: 2015-05-14 23:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-19 16:51:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Meng Bo 2013-09-13 08:50:29 UTC
Description of problem:
Create test data include ruby apps on devenv-stage_465, do mongo migration and node upgrade. All the started ruby apps cannot be accessed and cannot be started manually.

The app state shows the app is running.
# rhc app show ruby18 --state -pxx -k -lopenshift+migration1 
Cartridge ruby-1.8, postgresql-9.2 is started

# curl -I http://ruby18-migration1.dev.rhcloud.com/
HTTP/1.1 503 Service Temporarily Unavailable
Date: Fri, 13 Sep 2013 07:39:21 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

# rhc app restart ruby18 -p123123 -k -lopenshift+migration1 
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations.
Reference ID: bb8f0fe5a7945a4ba5d36adcf501f29e


Ruby apps on INT are also broken.
INT version: devenv_3779

Version-Release number of selected component (if applicable):
devenv-stage_465 to devenv-stage_471

How reproducible:
always

Steps to Reproduce:
1. Launch devenv-stage_465 ami
2. Prepare test data
3. yum clean all && yum update -y
4. Clear broker cache
5. Restart service 
rhc-broker
mcollective 
rhc-datastore
activemq
6. Clear pending ops
7. Migrate datastore
# rhc admin-migrate-datastore --version 2.0.33 --non-compatible
8. Migrate gears
# oo-admin-upgrade upgrade-node --version 2.0.33 --ignore-cartridge-version
9. Clear broker cache

Actual results:
Ruby apps are broken as description.

Expected results:
Apps should work well after migration.

Additional info:
Found some maybe useful info in the development log:

2013-09-13 03:51:16.964 [DEBUG] DEBUG: server results: restarting Ruby cart (pid:30060)
2013-09-13 03:51:16.966 [DEBUG] DEBUG: server results: (pid:30060)
2013-09-13 03:51:16.971 [DEBUG] DEBUG: server results: /usr/lib/openshift/cartridge_sdk/bash/sdk: line 419: /var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf: Permission denied (pid:30060)
2013-09-13 03:51:16.131 [ERROR] Failed to correctly execute all parallel operations (pid:30060)

Comment 1 Vojtech Vitek 2013-09-13 15:00:54 UTC
Seems to me like the migration didn't run the 'setup' - I mean this line:
https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-ruby/bin/setup#L20

Comment 2 Michal Fojtik 2013-09-13 15:09:41 UTC
Meng, could you please confirm that this file exists on the filesystem?

/var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf

If no, then it might be a migration problem (see Vojtech comment).

Comment 3 Vojtech Vitek 2013-09-13 15:15:53 UTC
According to the http://openshift.github.io/documentation/oo_cartridge_developers_guide.html#bin-setup, the setup is run only on non-backward compatible upgrade.

I think we didn't mark the new cartridges incompatible in the manifest.yml, is that correct, Jhon?

Comment 4 Vojtech Vitek 2013-09-13 15:18:55 UTC
The above statement would be related to the following cartridges:
perl
php
phpmyadmin
python
ruby

Comment 6 chunchen 2013-09-16 01:56:55 UTC
Michal Fojtik, the file is not existing as Comment 2 mentioned.

Comment 7 chunchen 2013-09-16 03:25:43 UTC
Michal Fojtik, the file is existing on STG as Comment 2 mentioned, and in Comment 6 it means on INT.

Comment 8 chunchen 2013-09-16 06:19:10 UTC
Michal Fojtik, i think Comment 6&7 may make you confuse, sorry for that, please ignore them and refer to the following information:

After checking it on STG today, found that the ruby applications can be started/restarted and accessed successfully via browser. So, this issue just occurred on INT/devenv and the passenv.conf file is not on devenv/INT's filesystem. 

According to above messages, lower the severity to medium. Thanks!

Comment 9 Xiaoli Tian 2013-09-16 06:50:00 UTC
This should have been fixed in current STG(devenv-stage_472), please move it to ON_QA when you think it's ready.

Comment 10 Meng Bo 2013-09-16 07:38:26 UTC
Existing ruby apps work well on STG after migration.

Move bug to verified.


Note You need to log in before you can comment on or make changes to this bug.