This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1007730 - Ruby apps return 503 when accessing and cannot be started/restarted after migration
Ruby apps return 503 when accessing and cannot be started/restarted after mig...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Michal Fojtik
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-13 04:50 EDT by Meng Bo
Modified: 2015-05-14 19:28 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-19 12:51:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Meng Bo 2013-09-13 04:50:29 EDT
Description of problem:
Create test data include ruby apps on devenv-stage_465, do mongo migration and node upgrade. All the started ruby apps cannot be accessed and cannot be started manually.

The app state shows the app is running.
# rhc app show ruby18 --state -pxx -k -lopenshift+migration1@redhat.com 
Cartridge ruby-1.8, postgresql-9.2 is started

# curl -I http://ruby18-migration1.dev.rhcloud.com/
HTTP/1.1 503 Service Temporarily Unavailable
Date: Fri, 13 Sep 2013 07:39:21 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

# rhc app restart ruby18 -p123123 -k -lopenshift+migration1@redhat.com 
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations.
Reference ID: bb8f0fe5a7945a4ba5d36adcf501f29e


Ruby apps on INT are also broken.
INT version: devenv_3779

Version-Release number of selected component (if applicable):
devenv-stage_465 to devenv-stage_471

How reproducible:
always

Steps to Reproduce:
1. Launch devenv-stage_465 ami
2. Prepare test data
3. yum clean all && yum update -y
4. Clear broker cache
5. Restart service 
rhc-broker
mcollective 
rhc-datastore
activemq
6. Clear pending ops
7. Migrate datastore
# rhc admin-migrate-datastore --version 2.0.33 --non-compatible
8. Migrate gears
# oo-admin-upgrade upgrade-node --version 2.0.33 --ignore-cartridge-version
9. Clear broker cache

Actual results:
Ruby apps are broken as description.

Expected results:
Apps should work well after migration.

Additional info:
Found some maybe useful info in the development log:

2013-09-13 03:51:16.964 [DEBUG] DEBUG: server results: restarting Ruby cart (pid:30060)
2013-09-13 03:51:16.966 [DEBUG] DEBUG: server results: (pid:30060)
2013-09-13 03:51:16.971 [DEBUG] DEBUG: server results: /usr/lib/openshift/cartridge_sdk/bash/sdk: line 419: /var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf: Permission denied (pid:30060)
2013-09-13 03:51:16.131 [ERROR] Failed to correctly execute all parallel operations (pid:30060)
Comment 1 Vojtech Vitek 2013-09-13 11:00:54 EDT
Seems to me like the migration didn't run the 'setup' - I mean this line:
https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-ruby/bin/setup#L20
Comment 2 Michal Fojtik 2013-09-13 11:09:41 EDT
Meng, could you please confirm that this file exists on the filesystem?

/var/lib/openshift/52327c15caf1de0aa3000004/ruby//etc/conf.d/passenv.conf

If no, then it might be a migration problem (see Vojtech comment).
Comment 3 Vojtech Vitek 2013-09-13 11:15:53 EDT
According to the http://openshift.github.io/documentation/oo_cartridge_developers_guide.html#bin-setup, the setup is run only on non-backward compatible upgrade.

I think we didn't mark the new cartridges incompatible in the manifest.yml, is that correct, Jhon?
Comment 4 Vojtech Vitek 2013-09-13 11:18:55 EDT
The above statement would be related to the following cartridges:
perl
php
phpmyadmin
python
ruby
Comment 6 chunchen 2013-09-15 21:56:55 EDT
Michal Fojtik, the file is not existing as Comment 2 mentioned.
Comment 7 chunchen 2013-09-15 23:25:43 EDT
Michal Fojtik, the file is existing on STG as Comment 2 mentioned, and in Comment 6 it means on INT.
Comment 8 chunchen 2013-09-16 02:19:10 EDT
Michal Fojtik, i think Comment 6&7 may make you confuse, sorry for that, please ignore them and refer to the following information:

After checking it on STG today, found that the ruby applications can be started/restarted and accessed successfully via browser. So, this issue just occurred on INT/devenv and the passenv.conf file is not on devenv/INT's filesystem. 

According to above messages, lower the severity to medium. Thanks!
Comment 9 Xiaoli Tian 2013-09-16 02:50:00 EDT
This should have been fixed in current STG(devenv-stage_472), please move it to ON_QA when you think it's ready.
Comment 10 Meng Bo 2013-09-16 03:38:26 EDT
Existing ruby apps work well on STG after migration.

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.