Bug 989463 - Apache based applications return 503 when access after upgrade and migration
Apache based applications return 503 when access after upgrade and migration
Status: CLOSED DUPLICATE of bug 989695
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Paul Morie
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-29 06:10 EDT by Meng Bo
Modified: 2015-05-14 19:25 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-30 17:03:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Meng Bo 2013-07-29 06:10:40 EDT
Description of problem:
Create apps on devenv-stage ami, upgrade the instance to devenv_3574, and run migration script, all the gears are passed the migration, but the ones which apache derived will return 503 when accessing.

And app cannot be accessed after restart and reload app.
r
Version-Release number of selected component (if applicable):
devenv-stage_426 to devenv_3574

How reproducible:
always

Steps to Reproduce:
1.Create apps on devenv-stage
2.Upgrade to devenv_3574 and run migration script with option --ignore-cartridge-version
#oo-admin-upgrade --version 2.0.31 --ignore-cartridge-version
3.Check the app via access them

Actual results:
Appache derived apps will return 503 page when accessing.

Expected results:
App can be accessed after migration.


Additional info:
Migrating app 'ruby181' gear 'ruby181' with uuid '0d693a94f82e11e2b9a022000aa4644e' on node 'ip-10-164-100-78' for user: bmeng@redhat.com
Migrating gear on node with: /usr/sbin/oo-admin-upgrade --login 'bmeng@redhat.com' --upgrade-gear '0d693a94f82e11e2b9a022000aa4644e' --app-name 'ruby181' --version '2.0.31' --ignore-cartridge-version
Upgrading on node...
***time_upgrade_on_node_measured_from_broker=21609***
Upgrade on node output:
 Beginning 2.0.31 upgrade for 0d693a94f82e11e2b9a022000aa4644e
Inspecting gear at /var/lib/openshift/0d693a94f82e11e2b9a022000aa4644e
Pre-upgrade state: started
Marking step inspect_gear_state complete
Refreshing cartridge redhat:ruby:1.8:0.0.4, ignoring cartridge version.
Creating itinerary entry for incompatible upgrade of redhat:ruby:1.8:0.0.4
Marking step compute_itinerary complete
Migrating gear at /var/lib/openshift/0d693a94f82e11e2b9a022000aa4644e
Stopping gear on node 'ip-10-164-100-78'
Marking step stop_gear complete
Incompatible upgrade of cartridge redhat:ruby:1.8:0.0.4
Executed setup for ruby
rc: 
stdout: 
stderr: 
Marking step ruby_setup complete
Marking step ruby_erb complete
No upgrade script exists for ruby; skipping
Marking step ruby_create_endpoints complete
Marking step ruby_connect_frontend complete
Marking step ruby_upgrade complete
Marking step ruby_rebuild_ident complete
Starting gear on node 'ip-10-164-100-78'
Start gear output: Starting Ruby cart

Marking step start_gear complete
Gear restart time (ms): 7741
Validating gear post-upgrade
Pre-upgrade state: started
Post-upgrade response code: 503
Problem detected with gear status.  Post-upgrade status: Cart status for ruby [PROBLEM]: CLIENT_RESULT: Application is either stopped or inaccessible


Marking step validate_gear complete
Cleaning up after upgrade
Total upgrade time on node (ms): 21476
Comment 1 chunchen 2013-07-30 06:36:57 EDT
It's also reproduced for zend app,please refer to the migration results:

Migrating app 'czd0' gear 'czd0' with uuid '51f7715f833b2f2194000001' on node 'ip-10-35-55-125' for user: chunchen
Migrating gear on node with: /usr/sbin/oo-admin-upgrade --login 'chunchen' --upgrade-gear '51f7715f833b2f2194000001' --app-name 'czd0' --version '2.0.31' --ignore-cartridge-version
Upgrading on node...
***time_upgrade_on_node_measured_from_broker=33646***
Upgrade on node output:
 Beginning 2.0.31 upgrade for 51f7715f833b2f2194000001
Inspecting gear at /var/lib/openshift/51f7715f833b2f2194000001
Pre-upgrade state: started
Marking step inspect_gear_state complete
Refreshing cartridge redhat:zend:5.6:0.0.2, ignoring cartridge version.
Creating itinerary entry for incompatible upgrade of redhat:zend:5.6:0.0.2
Marking step compute_itinerary complete
Migrating gear at /var/lib/openshift/51f7715f833b2f2194000001
Stopping gear on node 'ip-10-35-55-125'
Marking step stop_gear complete
Incompatible upgrade of cartridge redhat:zend:5.6:0.0.2
Executed setup for zend
rc: 
stdout: 
stderr: 
Marking step zend_setup complete
Marking step zend_erb complete
No upgrade script exists for zend; skipping
Marking step zend_create_endpoints complete
Marking step zend_connect_frontend complete
Marking step zend_upgrade complete
Marking step zend_rebuild_ident complete
Starting gear on node 'ip-10-35-55-125'
Start gear output: CLIENT_MESSAGE: Starting Zend Server Deployment daemon
Starting Deployment [OK]
[30.07.2013 05:34:50 SYSTEM] watchdog for zdd is running. 
[30.07.2013 05:34:50 SYSTEM] zdd is running. 
CLIENT_MESSAGE: Starting Zend Server Monitor node
Starting Zend Server Monitor node [OK]
[30.07.2013 05:34:51 SYSTEM] watchdog for monitor is running. 
[30.07.2013 05:34:51 SYSTEM] monitor is running. 
CLIENT_MESSAGE: Starting Apache
CLIENT_MESSAGE: Starting Zend Server JobQueue daemon
Starting JobQueue [OK]
[30.07.2013 05:34:53 SYSTEM] watchdog for jqd is running. 
[30.07.2013 05:34:53 SYSTEM] jqd is running. 
CLIENT_MESSAGE: Starting Zend Server Console
spawn-fcgi: child spawned successfully: PID: 28112
Starting Zend Server GUI [Lighttpd] [OK]
[30.07.2013 05:34:55 SYSTEM] watchdog for lighttpd is running. 
[30.07.2013 05:34:55 SYSTEM] lighttpd is running. 
CLIENT_RESULT: Zend Server 5.6 started successfully

Marking step start_gear complete
Gear restart time (ms): 6857
Validating gear post-upgrade
Pre-upgrade state: started
Post-upgrade response code: 503
Problem detected with gear status.  Post-upgrade status: Cart status for zend [PROBLEM]: Application is either stopped or inaccessible


Marking step validate_gear complete
Cleaning up after upgrade
Total upgrade time on node (ms): 32538
gear_upgrade_json={"gear_uuid":"51f7715f833b2f2194000001","hostname":"ip-10-35-55-125","steps":{"inspect_gear_state":{"status":"complete","errors":[],"context":{"preupgrade_state":"started"}},"compute_itinerary":{"status":"complete","errors":[],"context":{}},"stop_gear":{"status":"complete","errors":[],"context":{}},"zend_upgrade":{"status":"complete","errors":[],"context":{"cartridge":"zend","compatible":false}},"zend_setup":{"status":"complete","errors":[],"context":{"cartridge":"zend","stdout":""}},"zend_erb":{"status":"complete","errors":[],"context":{"cartridge":"zend"}},"upgrade_script_zend":{"status":"incomplete","errors":[],"context":{}},"zend_create_endpoints":{"status":"complete","errors":[],"context":{"cartridge":"zend"}},"zend_connect_frontend":{"status":"complete","errors":[],"context":{"cartridge":"zend"}},"zend_rebuild_ident":{"status":"complete","errors":[],"context":{"cartridge":"zend"}},"start_gear":{"status":"complete","errors":[],"context":{"output":"CLIENT_MESSAGE: Starting Zend Server Deployment daemon\nStarting Deployment \u001b[32m[OK]\u001b[0m\n[30.07.2013 05:34:50 SYSTEM] watchdog for zdd is running. \n[30.07.2013 05:34:50 SYSTEM] zdd is running. \nCLIENT_MESSAGE: Starting Zend Server Monitor node\nStarting Zend Server Monitor node \u001b[32m[OK]\u001b[0m\n[30.07.2013 05:34:51 SYSTEM] watchdog for monitor is running. \n[30.07.2013 05:34:51 SYSTEM] monitor is running. \nCLIENT_MESSAGE: Starting Apache\nCLIENT_MESSAGE: Starting Zend Server JobQueue daemon\nStarting JobQueue \u001b[32m[OK]\u001b[0m\n[30.07.2013 05:34:53 SYSTEM] watchdog for jqd is running. \n[30.07.2013 05:34:53 SYSTEM] jqd is running. \nCLIENT_MESSAGE: Starting Zend Server Console\nspawn-fcgi: child spawned successfully: PID: 28112\nStarting Zend Server GUI [Lighttpd] \u001b[32m[OK]\u001b[0m\n[30.07.2013 05:34:55 SYSTEM] watchdog for lighttpd is running. \n[30.07.2013 05:34:55 SYSTEM] lighttpd is running. \nCLIENT_RESULT: Zend Server 5.6 started successfully\n"}},"validate_gear":{"status":"complete","errors":[],"context":{"preupgrade_state":"started","postupgrade_response_code":"503","postupgrade_status":"Cart status for zend [PROBLEM]: Application is either stopped or inaccessible\n\n"}}},"upgrade_complete":true,"errors":[],"itinerary_entries":{"zend-5.6":"incompatible"},"times":{"upgrade_on_node_measured_from_node":32538,"restart":6857}}

***time_redeploy_httpd_proxy=0***
***time_restart=1***
***time_total_upgrade_gear_measured_from_broker=33652***
Comment 2 Paul Morie 2013-07-30 17:03:18 EDT
INT upgrades are run with --ignore-cartridge-version, which forces an incompatible upgrade when no new cartridge version exists, which is why so many apps were affected.

*** This bug has been marked as a duplicate of bug 989695 ***

Note You need to log in before you can comment on or make changes to this bug.