Bug 1024678 - The scaled up gear of some apps is always in deploying state
The scaled up gear of some apps is always in deploying state
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Andy Goldstein
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-30 04:50 EDT by Liang Xia
Modified: 2015-05-14 19:32 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-23 22:27:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
haproxy-status page. (85.86 KB, image/png)
2013-10-30 04:50 EDT, Liang Xia
no flags Details
development_log_for_jboss_scaleup (24.96 KB, text/plain)
2013-10-31 06:35 EDT, Meng Bo
no flags Details

  None (edit)
Description Liang Xia 2013-10-30 04:50:08 EDT
Created attachment 817343 [details]
haproxy-status page.

Description of problem:
Scale up the apps in INT after migration, found some gears are always in deploying state.

Version-Release number of selected component (if applicable):
INT(devenv_3965)

How reproducible:
sometimes

Steps to Reproduce:
1.create apps in INT.
2.upgrade and migrate the server.
3.scale up the apps.
4.check the status via rhc and haproxy-status page.


Actual results:
# rhc app show jbossews2s --gears
ID                       State     Cartridges                                         Size  SSH URL
------------------------ --------- -------------------------------------------------- ----- -----------------------------------------------------------------------------
5264c9876cec0e4ccb000372 started   mysql-5.1                                          small 5264c9876cec0e4ccb000372@5264c9876cec0e4ccb000372-lxiamigrate.int.rhcloud.com
5264c9886cec0e4ccb00037f started   postgresql-8.4                                     small 5264c9886cec0e4ccb00037f@5264c9886cec0e4ccb00037f-lxiamigrate.int.rhcloud.com
5264c9886cec0e4ccb00038b started   mongodb-2.2                                        small 5264c9886cec0e4ccb00038b@5264c9886cec0e4ccb00038b-lxiamigrate.int.rhcloud.com
5264c9876cec0e4ccb00036e started   jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5264c9876cec0e4ccb00036e@jbossews2s-lxiamigrate.int.rhcloud.com
5264cbff6cec0e4ccb00040a started   jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5264cbff6cec0e4ccb00040a@5264cbff6cec0e4ccb00040a-lxiamigrate.int.rhcloud.com
5270afeb6cec0e5b9d000026 deploying jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5270afeb6cec0e5b9d000026@5270afeb6cec0e5b9d000026-lxiamigrate.int.rhcloud.com
5270afeb6cec0e5b9d000027 deploying jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5270afeb6cec0e5b9d000027@5270afeb6cec0e5b9d000027-lxiamigrate.int.rhcloud.com

And the haproxy-status page only show 1 gear up. You can refer to the attachment screenshot.png for detail.

Expected results:
App scale up successfully, and the status should be consistent via rhc and haproxy-status page.

Additional info:
Scaling on server which in migrated from devenv_stage_528 to devenv_3966 works fine.
Comment 1 Dan McPherson 2013-10-30 13:03:34 EDT
We believe this has already been fixed.  What should happen next is keep testing in devenv and reverify once we hit stage.
Comment 2 Meng Bo 2013-10-31 06:35:38 EDT
Created attachment 817823 [details]
development_log_for_jboss_scaleup

Checked on devenv_3974, still meet problem for scalable jboss app after migration.

Scalable jbosseap app cannot be scaled-up (via rhc cartridge-scale) after migration.

And after that, only one local-gear shows in haproxy-status page.

# rhc cartridge-scale jboss 3 -a jbeap2g
...
Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ip-10-155-146-187.  This indicates an unexpected problem
during the execution of your request..


Found the mcollective-server log:

E, [2013-10-31T06:26:50.904287 #28565] ERROR -- : openshift.rb:312:in `rescue in with_container_from_args' Activation of new gears failed: e32d712c421611e3be5c22000a9b92bb: Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap
E, [2013-10-31T06:26:50.904562 #28565] ERROR -- : openshift.rb:313:in `rescue in with_container_from_args' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1289:in `update_cluster'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:1000:in `block in oo_update_cluster'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:301:in `with_container_from_args'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:999:in `oo_update_cluster'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:139:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
I, [2013-10-31T06:26:50.904917 #28565]  INFO -- : openshift.rb:150:in `execute_action' Finished executing action [update-cluster] (1)
I, [2013-10-31T06:26:50.952993 #28565]  INFO -- : openshift.rb:114:in `cartridge_do_action' cartridge_do_action failed (1)
------
Activation of new gears failed: e32d712c421611e3be5c22000a9b92bb: Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap


Following error in platform.log
October 31 06:26:50 INFO Activation result for gears e32d712c421611e3be5c22000a9b92bb: {:status=>"failure", :gear_results=>{"e32d712c421611e3be5c22000a9b92bb"=>{"status"=>"failure", "messages"=>["Activating gear e32d712c421611e3be5c22000a9b92bb, deployment id: 87a1ec1e, --post-install\n", "Starting application jbeap2g"], "errors"=>["Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1266:in `block in do_control_with_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `process_cartridges'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1229:in `do_control_with_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1089:in `do_control'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:850:in `activate_local_gear'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:710:in `block in activate'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1013:in `rotate_and_yield'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:977:in `block in with_gear_rotation'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call_with_index'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:188:in `block (3 levels) in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:352:in `with_instrumentation'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:186:in `block (2 levels) in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `block in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:65:in `block (2 levels) in in_threads'"], "gear_uuid"=>"e32d712c421611e3be5c22000a9b92bb", "deployment_id"=>"87a1ec1e"}}}


development.log attached.


Not sure if the issue is fixed or not, paste the log for further investigating.
Comment 3 Andy Goldstein 2013-10-31 13:51:01 EDT
https://github.com/openshift/li/pull/2073
Comment 4 openshift-github-bot 2013-10-31 15:42:25 EDT
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/5a90f447e5e942fde3e5bd2e2d3d2631b3a3d1f7
Fix dependency paths for jboss carts.

Bug 1024678
Bug 1024752
Comment 5 Meng Bo 2013-10-31 23:31:22 EDT
Checked upgrade devenv-stage_528 to latest stage branch.

The scalable jboss apps (and also for other cartridges) can be scaled-up successfully now.
And the haproxy-status page can show the scaled-up gears correctly.

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.