| Summary: | The scaled up gear of some apps is always in deploying state | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Liang Xia <lxia> | ||||||
| Component: | Containers | Assignee: | Andy Goldstein <agoldste> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 2.x | CC: | agoldste, bmeng, dmcphers, xtian | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2014-01-24 03:27:45 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Attachments: |
|
||||||||
We believe this has already been fixed. What should happen next is keep testing in devenv and reverify once we hit stage. Created attachment 817823 [details]
development_log_for_jboss_scaleup
Checked on devenv_3974, still meet problem for scalable jboss app after migration.
Scalable jbosseap app cannot be scaled-up (via rhc cartridge-scale) after migration.
And after that, only one local-gear shows in haproxy-status page.
# rhc cartridge-scale jboss 3 -a jbeap2g
...
Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ip-10-155-146-187. This indicates an unexpected problem
during the execution of your request..
Found the mcollective-server log:
E, [2013-10-31T06:26:50.904287 #28565] ERROR -- : openshift.rb:312:in `rescue in with_container_from_args' Activation of new gears failed: e32d712c421611e3be5c22000a9b92bb: Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap
E, [2013-10-31T06:26:50.904562 #28565] ERROR -- : openshift.rb:313:in `rescue in with_container_from_args' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1289:in `update_cluster'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:1000:in `block in oo_update_cluster'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:301:in `with_container_from_args'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:999:in `oo_update_cluster'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:139:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
I, [2013-10-31T06:26:50.904917 #28565] INFO -- : openshift.rb:150:in `execute_action' Finished executing action [update-cluster] (1)
I, [2013-10-31T06:26:50.952993 #28565] INFO -- : openshift.rb:114:in `cartridge_do_action' cartridge_do_action failed (1)
------
Activation of new gears failed: e32d712c421611e3be5c22000a9b92bb: Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap
Following error in platform.log
October 31 06:26:50 INFO Activation result for gears e32d712c421611e3be5c22000a9b92bb: {:status=>"failure", :gear_results=>{"e32d712c421611e3be5c22000a9b92bb"=>{"status"=>"failure", "messages"=>["Activating gear e32d712c421611e3be5c22000a9b92bb, deployment id: 87a1ec1e, --post-install\n", "Starting application jbeap2g"], "errors"=>["Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1266:in `block in do_control_with_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `process_cartridges'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1229:in `do_control_with_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1089:in `do_control'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:850:in `activate_local_gear'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:710:in `block in activate'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1013:in `rotate_and_yield'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:977:in `block in with_gear_rotation'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call_with_index'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:188:in `block (3 levels) in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:352:in `with_instrumentation'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:186:in `block (2 levels) in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `block in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:65:in `block (2 levels) in in_threads'"], "gear_uuid"=>"e32d712c421611e3be5c22000a9b92bb", "deployment_id"=>"87a1ec1e"}}}
development.log attached.
Not sure if the issue is fixed or not, paste the log for further investigating.
Commit pushed to master at https://github.com/openshift/li https://github.com/openshift/li/commit/5a90f447e5e942fde3e5bd2e2d3d2631b3a3d1f7 Fix dependency paths for jboss carts. Bug 1024678 Bug 1024752 Checked upgrade devenv-stage_528 to latest stage branch. The scalable jboss apps (and also for other cartridges) can be scaled-up successfully now. And the haproxy-status page can show the scaled-up gears correctly. Move bug to verified. |
Created attachment 817343 [details] haproxy-status page. Description of problem: Scale up the apps in INT after migration, found some gears are always in deploying state. Version-Release number of selected component (if applicable): INT(devenv_3965) How reproducible: sometimes Steps to Reproduce: 1.create apps in INT. 2.upgrade and migrate the server. 3.scale up the apps. 4.check the status via rhc and haproxy-status page. Actual results: # rhc app show jbossews2s --gears ID State Cartridges Size SSH URL ------------------------ --------- -------------------------------------------------- ----- ----------------------------------------------------------------------------- 5264c9876cec0e4ccb000372 started mysql-5.1 small 5264c9876cec0e4ccb000372.rhcloud.com 5264c9886cec0e4ccb00037f started postgresql-8.4 small 5264c9886cec0e4ccb00037f.rhcloud.com 5264c9886cec0e4ccb00038b started mongodb-2.2 small 5264c9886cec0e4ccb00038b.rhcloud.com 5264c9876cec0e4ccb00036e started jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5264c9876cec0e4ccb00036e.rhcloud.com 5264cbff6cec0e4ccb00040a started jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5264cbff6cec0e4ccb00040a.rhcloud.com 5270afeb6cec0e5b9d000026 deploying jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5270afeb6cec0e5b9d000026.rhcloud.com 5270afeb6cec0e5b9d000027 deploying jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5270afeb6cec0e5b9d000027.rhcloud.com And the haproxy-status page only show 1 gear up. You can refer to the attachment screenshot.png for detail. Expected results: App scale up successfully, and the status should be consistent via rhc and haproxy-status page. Additional info: Scaling on server which in migrated from devenv_stage_528 to devenv_3966 works fine.