Hide Forgot
Created attachment 817343 [details] haproxy-status page. Description of problem: Scale up the apps in INT after migration, found some gears are always in deploying state. Version-Release number of selected component (if applicable): INT(devenv_3965) How reproducible: sometimes Steps to Reproduce: 1.create apps in INT. 2.upgrade and migrate the server. 3.scale up the apps. 4.check the status via rhc and haproxy-status page. Actual results: # rhc app show jbossews2s --gears ID State Cartridges Size SSH URL ------------------------ --------- -------------------------------------------------- ----- ----------------------------------------------------------------------------- 5264c9876cec0e4ccb000372 started mysql-5.1 small 5264c9876cec0e4ccb000372.rhcloud.com 5264c9886cec0e4ccb00037f started postgresql-8.4 small 5264c9886cec0e4ccb00037f.rhcloud.com 5264c9886cec0e4ccb00038b started mongodb-2.2 small 5264c9886cec0e4ccb00038b.rhcloud.com 5264c9876cec0e4ccb00036e started jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5264c9876cec0e4ccb00036e.rhcloud.com 5264cbff6cec0e4ccb00040a started jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5264cbff6cec0e4ccb00040a.rhcloud.com 5270afeb6cec0e5b9d000026 deploying jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5270afeb6cec0e5b9d000026.rhcloud.com 5270afeb6cec0e5b9d000027 deploying jbossews-2.0 haproxy-1.4 cron-1.4 jenkins-client-1 small 5270afeb6cec0e5b9d000027.rhcloud.com And the haproxy-status page only show 1 gear up. You can refer to the attachment screenshot.png for detail. Expected results: App scale up successfully, and the status should be consistent via rhc and haproxy-status page. Additional info: Scaling on server which in migrated from devenv_stage_528 to devenv_3966 works fine.
We believe this has already been fixed. What should happen next is keep testing in devenv and reverify once we hit stage.
Created attachment 817823 [details] development_log_for_jboss_scaleup Checked on devenv_3974, still meet problem for scalable jboss app after migration. Scalable jbosseap app cannot be scaled-up (via rhc cartridge-scale) after migration. And after that, only one local-gear shows in haproxy-status page. # rhc cartridge-scale jboss 3 -a jbeap2g ... Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ip-10-155-146-187. This indicates an unexpected problem during the execution of your request.. Found the mcollective-server log: E, [2013-10-31T06:26:50.904287 #28565] ERROR -- : openshift.rb:312:in `rescue in with_container_from_args' Activation of new gears failed: e32d712c421611e3be5c22000a9b92bb: Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap E, [2013-10-31T06:26:50.904562 #28565] ERROR -- : openshift.rb:313:in `rescue in with_container_from_args' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1289:in `update_cluster' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:1000:in `block in oo_update_cluster' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:301:in `with_container_from_args' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:999:in `oo_update_cluster' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:139:in `execute_action' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:104:in `cartridge_do_action' /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg' /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch' I, [2013-10-31T06:26:50.904917 #28565] INFO -- : openshift.rb:150:in `execute_action' Finished executing action [update-cluster] (1) I, [2013-10-31T06:26:50.952993 #28565] INFO -- : openshift.rb:114:in `cartridge_do_action' cartridge_do_action failed (1) ------ Activation of new gears failed: e32d712c421611e3be5c22000a9b92bb: Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap Following error in platform.log October 31 06:26:50 INFO Activation result for gears e32d712c421611e3be5c22000a9b92bb: {:status=>"failure", :gear_results=>{"e32d712c421611e3be5c22000a9b92bb"=>{"status"=>"failure", "messages"=>["Activating gear e32d712c421611e3be5c22000a9b92bb, deployment id: 87a1ec1e, --post-install\n", "Starting application jbeap2g"], "errors"=>["Error activating gear: Failed to execute: 'control deploy' for /var/lib/openshift/e32d712c421611e3be5c22000a9b92bb/jbosseap", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1266:in `block in do_control_with_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1067:in `process_cartridges'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1229:in `do_control_with_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/v2_cart_model.rb:1089:in `do_control'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:850:in `activate_local_gear'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:710:in `block in activate'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1013:in `rotate_and_yield'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.9/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:977:in `block in with_gear_rotation'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call_with_index'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:188:in `block (3 levels) in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:352:in `with_instrumentation'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:186:in `block (2 levels) in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `block in work_in_threads'", "/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:65:in `block (2 levels) in in_threads'"], "gear_uuid"=>"e32d712c421611e3be5c22000a9b92bb", "deployment_id"=>"87a1ec1e"}}} development.log attached. Not sure if the issue is fixed or not, paste the log for further investigating.
https://github.com/openshift/li/pull/2073
Commit pushed to master at https://github.com/openshift/li https://github.com/openshift/li/commit/5a90f447e5e942fde3e5bd2e2d3d2631b3a3d1f7 Fix dependency paths for jboss carts. Bug 1024678 Bug 1024752
Checked upgrade devenv-stage_528 to latest stage branch. The scalable jboss apps (and also for other cartridges) can be scaled-up successfully now. And the haproxy-status page can show the scaled-up gears correctly. Move bug to verified.