Bug 833543 - Can't build success after move haproxy gear of all scalable -app
Can't build success after move haproxy gear of all scalable -app
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Dan Mace
libra bugs
: TestBlocker, Triaged
Depends On: 832412
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-19 14:00 EDT by Dan Mace
Modified: 2015-05-14 18:55 EDT (History)
7 users (show)

See Also:
Fixed In Version: devenv_1937+
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 832412
Environment:
Last Closed: 2012-08-07 16:42:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
JBosseap jenkins log (6.18 KB, text/plain)
2012-06-19 14:01 EDT, Dan Mace
no flags Details
move log from development.log (26.02 KB, text/plain)
2012-07-30 06:06 EDT, Rony Gong
no flags Details
latest move log. (18.11 KB, text/plain)
2012-08-03 05:52 EDT, Rony Gong
no flags Details

  None (edit)
Description Dan Mace 2012-06-19 14:00:33 EDT
+++ This bug was initially created as a clone of Bug #832412 +++

This issue is split off to be specific to the jbosseap cart.

Created attachment 592103 [details]
jenkins log

Description of problem:



Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1.Create scalable jbosseap application, with jenkins embed
2.do move(move the jboss gear) within one district

  
Actual results:
After move, can't build success.

Expected results:
build success.

Additional info:
attachment is jenkins log.
--- Additional comment from qgong@redhat.com on 2012-06-19 06:24:24 EDT ---

I retest it on devenv_1857
1.create mulit node evn with 2 small nodes(dist1), 1 medium node(dist2)
2.create an scalable jbosseap, embed jenkins, could build success
3.move the jbosseap gear(not proxy gear),check, the proxy is down, can't access the proxy page, 503 error, but could access the jbosseap gear page.

below is the move log from development.log:

DEBUG: Cartridge command jbosseap-6.0::status exitcode = 0
DEBUG: Gear component 'jbosseap-6.0' was running
DEBUG: Stopping existing app cartridge 'jbosseap-6.0' before moving
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a0878b8>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"stop"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a05f340 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"ATTR: status=ALREADY_STOPPED\n", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Cartridge command jbosseap-6.0::stop exitcode = 0
DEBUG: Force stopping existing app cartridge 'jbosseap-6.0' before moving
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a05af70>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"force-stop"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a49d9c0 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Cartridge command jbosseap-6.0::force-stop exitcode = 0
DEBUG: Creating new account for gear 'd2a715955a' on ip-10-124-189-58
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a495950>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-uid"=>1027, "--with-namespace"=>"qgong1", "--with-app-name"=>"qs2jbosseap", "--with-quota-files"=>"40000", "--with-app-uuid"=>"b324a4c07cdc440a9dfba45e392aadc4", "--with-quota-blocks"=>"1048576", "--with-container-name"=>"d2a715955a", "--with-container-uuid"=>"d2a715955ab945a79f1376b58eac482c"}, :cartridge=>"stickshift-node", :action=>"app-create"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a4442f8 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Moving content for app 'qs2jbosseap', gear 'd2a715955a' to ip-10-124-189-58
Agent pid 11090
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 11090 killed;
DEBUG: Performing cartridge level move for 'jbosseap-6.0' on ip-10-124-189-58
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3f66c0>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"move"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a3c4968 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Cartridge command jbosseap-6.0::move exitcode = 0
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3c1d58>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"expose-port"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a389020 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"CART_DATA: PROXY_HOST=d2a715955a-qgong1.dev.rhcloud.com\nCART_DATA: PROXY_PORT=38166\nCART_DATA: HOST=127.2.1.129\nCART_DATA: PORT=8080\nCART_DATA: JBOSS_CLUSTER_PROXY_HOST=d2a715955a-qgong1.dev.rhcloud.com\nCART_DATA: JBOSS_CLUSTER_PROXY_PORT=38167\nCART_DATA: JBOSS_CLUSTER_HOST=127.2.1.129\nCART_DATA: JBOSS_CLUSTER_PORT=7600\nCART_DATA: JBOSS_MESSAGING_PORT=38168\nCART_DATA: JBOSS_MESSAGING_THROUGHPUT_PORT=38169\nCART_DATA: JBOSS_REMOTING_PORT=38170\n", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Cartridge command jbosseap-6.0::expose-port exitcode = 0
DEBUG: Starting cartridge 'jbosseap-6.0' in 'qs2jbosseap' after move on ip-10-124-189-58
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3834e0>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"start"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a2f75f8 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Cartridge command jbosseap-6.0::start exitcode = 0
DEBUG: Fixing DNS and mongo for gear 'd2a715955a' after move
DEBUG: Changing server identity of 'd2a715955a' from 'ip-10-92-241-213' to 'ip-10-124-189-58'
DEBUG: DYNECT Login with path: /REST/Session/
DEBUG: DYNECT Response: {"status": "success", "data": {"token": "kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ=", "version": "3.3.3"}, "job_id": 118300044, "msgs": [{"INFO": "login: Login successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}
DEBUG: DYNECT Response data: version3.3.3tokenkQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ=
DEBUG: DYNECT delete with path: /REST/CNAMERecord/rhcloud.com/d2a715955a-qgong1.dev.rhcloud.com/ and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="}
DEBUG: DYNECT Response: {"status": "success", "data": {}, "job_id": 118300046, "msgs": [{"INFO": "delete: 1 records deleted", "SOURCE": "API-B", "ERR_CD": null, "LVL": "INFO"}]}
DEBUG: DYNECT Response data: 
DEBUG: Public ip being configured 'ec2-23-22-228-189.compute-1.amazonaws.com' to app 'd2a715955a'
DEBUG: DYNECT put/post with path: /REST/CNAMERecord/rhcloud.com/d2a715955a-qgong1.dev.rhcloud.com/ json data: {"rdata":{"cname":"ec2-23-22-228-189.compute-1.amazonaws.com"},"ttl":"60"} and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="}
DEBUG: DYNECT Response: {"status": "success", "data": {"zone": "rhcloud.com", "ttl": 60, "fqdn": "d2a715955a-qgong1.dev.rhcloud.com", "record_type": "CNAME", "rdata": {"cname": "ec2-23-22-228-189.compute-1.amazonaws.com."}, "record_id": 0}, "job_id": 118300059, "msgs": [{"INFO": "add: Record added", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}
DEBUG: DYNECT Response data: record_id0zonerhcloud.comrdatacnameec2-23-22-228-189.compute-1.amazonaws.com.record_typeCNAMEfqdnd2a715955a-qgong1.dev.rhcloud.comttl60
DEBUG: DYNECT put/post with path: /REST/Zone/rhcloud.com/ json data: {"publish":"true"} and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="}
DEBUG: DYNECT Response: {"status": "success", "data": {"zone_type": "Primary", "serial_style": "increment", "serial": 980682, "zone": "rhcloud.com"}, "job_id": 118300067, "msgs": [{"INFO": "publish: rhcloud.com published", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}
DEBUG: DYNECT Response data: zonerhcloud.comserial980682zone_typePrimaryserial_styleincrement
DEBUG: DYNECT delete with path: /REST/Session/ and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="}
DEBUG: DYNECT Response: {"status": "success", "data": {}, "job_id": 118300085, "msgs": [{"INFO": "logout: Logout successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}
DEBUG: DYNECT Response data: 
DEBUG: Output of parallel execute: result_stdoutqs2jbosseap-qgong1.dev.rhcloud.com:38162
jobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4--hook-namepublish_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
Output of publisher - ''b324a4c07cdc440a9dfba45e392aadc4'='qs2jbosseap-qgong1.dev.rhcloud.com:38162
''
DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'b324a4c07cdc440a9dfba45e392aadc4\'\=\'qs2jbosseap-qgong1.dev.rhcloud.com:38162'
'\'--hook-nameset_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
DEBUG: Output of parallel execute: result_stdoutqs2jbosseap-qgong1.dev.rhcloud.com:38165
jobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4--hook-namepublish_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
Output of publisher - ''b324a4c07cdc440a9dfba45e392aadc4'='qs2jbosseap-qgong1.dev.rhcloud.com:38165
''
DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'b324a4c07cdc440a9dfba45e392aadc4\'\=\'qs2jbosseap-qgong1.dev.rhcloud.com:38165'
'\'--hook-nameset_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
DEBUG: Output of parallel execute: result_stdoutd2a715955a-qgong1.dev.rhcloud.com:38167
jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0
Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955a-qgong1.dev.rhcloud.com:38167
''
DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955a-qgong1.dev.rhcloud.com:38167'
'\'--hook-nameset_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0
DEBUG: Output of parallel execute: result_stdoutd2a715955a-qgong1.dev.rhcloud.com:38170
jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0
Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955a-qgong1.dev.rhcloud.com:38170
''
DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955a-qgong1.dev.rhcloud.com:38170'
'\'--hook-nameset_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0
DEBUG: Output of parallel execute: result_stdoutCould not find /usr/libexec/stickshift/cartridges/jenkins-client-1.4/info/connection-hooks/colocate-with-haproxy-hackjobargs--cart-namejenkins-client-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4--hook-namecolocate-with-haproxy-hackcartridgestickshift-nodeactionconnector-executeresult_stderrCould not find /usr/libexec/stickshift/cartridges/jenkins-client-1.4/info/connection-hooks/colocate-with-haproxy-hackresult_exit_code127gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
Output of publisher - ''
DEBUG: Output of parallel execute: result_stdoutCould not find /usr/libexec/stickshift/cartridges/haproxy-1.4/info/connection-hooks/haproxy-colocation-hackjobargs--cart-namehaproxy-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 ''--hook-namehaproxy-colocation-hackcartridgestickshift-nodeactionconnector-executeresult_stderrCould not find /usr/libexec/stickshift/cartridges/haproxy-1.4/info/connection-hooks/haproxy-colocation-hackresult_exit_code127gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
DEBUG: Output of parallel execute: result_stdoutd2a715955ab945a79f1376b58eac482c@10.124.189.58:d2a715955a;d2a715955a-qgong1.dev.rhcloud.com
jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish-gear-endpointcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0
Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955ab945a79f1376b58eac482c@10.124.189.58:d2a715955a;d2a715955a-qgong1.dev.rhcloud.com
''
DEBUG: Output of parallel execute: result_stdoutTue Jun 19 06:10:23 EDT 2012: Reloading haproxy service 
jobargs--cart-namehaproxy-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955ab945a79f1376b58eac482c@10.124.189.58:d2a715955a\;d2a715955a-qgong1.dev.rhcloud.com'
'\'--hook-nameset-gear-endpointscartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
DEBUG: Output of parallel execute: result_stdoutd2a715955a-qgong1.dev.rhcloud.com|10.124.189.58:38166
jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish-http-urlcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0
Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955a-qgong1.dev.rhcloud.com|10.124.189.58:38166
''
DEBUG: Output of parallel execute: result_stdoutTue Jun 19 06:10:30 EDT 2012: Reloading haproxy service 
jobargs--cart-namehaproxy-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955a-qgong1.dev.rhcloud.com\|10.124.189.58:38166'
'\'--hook-nameset-proxycartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0
MongoDataStore.save(Application, qgong@redhat.com, qs2jbosseap, #hidden)

DEBUG: Deconfiguring old app 'qs2jbosseap' on ip-10-92-241-213 after move
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a411808>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"deconfigure"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a3c7910 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
DEBUG: Cartridge command jbosseap-6.0::deconfigure exitcode = 0
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3c5610>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-app-uuid"=>"b324a4c07cdc440a9dfba45e392aadc4", "--with-container-uuid"=>"d2a715955ab945a79f1376b58eac482c"}, :cartridge=>"stickshift-node", :action=>"app-destroy"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fc09a394d30 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>]
Successfully moved 'qs2jbosseap' with gear uuid 'd2a715955ab945a79f1376b58eac482c' from 'ip-10-92-241-213' to 'ip-10-124-189-58'


Started POST "/broker/userinfo" for 203.114.244.88 at Tue Jun 19 06:11:39 -0400 2012
  Processing by LegacyBrokerController#user_info_post as 
  Parameters: {"json_data"=>"{\"api\":\"1.1.3\",\"rhlogin\":\"qgong@redhat.com\"}", "password"=>"[FILTERED]"}
User-Agent = ''
MongoDataStore.find(CloudUser, qgong@redhat.com, qgong@redhat.com)

[REQ_ID=c0ce0baba1bf481a94b234705c8b4d71] ACTION=LEGACY_USER_INFO 
Completed 200 OK in 156ms (Views: 3.1ms)

--- Additional comment from qgong@redhat.com on 2012-06-19 06:26:46 EDT ---

Created attachment 592902 [details]
new jenkins_log of after move proxy gear
Comment 1 Dan Mace 2012-06-19 14:01:21 EDT
Created attachment 593028 [details]
JBosseap jenkins log
Comment 2 Johnny Liu 2012-06-20 08:36:06 EDT
Re-test this bug on devenv_1589, and failed. 

Here are reproduce steps:
1. Setup a multi-nodes, env, here I create two disricts, one is "small" that have two nodes, another one is "medium" that have one node.
2. Create a scalable jbosseap app with jenkins client embeded.
3. Move haproxy gear of this app from one node to another one.
4. Do some change to app, then git push to trigger jenkins build.


$ git commit -a -mx; git push
[master a183ac0] x
 1 file changed, 1 insertion(+), 1 deletion(-)
Warning: Permanently added 'eap1-jialiu.dev.rhcloud.com,23.22.247.24' (RSA) to the list of known hosts.
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 270 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins-jialiu.dev.rhcloud.com/job/eap1-build
remote: 
remote: Waiting for build to schedule.....................................................................................Done
remote: Waiting for job to complete................................................Done
remote: FAILED
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
To ssh://54cb91134d734397a4ef82cca7a8d231@eap1-jialiu.dev.rhcloud.com/~/git/eap1.git/
   5d4c0a3..a183ac0  master -> master


Seen console output in jenkins job url, get the following info:
...snip...
SSH_CMD: ssh d7cad83a53844a22b95ac80a2d8f55b4@10.70.85.199
++ date
+ echo 'Syncing to gear: d7cad83a53844a22b95ac80a2d8f55b4@10.70.85.199:d7cad83a53 @ ' Wed Jun 20 08:24:38 EDT 2012
Syncing to gear: d7cad83a53844a22b95ac80a2d8f55b4@10.70.85.199:d7cad83a53 @  Wed Jun 20 08:24:38 EDT 2012
+ for rpccall in '"${OPENSHIFT_SYNC_GEARS_PRE[@]}"'
+ ssh d7cad83a53844a22b95ac80a2d8f55b4@10.70.85.199 'ctl_all stop'
Warning: Identity file /var/lib/stickshift/54cb91134d734397a4ef82cca7a8d231//haproxy-1.4/.ssh/haproxy_id_rsa not accessible: Permission denied.
Exit code: 1
Build step 'Execute shell' marked build as failure

Archiving artifacts

Finished: FAILURE
Comment 3 Rony Gong 2012-06-21 04:59:45 EDT
This error happens for all applications.

maybe, the move action changed the selinux for the haproxy_id_rsa file
, you can see it change from "libra_var_lib_t" to "object_r:ssh_home_t"

before move:
[root@ip-10-120-206-123 stickshift]# ls -Z  /var/lib/stickshift/e53a3a86127f4de487d929d2212ab533/haproxy-1.4/.ssh/haproxy_id_rsa
-rw-r-----. root e53a3a86127f4de487d929d2212ab533 unconfined_u:object_r:libra_var_lib_t:s0:c1,c21 /var/lib/stickshift/e53a3a86127f4de487d929d2212ab533/haproxy-1.4/.ssh/haproxy_id_rsa


after move:
[root@ip-10-120-206-123 stickshift]# ls -Z /var/lib/stickshift/9a6666b3889c4984aa34dabc55394f6c//haproxy-1.4/.ssh/haproxy_id_rsa
-rw-r-----. root 9a6666b3889c4984aa34dabc55394f6c unconfined_u:object_r:ssh_home_t:s0:c1,c17 /var/lib/stickshift/9a6666b3889c4984aa34dabc55394f6c//haproxy-1.4/.ssh/haproxy_id_rsa


so when jenkins build, it show error:
 Identity file /var/lib/stickshift/9a6666b3889c4984aa34dabc55394f6c//haproxy-1.4/.ssh/haproxy_id_rsa not accessible: Permission denied.
Exit code: 1
Comment 4 Rony Gong 2012-07-30 05:43:32 EDT
This error still exist on devenv_1920,almost I can't build success after move scalable app gear, caused by permission deny.

I record the file permission by "ls -Z"
before move:
-rw-r-----. root 69744a2d81c4413ea9e1e1714c3dd378 unconfined_u:object_r:libra_var_lib_t:s0:c1,c12 69744a2d81c4413ea9e1e1714c3dd378/haproxy-1.4/.ssh/haproxy_id_rsa


after move:
-rw-r-----. root 69744a2d81c4413ea9e1e1714c3dd378 unconfined_u:object_r:ssh_home_t:s0:c1,c12 69744a2d81c4413ea9e1e1714c3dd378/haproxy-1.4/.ssh/haproxy_id_rsa
Comment 5 Rony Gong 2012-07-30 06:06:51 EDT
Created attachment 601195 [details]
move log from development.log
Comment 6 Rony Gong 2012-07-30 06:31:20 EDT
This error happened for all scalable app gear(doesn't happens for non-scalable app), whether jenkins client embed or not will cause this error.
Comment 7 Dan Mace 2012-07-30 17:56:55 EDT
Root cause is believed to be bug 842823, which has been re-opened in response to this report.
Comment 8 Rob Millner 2012-07-30 20:01:03 EDT
The selinux configuration for 842823 apparently only works on recent Fedora installs; not RHEL 6.  842823's work-around was to run rhc-restrecon.

You may have to borrow from the rhc-restorecon behaviour in order to properly configure selinux after a gear move.  We may not be able to resolve 842823 beyond re-running rhc-restorecon after the system has done an selinux relabel.
Comment 9 Dan Mace 2012-07-31 10:24:03 EDT
Removing dependency on bug 842823 as the fix is not portable to this context per Rob Millner's comments.
Comment 10 John Poelstra 2012-07-31 13:21:30 EDT
Should have fix in today
Comment 11 Dan Mace 2012-07-31 17:01:42 EDT
Resolved by https://github.com/openshift/crankcase/pull/306
Comment 12 Rony Gong 2012-08-01 05:10:21 EDT
I check your patch has get in build devenv_1931, but still fail as same error, no change.
Comment 13 Chandrika Gole 2012-08-01 14:37:28 EDT
Dan cannot reproduce this on his environment. I am trying to reproduce this.
Comment 14 Chandrika Gole 2012-08-01 16:35:51 EDT
verified this on devenv_1932. Cannot reproduce the error. Tried this with a scalable php and a jboss app 


1. Set up two nodes in a multi-node configuration
2. Created a district, added both nodes to the district
3. Created a scalable php-5.3 application
4. Created a jenkins application
5. Embedded the jenkins-client into the php applicatoin
6. Pushed a change to the php application, resulting in a successful 
Jenkins build
7. Moved the php application's haproxy gear to the opposite node
8. Pushed another change to the php application, resulting in another 
successful Jenkins build
Comment 15 Rony Gong 2012-08-03 05:52:42 EDT
Created attachment 602099 [details]
latest move log.
Comment 16 Rony Gong 2012-08-03 05:53:38 EDT
Reopen it on devenv_1936 and stage_228, only jbossas and jbosseap can't build success after move it's application's haproxy gear.(other app success), error is same as above like: "Identity file /var/lib/stickshift/54cb91134d734397a4ef82cca7a8d231//haproxy-1.4/.ssh/haproxy_id_rsa not accessible: Permission denied."

please see attachment of latest move log.


But I remember I could build success(after move haproxy gear) in devenv_1933, please the patch for those build recently.
Comment 17 Dan Mace 2012-08-03 18:00:53 EDT
Earlier patch was made unstable due to a broker issue which is now resolved by:

https://github.com/openshift/crankcase/pull/333

Prior to the patch in 333, the gear move would break randomly depending on what order the broker happened to perform some of the move operations, explaining why the fix appeared to work sometimes, but not others. Now it will be stable.
Comment 18 Rony Gong 2012-08-06 05:52:54 EDT
verified on stg_229, test jbossas, jbosseap, php with steps:
1. Set up two nodes in a multi-node configuration
2. Created a district, added both nodes to the district
3. Created a scalable php-5.3 application
4. Created a jenkins application
5. Embedded the jenkins-client into the php applicatoin
6. Pushed a change to the php application, resulting in a successful 
Jenkins build
7. Moved the php application's haproxy gear to the opposite node
8. Pushed another change to the php application, resulting in another 
successful Jenkins build

Note You need to log in before you can comment on or make changes to this bug.