+++ This bug was initially created as a clone of Bug #832412 +++ This issue is split off to be specific to the jbosseap cart. Created attachment 592103 [details] jenkins log Description of problem: Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1.Create scalable jbosseap application, with jenkins embed 2.do move(move the jboss gear) within one district Actual results: After move, can't build success. Expected results: build success. Additional info: attachment is jenkins log. --- Additional comment from qgong on 2012-06-19 06:24:24 EDT --- I retest it on devenv_1857 1.create mulit node evn with 2 small nodes(dist1), 1 medium node(dist2) 2.create an scalable jbosseap, embed jenkins, could build success 3.move the jbosseap gear(not proxy gear),check, the proxy is down, can't access the proxy page, 503 error, but could access the jbosseap gear page. below is the move log from development.log: DEBUG: Cartridge command jbosseap-6.0::status exitcode = 0 DEBUG: Gear component 'jbosseap-6.0' was running DEBUG: Stopping existing app cartridge 'jbosseap-6.0' before moving DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a0878b8> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"stop"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a05f340 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"ATTR: status=ALREADY_STOPPED\n", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Cartridge command jbosseap-6.0::stop exitcode = 0 DEBUG: Force stopping existing app cartridge 'jbosseap-6.0' before moving DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a05af70> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"force-stop"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a49d9c0 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Cartridge command jbosseap-6.0::force-stop exitcode = 0 DEBUG: Creating new account for gear 'd2a715955a' on ip-10-124-189-58 DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a495950> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-uid"=>1027, "--with-namespace"=>"qgong1", "--with-app-name"=>"qs2jbosseap", "--with-quota-files"=>"40000", "--with-app-uuid"=>"b324a4c07cdc440a9dfba45e392aadc4", "--with-quota-blocks"=>"1048576", "--with-container-name"=>"d2a715955a", "--with-container-uuid"=>"d2a715955ab945a79f1376b58eac482c"}, :cartridge=>"stickshift-node", :action=>"app-create"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a4442f8 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Moving content for app 'qs2jbosseap', gear 'd2a715955a' to ip-10-124-189-58 Agent pid 11090 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 11090 killed; DEBUG: Performing cartridge level move for 'jbosseap-6.0' on ip-10-124-189-58 DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3f66c0> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"move"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a3c4968 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Cartridge command jbosseap-6.0::move exitcode = 0 DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3c1d58> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"expose-port"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a389020 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"CART_DATA: PROXY_HOST=d2a715955a-qgong1.dev.rhcloud.com\nCART_DATA: PROXY_PORT=38166\nCART_DATA: HOST=127.2.1.129\nCART_DATA: PORT=8080\nCART_DATA: JBOSS_CLUSTER_PROXY_HOST=d2a715955a-qgong1.dev.rhcloud.com\nCART_DATA: JBOSS_CLUSTER_PROXY_PORT=38167\nCART_DATA: JBOSS_CLUSTER_HOST=127.2.1.129\nCART_DATA: JBOSS_CLUSTER_PORT=7600\nCART_DATA: JBOSS_MESSAGING_PORT=38168\nCART_DATA: JBOSS_MESSAGING_THROUGHPUT_PORT=38169\nCART_DATA: JBOSS_REMOTING_PORT=38170\n", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Cartridge command jbosseap-6.0::expose-port exitcode = 0 DEBUG: Starting cartridge 'jbosseap-6.0' in 'qs2jbosseap' after move on ip-10-124-189-58 DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3834e0> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"start"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a2f75f8 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-124-189-58", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Cartridge command jbosseap-6.0::start exitcode = 0 DEBUG: Fixing DNS and mongo for gear 'd2a715955a' after move DEBUG: Changing server identity of 'd2a715955a' from 'ip-10-92-241-213' to 'ip-10-124-189-58' DEBUG: DYNECT Login with path: /REST/Session/ DEBUG: DYNECT Response: {"status": "success", "data": {"token": "kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ=", "version": "3.3.3"}, "job_id": 118300044, "msgs": [{"INFO": "login: Login successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: version3.3.3tokenkQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ= DEBUG: DYNECT delete with path: /REST/CNAMERecord/rhcloud.com/d2a715955a-qgong1.dev.rhcloud.com/ and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="} DEBUG: DYNECT Response: {"status": "success", "data": {}, "job_id": 118300046, "msgs": [{"INFO": "delete: 1 records deleted", "SOURCE": "API-B", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: DEBUG: Public ip being configured 'ec2-23-22-228-189.compute-1.amazonaws.com' to app 'd2a715955a' DEBUG: DYNECT put/post with path: /REST/CNAMERecord/rhcloud.com/d2a715955a-qgong1.dev.rhcloud.com/ json data: {"rdata":{"cname":"ec2-23-22-228-189.compute-1.amazonaws.com"},"ttl":"60"} and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="} DEBUG: DYNECT Response: {"status": "success", "data": {"zone": "rhcloud.com", "ttl": 60, "fqdn": "d2a715955a-qgong1.dev.rhcloud.com", "record_type": "CNAME", "rdata": {"cname": "ec2-23-22-228-189.compute-1.amazonaws.com."}, "record_id": 0}, "job_id": 118300059, "msgs": [{"INFO": "add: Record added", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: record_id0zonerhcloud.comrdatacnameec2-23-22-228-189.compute-1.amazonaws.com.record_typeCNAMEfqdnd2a715955a-qgong1.dev.rhcloud.comttl60 DEBUG: DYNECT put/post with path: /REST/Zone/rhcloud.com/ json data: {"publish":"true"} and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="} DEBUG: DYNECT Response: {"status": "success", "data": {"zone_type": "Primary", "serial_style": "increment", "serial": 980682, "zone": "rhcloud.com"}, "job_id": 118300067, "msgs": [{"INFO": "publish: rhcloud.com published", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: zonerhcloud.comserial980682zone_typePrimaryserial_styleincrement DEBUG: DYNECT delete with path: /REST/Session/ and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"kQXyqtrKBszHoCjAwvDlwlnHvYgGb/a4XtmgganmSgbte5zJCIpWt+rVLdxYVSamnRvNnhVWQNF90uD89X7S/100jwuQ+ftKFZBHeTX/kqCCMThqHcBODmGV8mU1TcroPZ1CiUruEiMKNw3kg+5oE+c0DCITpzTn+bb/B3GycpQ="} DEBUG: DYNECT Response: {"status": "success", "data": {}, "job_id": 118300085, "msgs": [{"INFO": "logout: Logout successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: DEBUG: Output of parallel execute: result_stdoutqs2jbosseap-qgong1.dev.rhcloud.com:38162 jobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4--hook-namepublish_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 Output of publisher - ''b324a4c07cdc440a9dfba45e392aadc4'='qs2jbosseap-qgong1.dev.rhcloud.com:38162 '' DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'b324a4c07cdc440a9dfba45e392aadc4\'\=\'qs2jbosseap-qgong1.dev.rhcloud.com:38162' '\'--hook-nameset_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 DEBUG: Output of parallel execute: result_stdoutqs2jbosseap-qgong1.dev.rhcloud.com:38165 jobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4--hook-namepublish_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 Output of publisher - ''b324a4c07cdc440a9dfba45e392aadc4'='qs2jbosseap-qgong1.dev.rhcloud.com:38165 '' DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'b324a4c07cdc440a9dfba45e392aadc4\'\=\'qs2jbosseap-qgong1.dev.rhcloud.com:38165' '\'--hook-nameset_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 DEBUG: Output of parallel execute: result_stdoutd2a715955a-qgong1.dev.rhcloud.com:38167 jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0 Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955a-qgong1.dev.rhcloud.com:38167 '' DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955a-qgong1.dev.rhcloud.com:38167' '\'--hook-nameset_jboss_clustercartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0 DEBUG: Output of parallel execute: result_stdoutd2a715955a-qgong1.dev.rhcloud.com:38170 jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0 Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955a-qgong1.dev.rhcloud.com:38170 '' DEBUG: Output of parallel execute: result_stdoutjobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955a-qgong1.dev.rhcloud.com:38170' '\'--hook-nameset_jboss_remotingcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0 DEBUG: Output of parallel execute: result_stdoutCould not find /usr/libexec/stickshift/cartridges/jenkins-client-1.4/info/connection-hooks/colocate-with-haproxy-hackjobargs--cart-namejenkins-client-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4--hook-namecolocate-with-haproxy-hackcartridgestickshift-nodeactionconnector-executeresult_stderrCould not find /usr/libexec/stickshift/cartridges/jenkins-client-1.4/info/connection-hooks/colocate-with-haproxy-hackresult_exit_code127gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 Output of publisher - '' DEBUG: Output of parallel execute: result_stdoutCould not find /usr/libexec/stickshift/cartridges/haproxy-1.4/info/connection-hooks/haproxy-colocation-hackjobargs--cart-namehaproxy-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 ''--hook-namehaproxy-colocation-hackcartridgestickshift-nodeactionconnector-executeresult_stderrCould not find /usr/libexec/stickshift/cartridges/haproxy-1.4/info/connection-hooks/haproxy-colocation-hackresult_exit_code127gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 DEBUG: Output of parallel execute: result_stdoutd2a715955ab945a79f1376b58eac482c.189.58:d2a715955a;d2a715955a-qgong1.dev.rhcloud.com jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish-gear-endpointcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0 Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955ab945a79f1376b58eac482c.189.58:d2a715955a;d2a715955a-qgong1.dev.rhcloud.com '' DEBUG: Output of parallel execute: result_stdoutTue Jun 19 06:10:23 EDT 2012: Reloading haproxy service jobargs--cart-namehaproxy-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955ab945a79f1376b58eac482c.189.58:d2a715955a\;d2a715955a-qgong1.dev.rhcloud.com' '\'--hook-nameset-gear-endpointscartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 DEBUG: Output of parallel execute: result_stdoutd2a715955a-qgong1.dev.rhcloud.com|10.124.189.58:38166 jobargs--cart-namejbosseap-6.0--gear-uuidd2a715955ab945a79f1376b58eac482c--input-argsd2a715955a qgong1 d2a715955ab945a79f1376b58eac482c--hook-namepublish-http-urlcartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0geard2a715955ab945a79f1376b58eac482ctag, status: 0 Output of publisher - ''d2a715955ab945a79f1376b58eac482c'='d2a715955a-qgong1.dev.rhcloud.com|10.124.189.58:38166 '' DEBUG: Output of parallel execute: result_stdoutTue Jun 19 06:10:30 EDT 2012: Reloading haproxy service jobargs--cart-namehaproxy-1.4--gear-uuidb324a4c07cdc440a9dfba45e392aadc4--input-argsqs2jbosseap qgong1 b324a4c07cdc440a9dfba45e392aadc4 \'d2a715955ab945a79f1376b58eac482c\'\=\'d2a715955a-qgong1.dev.rhcloud.com\|10.124.189.58:38166' '\'--hook-nameset-proxycartridgestickshift-nodeactionconnector-executeresult_stderrresult_exit_code0gearb324a4c07cdc440a9dfba45e392aadc4tag, status: 0 MongoDataStore.save(Application, qgong, qs2jbosseap, #hidden) DEBUG: Deconfiguring old app 'qs2jbosseap' on ip-10-92-241-213 after move DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a411808> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>"'d2a715955a' 'qgong1' 'd2a715955ab945a79f1376b58eac482c'", :cartridge=>"jbosseap-6.0", :action=>"deconfigure"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a3c7910 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] DEBUG: Cartridge command jbosseap-6.0::deconfigure exitcode = 0 DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fc09a3c5610> DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-app-uuid"=>"b324a4c07cdc440a9dfba45e392aadc4", "--with-container-uuid"=>"d2a715955ab945a79f1376b58eac482c"}, :cartridge=>"stickshift-node", :action=>"app-destroy"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fc09a394d30 @action="cartridge_do", @agent="libra", @results={:sender=>"ip-10-92-241-213", :data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0}>] Successfully moved 'qs2jbosseap' with gear uuid 'd2a715955ab945a79f1376b58eac482c' from 'ip-10-92-241-213' to 'ip-10-124-189-58' Started POST "/broker/userinfo" for 203.114.244.88 at Tue Jun 19 06:11:39 -0400 2012 Processing by LegacyBrokerController#user_info_post as Parameters: {"json_data"=>"{\"api\":\"1.1.3\",\"rhlogin\":\"qgong\"}", "password"=>"[FILTERED]"} User-Agent = '' MongoDataStore.find(CloudUser, qgong, qgong) [REQ_ID=c0ce0baba1bf481a94b234705c8b4d71] ACTION=LEGACY_USER_INFO Completed 200 OK in 156ms (Views: 3.1ms) --- Additional comment from qgong on 2012-06-19 06:26:46 EDT --- Created attachment 592902 [details] new jenkins_log of after move proxy gear
Created attachment 593028 [details] JBosseap jenkins log
Re-test this bug on devenv_1589, and failed. Here are reproduce steps: 1. Setup a multi-nodes, env, here I create two disricts, one is "small" that have two nodes, another one is "medium" that have one node. 2. Create a scalable jbosseap app with jenkins client embeded. 3. Move haproxy gear of this app from one node to another one. 4. Do some change to app, then git push to trigger jenkins build. $ git commit -a -mx; git push [master a183ac0] x 1 file changed, 1 insertion(+), 1 deletion(-) Warning: Permanently added 'eap1-jialiu.dev.rhcloud.com,23.22.247.24' (RSA) to the list of known hosts. Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 270 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins-jialiu.dev.rhcloud.com/job/eap1-build remote: remote: Waiting for build to schedule.....................................................................................Done remote: Waiting for job to complete................................................Done remote: FAILED remote: !!!!!!!! remote: Deployment Halted! remote: If the build failed before the deploy step, your previous remote: build is still running. Otherwise, your application may be remote: partially deployed or inaccessible. remote: Fix the build and try again. remote: !!!!!!!! To ssh://54cb91134d734397a4ef82cca7a8d231.rhcloud.com/~/git/eap1.git/ 5d4c0a3..a183ac0 master -> master Seen console output in jenkins job url, get the following info: ...snip... SSH_CMD: ssh d7cad83a53844a22b95ac80a2d8f55b4.85.199 ++ date + echo 'Syncing to gear: d7cad83a53844a22b95ac80a2d8f55b4.85.199:d7cad83a53 @ ' Wed Jun 20 08:24:38 EDT 2012 Syncing to gear: d7cad83a53844a22b95ac80a2d8f55b4.85.199:d7cad83a53 @ Wed Jun 20 08:24:38 EDT 2012 + for rpccall in '"${OPENSHIFT_SYNC_GEARS_PRE[@]}"' + ssh d7cad83a53844a22b95ac80a2d8f55b4.85.199 'ctl_all stop' Warning: Identity file /var/lib/stickshift/54cb91134d734397a4ef82cca7a8d231//haproxy-1.4/.ssh/haproxy_id_rsa not accessible: Permission denied. Exit code: 1 Build step 'Execute shell' marked build as failure Archiving artifacts Finished: FAILURE
This error happens for all applications. maybe, the move action changed the selinux for the haproxy_id_rsa file , you can see it change from "libra_var_lib_t" to "object_r:ssh_home_t" before move: [root@ip-10-120-206-123 stickshift]# ls -Z /var/lib/stickshift/e53a3a86127f4de487d929d2212ab533/haproxy-1.4/.ssh/haproxy_id_rsa -rw-r-----. root e53a3a86127f4de487d929d2212ab533 unconfined_u:object_r:libra_var_lib_t:s0:c1,c21 /var/lib/stickshift/e53a3a86127f4de487d929d2212ab533/haproxy-1.4/.ssh/haproxy_id_rsa after move: [root@ip-10-120-206-123 stickshift]# ls -Z /var/lib/stickshift/9a6666b3889c4984aa34dabc55394f6c//haproxy-1.4/.ssh/haproxy_id_rsa -rw-r-----. root 9a6666b3889c4984aa34dabc55394f6c unconfined_u:object_r:ssh_home_t:s0:c1,c17 /var/lib/stickshift/9a6666b3889c4984aa34dabc55394f6c//haproxy-1.4/.ssh/haproxy_id_rsa so when jenkins build, it show error: Identity file /var/lib/stickshift/9a6666b3889c4984aa34dabc55394f6c//haproxy-1.4/.ssh/haproxy_id_rsa not accessible: Permission denied. Exit code: 1
This error still exist on devenv_1920,almost I can't build success after move scalable app gear, caused by permission deny. I record the file permission by "ls -Z" before move: -rw-r-----. root 69744a2d81c4413ea9e1e1714c3dd378 unconfined_u:object_r:libra_var_lib_t:s0:c1,c12 69744a2d81c4413ea9e1e1714c3dd378/haproxy-1.4/.ssh/haproxy_id_rsa after move: -rw-r-----. root 69744a2d81c4413ea9e1e1714c3dd378 unconfined_u:object_r:ssh_home_t:s0:c1,c12 69744a2d81c4413ea9e1e1714c3dd378/haproxy-1.4/.ssh/haproxy_id_rsa
Created attachment 601195 [details] move log from development.log
This error happened for all scalable app gear(doesn't happens for non-scalable app), whether jenkins client embed or not will cause this error.
Root cause is believed to be bug 842823, which has been re-opened in response to this report.
The selinux configuration for 842823 apparently only works on recent Fedora installs; not RHEL 6. 842823's work-around was to run rhc-restrecon. You may have to borrow from the rhc-restorecon behaviour in order to properly configure selinux after a gear move. We may not be able to resolve 842823 beyond re-running rhc-restorecon after the system has done an selinux relabel.
Removing dependency on bug 842823 as the fix is not portable to this context per Rob Millner's comments.
Should have fix in today
Resolved by https://github.com/openshift/crankcase/pull/306
I check your patch has get in build devenv_1931, but still fail as same error, no change.
Dan cannot reproduce this on his environment. I am trying to reproduce this.
verified this on devenv_1932. Cannot reproduce the error. Tried this with a scalable php and a jboss app 1. Set up two nodes in a multi-node configuration 2. Created a district, added both nodes to the district 3. Created a scalable php-5.3 application 4. Created a jenkins application 5. Embedded the jenkins-client into the php applicatoin 6. Pushed a change to the php application, resulting in a successful Jenkins build 7. Moved the php application's haproxy gear to the opposite node 8. Pushed another change to the php application, resulting in another successful Jenkins build
Created attachment 602099 [details] latest move log.
Reopen it on devenv_1936 and stage_228, only jbossas and jbosseap can't build success after move it's application's haproxy gear.(other app success), error is same as above like: "Identity file /var/lib/stickshift/54cb91134d734397a4ef82cca7a8d231//haproxy-1.4/.ssh/haproxy_id_rsa not accessible: Permission denied." please see attachment of latest move log. But I remember I could build success(after move haproxy gear) in devenv_1933, please the patch for those build recently.
Earlier patch was made unstable due to a broker issue which is now resolved by: https://github.com/openshift/crankcase/pull/333 Prior to the patch in 333, the gear move would break randomly depending on what order the broker happened to perform some of the move operations, explaining why the fix appeared to work sometimes, but not others. Now it will be stable.
verified on stg_229, test jbossas, jbosseap, php with steps: 1. Set up two nodes in a multi-node configuration 2. Created a district, added both nodes to the district 3. Created a scalable php-5.3 application 4. Created a jenkins application 5. Embedded the jenkins-client into the php applicatoin 6. Pushed a change to the php application, resulting in a successful Jenkins build 7. Moved the php application's haproxy gear to the opposite node 8. Pushed another change to the php application, resulting in another successful Jenkins build