Description of problem:

During the run of the test cases we are getting:

[9573] DEBUG [REQ_ID=2e6d0ecbddb64db9a67a22b40f321c8b] ACTION=EMBED_CARTRIDGE Embedded cartridge mysql-5.1 in application app1
[9573] INFO  Completed 201 Created in 25923ms (Views: 186.4ms)
[9573] INFO  Started POST "/broker/rest/domains/98115fe29c/applications/app1/events" for 127.0.0.1 at Thu Aug 16 10:56:31 -0400 2012
[9573] INFO    Processing by AppEventsController#create as JSON
[9573] INFO    Parameters: {"event"=>"expose-port", "application_id"=>"app1", "nolinks"=>"false", "domain_id"=>"98115fe29c"}
[9573] DEBUG MongoDataStore.find(CloudUser, test-user9a1c85edfd, test-user9a1c85edfd)
[9573] DEBUG DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fbbcfde27d0>
[9573] DEBUG DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"expose-port", :args=>"'9a6e88cf79' '98115fe29c' '9a6e88cf79eb448b9aa1d14876059f94'", :cartridge=>"mysql-5.1"}, @id, {'identity' => @id})
[9573] DEBUG DEBUG: [#<MCollective::RPC::Result:0x7fbbcfd5b348 @action="cartridge_do", @results={:statusmsg=>"OK", :sender=>"ip-10-194-173-235", :data=>{:exitcode=>0, :output=>"CART_DATA: PROXY_HOST=9a6e88cf79-98115fe29c.dev.rhcloud.com\nCART_DATA: PROXY_PORT=35556\nCART_DATA: HOST=127.0.252.129\nCART_DATA: PORT=3306\n"}, :statuscode=>0}, @agent="stickshift">]
[9573] DEBUG DEBUG: Cartridge command mysql-5.1::expose-port exitcode = 0
[9573] DEBUG DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fbbcfd4cc08>
[9573] DEBUG DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"expose-port", :args=>"'app1' '98115fe29c' '831f286715214302b0229cfbcc8295f2'", :cartridge=>"php-5.3"}, @id, {'identity' => @id})
[9573] DEBUG DEBUG: [#<MCollective::RPC::Result:0x7fbbcfcd7548 @action="cartridge_do", @results={:statusmsg=>"OK", :sender=>"ip-10-194-173-235", :data=>{:exitcode=>0, :output=>"CART_DATA: PROXY_HOST=app1-98115fe29c.dev.rhcloud.com\nCART_DATA: PROXY_PORT=35546\nCART_DATA: HOST=127.0.251.129\nCART_DATA: PORT=8080\n"}, :statuscode=>0}, @agent="stickshift">]
[9573] DEBUG DEBUG: Cartridge command php-5.3::expose-port exitcode = 0
[9573] DEBUG DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fbbcfca52c8>
[9573] DEBUG DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"expose-port", :args=>"'app1' '98115fe29c' '831f286715214302b0229cfbcc8295f2'", :cartridge=>"haproxy-1.4"}, @id, {'identity' => @id})
[9573] DEBUG DEBUG: [#<MCollective::RPC::Result:0x7fbbcfc389c0 @action="cartridge_do", @results={:statusmsg=>"cartridge_do_action ERROR action 'expose-port' not found.", :sender=>"ip-10-194-173-235", :data=>{:exitcode=>127}, :statuscode=>1}, @agent="stickshift">]
[9573] DEBUG DEBUG: Cartridge command haproxy-1.4::expose-port exitcode = 127
[9573] ERROR Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.


The smoking gun being:

[9573] DEBUG DEBUG: [#<MCollective::RPC::Result:0x7fbbcfc389c0 @action="cartridge_do", @results={:statusmsg=>"cartridge_do_action ERROR action 'expose-port' not found.", :sender=>"ip-10-194-173-235", :data=>{:exitcode=>127}, :statuscode=>1}, @agent="stickshift">]

And ha-proxy doesn't have an expose-port


The full log is here:

https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/130/artifact/li/rhc/log/broker/development.log




Version-Release number of selected component (if applicable):


How reproducible:
Not sure.  I am debugging all the errors we see running tests and I see this one.


Steps to Reproduce:
1. Run devenv test enough times and troll through the logs
2.
3.