Bug 806401 - [REST API] node execution failure when trigger 'show-port' event for a scalable application
[REST API] node execution failure when trigger 'show-port' event for a scalab...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Rob Millner
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-23 12:03 EDT by Xavier Coulon
Modified: 2015-05-14 21:49 EDT (History)
4 users (show)

See Also:
Fixed In Version: rubygem-stickshift-controller-0.9.10-1+
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-04-27 16:45:32 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xavier Coulon 2012-03-23 12:03:28 EDT
When querying for 'show-port' event as below:
curl .... https://openshift.redhat.com/broker/rest/domains/xcoulon/applications/test2/events -X POST -v --form event=show-port 

I get the following error response:

<response>
  <version>1.0</version>
  <type nil="true"></type>
  <messages>
    <message>
      <field nil="true"></field>
      <text>Failed to add event show-port to application test2 due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.</text>
      <severity>error</severity>
      <exit-code>143</exit-code>
    </message>
  </messages>
  <data nil="true"></data>
  <status>internal_server_error</status>
</response>
Comment 1 Rob Millner 2012-04-10 17:54:10 EDT
Fixed in commit dac14b9.
Comment 2 Johnny Liu 2012-04-11 02:23:27 EDT
Re-test this bug with devenv_1715, still failed.

$ curl -k -X POST -H 'Accept: application/xml' -d event=show-port --user jialiu@redhat.com:xx https://ec2-23-20-149-175.compute-1.amazonaws.com/broker/rest/domains/jialiu/applications/app1/events
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <message>
      <exit-code>143</exit-code>
      <severity>error</severity>
      <text>Failed to add event show-port to application app1 due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.</text>
      <field nil="true"></field>
    </message>
  </messages>
  <version>1.0</version>
  <type nil="true"></type>
  <data>
    <datum nil="true"></datum>
  </data>
  <status>internal_server_error</status>
</response>



I dig more about this issue, found there are two issues:
1) For a non-scalable app, in <li.repo>/stickshift/abstract/abstract/info/lib/network, when failed to find proxy, show_proxy_port function is using "exit", but not "return". That is why show-port event failure happen for a non-scalable app.

2) For a scalable app, I got the following message in the broker log:
<--snip-->
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7f1fb7625f10>
DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"haproxy-1.4", :args=>"'myapp' 'jialiu' '45da98a611cf452fad3dad8c819ba0ee'", :action=>"show-port"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7f1fb75b3488 @results={:statusmsg=>"cartridge_do_action ERROR action 'show-port' not found.", :sender=>"ip-10-190-26-158", :statuscode=>1, :data=>{:exitcode=>127}}, @action="cartridge_do", @agent="libra">]
Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
#<StickShift::NodeException: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.>
<--snip-->
Comment 3 Rob Millner 2012-04-11 14:28:59 EDT
The exits pointed out and the "set -e" at the top of each script were both causing it to fail rather than send back useful output to the broker.  Fixed in 91c0945a and 2dafbaf.
Comment 4 Rob Millner 2012-04-12 02:19:58 EDT
Re-broken by another commit.  Taking the ticket back to fix.
Comment 5 Rob Millner 2012-04-12 15:04:30 EDT
Fixed by commits 30da586e6 and 0c242d2d5.
Comment 6 Johnny Liu 2012-04-16 04:27:24 EDT
Re-test this bug on devenv_1723, but still failed.

Because this bug involve scalable application and non-scalable application, to be more clear, this bug will only track scalable application, another bug for non-scalable application will be opened.


1. Create a scalable application.
2. Run the following command.
$ curl -k -X POST -H 'Accept: application/xml' -d event=show-port --user jialiu@redhat.com:214214 https://ec2-107-22-102-43.compute-1.amazonaws.com/broker/rest/domains/jialiu/applications/myapp/events
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <status>internal_server_error</status>
  <type nil="true"></type>
  <version>1.0</version>
  <messages>
    <message>
      <severity>error</severity>
      <field nil="true"></field>
      <exit-code>143</exit-code>
      <text>Failed to add event show-port to application myapp due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.</text>
    </message>
  </messages>
  <data>
    <datum nil="true"></datum>
  </data>
</response>

Mcollective log:
<--snip-->
D, [2012-04-16T04:04:46.373366 #1015] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f6b6b6f6a70
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:action=>"show-port",
   :process_results=>true,
   :args=>"'myapp' 'jialiu' 'f84e23d3ee10439ab820d167ac56d903'",
   :cartridge=>"php-5.3"},
 @sender="ip-10-62-7-53",
 @time=1334563486,
 @uniqid="8c00c2fcbb05152ebdccc1bf306034da">

D, [2012-04-16T04:04:46.373717 #1015] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = php-5.3 show-port 'myapp' 'jialiu' 'f84e23d3ee10439ab820d167ac56d903'
D, [2012-04-16T04:04:46.575362 #1015] DEBUG -- : libra.rb:102:in `cartridge_do_action' cartridge_do_action (0)
------
CART_DATA: PROXY_HOST=myapp-jialiu.dev.rhcloud.com
CART_DATA: PROXY_PORT=35546
CART_DATA: HOST=127.0.251.129
CART_DATA: PORT=8080

------)
D, [2012-04-16T04:04:46.576010 #1015] DEBUG -- : aes_security.rb:142:in `serialize' Serializing using yaml
<--snip-->
D, [2012-04-16T04:04:46.955734 #1015] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f6b6b6d59b0
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:action=>"show-port",
   :process_results=>true,
   :args=>"'de83f8805a' 'jialiu' 'de83f8805abd40be9f0264bf26361919'",
   :cartridge=>"myapp"},
 @sender="ip-10-62-7-53",
 @time=1334563486,
 @uniqid="b98f661eb25aa7874818e56a8ad37de3">

D, [2012-04-16T04:04:46.956035 #1015] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = myapp show-port 'de83f8805a' 'jialiu' 'de83f8805abd40be9f0264bf26361919'
D, [2012-04-16T04:04:46.956579 #1015] DEBUG -- : aes_security.rb:142:in `serialize' Serializing using yaml
<--snip-->

Broker log:
<--snip-->
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7f84cd2a7098>
DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"myapp", :args=>"'de83f8805a' 'jialiu' 'de83f8805abd40be9f0264bf26361919'", :action=>"show-port"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7f84cda4e648 @results={:statusmsg=>"cartridge_do_action ERROR action 'show-port' not found.", :sender=>"ip-10-62-7-53", :statuscode=>1, :data=>{:exitcode=>127}}, @action="cartridge_do", @agent="libra">]
Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
#<StickShift::NodeException: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.>
<--snip-->
Comment 7 Rob Millner 2012-04-23 16:23:35 EDT
The new error looks like a change somewhere in broker has the event delivered to the application rather than the cartridge.

From mcollective.log...

D, [2012-04-23T16:21:06.790061 #4379] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7ff1a78c69f0
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:cartridge=>"rmtest",
   :action=>"show-port",
   :args=>"'6b86e78bc0' 'rmillner0126' '6b86e78bc09c441ab0048ca1e8a216f6'",
   :process_results=>true},
 @sender="ip-10-114-49-240",
 @time=1335212466,
 @uniqid="7f231c2de8aa1b8a57a582654ff7baff">


Notice that the cartridge is the application name, not the application cartridge.
Comment 8 Rob Millner 2012-04-23 21:05:28 EDT
Found an additional issue, the hook is now called on every cartridge including haproxy which doesn't have one.
Comment 9 Rob Millner 2012-04-23 21:30:30 EDT
Fixed in commits 7912787, 3a32e804, and 0ec053d.
Comment 10 Johnny Liu 2012-04-23 23:30:36 EDT
Retest this bug with devenv_1738, still failed.


broker log:
<--snip-->
#<ComponentInstance:0x7f8f95610f40 @changed_attributes={"parent_cart_group"=>nil, "parent_cart_name"=>nil, "name"=>nil, "group_instance_name"=>nil, "dependencies"=>nil, "parent_cart_profile"=>nil, "cart_data"=>[], "parent_component_name"=>nil, "exec_order"=>nil}, @parent_component_name="default", @cart_data=[""], @exec_order=[], @dependencies=[], @parent_cart_profile="default", @name="@@app/comp-proxy/cart-haproxy-1.4", @group_instance_name="@@app/comp-proxy/cart-haproxy-1.4", @parent_cart_name="haproxy-1.4", @parent_cart_group="default">

DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7f8f95555088>
DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"show-port", :args=>"'myapp' 'jialiu' '46df010dc6a84e67a2c041844d6269c0'", :cartridge=>"haproxy-1.4"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7f8f95685520 @agent="libra", @results={:sender=>"ip-10-62-97-178", :statusmsg=>"cartridge_do_action ERROR action 'show-port' not found.", :statuscode=>1, :data=>{:exitcode=>127}}, @action="cartridge_do">]
Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
#<StickShift::NodeException: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.>
<--snip-->


mcollective log:
<--snip-->
D, [2012-04-23T23:25:06.200108 #1014] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f2da7592e10
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:process_results=>true,
   :cartridge=>"haproxy-1.4",
   :args=>"'myapp' 'jialiu' '46df010dc6a84e67a2c041844d6269c0'",
   :action=>"show-port"},
 @sender="ip-10-62-97-178",
 @time=1335237906,
 @uniqid="4838f0c0a28a760e6d6d462ebcffc99e">

D, [2012-04-23T23:25:06.200596 #1014] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = haproxy-1.4 show-port 'myapp' 'jialiu' '46df010dc6a84e67a2c041844d6269c0'
<--snip-->


Look like broker is trying to call show-port hooks against haproxy-1.4
which doesn't have one.
Comment 11 Rob Millner 2012-04-24 14:24:58 EDT
Sorry about that - the commits missed devenv_1723 but are in devenv_1739 and
later.

$ devrun 

Launching latest DevEnv instance ami-ca66bda3
I, [2012-04-24T10:54:41.846568 #3166]  INFO -- : Creating new instance...

...

$ rhc app create -p [hidden] -a rmtest -t python-2.6 -s
Creating application: rmtest in rmillner0127
...
Successfully created application: rmtest


$ curl -k -X POST -H 'Accept: application/xml' -d event=show-port  --user "rmillner:[hidden]" https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <version>1.0</version>
  <type>application</type>
  <data>
    <application>
      <name>rmtest</name>
      <uuid>e6561fd52fec491880a0ad975c531bfa</uuid>
      <git-url>ssh://e6561fd52fec491880a0ad975c531bfa@rmtest-rmillner0127.dev.rhcloud.com/~/git/rmtest.git/</git-url>
      <aliases/>
      <app-url>http://rmtest-rmillner0127.dev.rhcloud.com/</app-url>
      <health-check-path>health</health-check-path>
      <scalable nil="true"></scalable>
      <embedded>
        <haproxy-1.4>
          <info></info>
        </haproxy-1.4>
      </embedded>
      <creation-time>2012-04-24T14:18:54-04:00</creation-time>
      <links>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>restart</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Restart application</rel>
        </link>
        <link>
          <method>GET</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/cartridges</href>
          <required-params/>
          <optional-params/>
          <rel>List embedded cartridges</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>expose-port</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Expose port</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>start</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Start application</rel>
        </link>
        <link>
          <method>GET</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/gears</href>
          <required-params/>
          <optional-params/>
          <rel>Get application gears</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>show-port</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Show port</rel>
        </link>
        <link>
          <method>DELETE</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest</href>
          <required-params/>
          <optional-params/>
          <rel>Delete application</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>scale-down</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Scale down application</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>conceal-port</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Conceal port</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>stop</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Stop application</rel>
        </link>
        <link>
          <method>GET</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/descriptor</href>
          <required-params/>
          <optional-params/>
          <rel>Get application descriptor</rel>
        </link>
        <link>
          <method>GET</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest</href>
          <required-params/>
          <optional-params/>
          <rel>Get application</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/cartridges</href>
          <required-params>
            <param>
              <name>cartridge</name>
              <description>framework-type, e.g.: mysql-5.1</description>
              <type>string</type>
              <valid-options>
                <valid-option>mongodb-2.0</valid-option>
                <valid-option>cron-1.4</valid-option>
                <valid-option>mysql-5.1</valid-option>
                <valid-option>postgresql-8.4</valid-option>
                <valid-option>haproxy-1.4</valid-option>
                <valid-option>10gen-mms-agent-0.1</valid-option>
                <valid-option>phpmyadmin-3.4</valid-option>
                <valid-option>metrics-0.1</valid-option>
                <valid-option>phpmoadmin-1.0</valid-option>
                <valid-option>rockmongo-1.1</valid-option>
                <valid-option>jenkins-client-1.4</valid-option>
              </valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Add embedded cartridge</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>scale-up</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Scale up application</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>add-alias</valid-options>
            </param>
            <param>
              <name>alias</name>
              <description>The server alias for the application</description>
              <type>string</type>
              <valid-options/>
            </param>
          </required-params>
          <optional-params/>
          <rel>Add application alias</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>remove-alias</valid-options>
            </param>
            <param>
              <name>alias</name>
              <description>The application alias to be removed</description>
              <type>string</type>
              <valid-options/>
            </param>
          </required-params>
          <optional-params/>
          <rel>Remove application alias</rel>
        </link>
        <link>
          <method>POST</method>
          <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href>
          <required-params>
            <param>
              <name>event</name>
              <description>event</description>
              <type>string</type>
              <valid-options>force-stop</valid-options>
            </param>
          </required-params>
          <optional-params/>
          <rel>Force stop application</rel>
        </link>
      </links>
      <domain-id>rmillner0127</domain-id>
      <framework>python-2.6</framework>
      <gear-profile>small</gear-profile>
    </application>
  </data>
  <messages>
    <message>
      <exit-code nil="true"></exit-code>
      <text>Added show-port to application rmtest</text>
      <field nil="true"></field>
      <severity>INFO</severity>
    </message>
  </messages>
  <status>ok</status>
  <supported-api-versions>
    <supported-api-version>1.0</supported-api-version>
  </supported-api-versions>
</response>
Comment 12 Johnny Liu 2012-04-24 22:50:53 EDT
Verified this bug on devenv_1741, and PASS.

Note You need to log in before you can comment on or make changes to this bug.