| Summary: | [REST API] node execution failure when trigger 'show-port' event for a scalable application | ||
|---|---|---|---|
| Product: | OKD | Reporter: | Xavier Coulon <xcoulon> |
| Component: | Pod | Assignee: | Rob Millner <rmillner> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> |
| Severity: | low | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.x | CC: | adietish, jialiu, mfisher, mpatel |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | rubygem-stickshift-controller-0.9.10-1+ | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-04-27 20:45:32 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Xavier Coulon
2012-03-23 16:03:28 UTC
Fixed in commit dac14b9. Re-test this bug with devenv_1715, still failed. $ curl -k -X POST -H 'Accept: application/xml' -d event=show-port --user jialiu:xx https://ec2-23-20-149-175.compute-1.amazonaws.com/broker/rest/domains/jialiu/applications/app1/events <?xml version="1.0" encoding="UTF-8"?> <response> <messages> <message> <exit-code>143</exit-code> <severity>error</severity> <text>Failed to add event show-port to application app1 due to: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.</text> <field nil="true"></field> </message> </messages> <version>1.0</version> <type nil="true"></type> <data> <datum nil="true"></datum> </data> <status>internal_server_error</status> </response> I dig more about this issue, found there are two issues: 1) For a non-scalable app, in <li.repo>/stickshift/abstract/abstract/info/lib/network, when failed to find proxy, show_proxy_port function is using "exit", but not "return". That is why show-port event failure happen for a non-scalable app. 2) For a scalable app, I got the following message in the broker log: <--snip--> DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7f1fb7625f10> DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"haproxy-1.4", :args=>"'myapp' 'jialiu' '45da98a611cf452fad3dad8c819ba0ee'", :action=>"show-port"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7f1fb75b3488 @results={:statusmsg=>"cartridge_do_action ERROR action 'show-port' not found.", :sender=>"ip-10-190-26-158", :statuscode=>1, :data=>{:exitcode=>127}}, @action="cartridge_do", @agent="libra">] Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. #<StickShift::NodeException: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.> <--snip--> The exits pointed out and the "set -e" at the top of each script were both causing it to fail rather than send back useful output to the broker. Fixed in 91c0945a and 2dafbaf. Re-broken by another commit. Taking the ticket back to fix. Fixed by commits 30da586e6 and 0c242d2d5. Re-test this bug on devenv_1723, but still failed. Because this bug involve scalable application and non-scalable application, to be more clear, this bug will only track scalable application, another bug for non-scalable application will be opened. 1. Create a scalable application. 2. Run the following command. $ curl -k -X POST -H 'Accept: application/xml' -d event=show-port --user jialiu:214214 https://ec2-107-22-102-43.compute-1.amazonaws.com/broker/rest/domains/jialiu/applications/myapp/events <?xml version="1.0" encoding="UTF-8"?> <response> <status>internal_server_error</status> <type nil="true"></type> <version>1.0</version> <messages> <message> <severity>error</severity> <field nil="true"></field> <exit-code>143</exit-code> <text>Failed to add event show-port to application myapp due to: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.</text> </message> </messages> <data> <datum nil="true"></datum> </data> </response> Mcollective log: <--snip--> D, [2012-04-16T04:04:46.373366 #1015] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f6b6b6f6a70 @action="cartridge_do", @agent="libra", @caller="cert=mcollective-public", @data= {:action=>"show-port", :process_results=>true, :args=>"'myapp' 'jialiu' 'f84e23d3ee10439ab820d167ac56d903'", :cartridge=>"php-5.3"}, @sender="ip-10-62-7-53", @time=1334563486, @uniqid="8c00c2fcbb05152ebdccc1bf306034da"> D, [2012-04-16T04:04:46.373717 #1015] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = php-5.3 show-port 'myapp' 'jialiu' 'f84e23d3ee10439ab820d167ac56d903' D, [2012-04-16T04:04:46.575362 #1015] DEBUG -- : libra.rb:102:in `cartridge_do_action' cartridge_do_action (0) ------ CART_DATA: PROXY_HOST=myapp-jialiu.dev.rhcloud.com CART_DATA: PROXY_PORT=35546 CART_DATA: HOST=127.0.251.129 CART_DATA: PORT=8080 ------) D, [2012-04-16T04:04:46.576010 #1015] DEBUG -- : aes_security.rb:142:in `serialize' Serializing using yaml <--snip--> D, [2012-04-16T04:04:46.955734 #1015] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f6b6b6d59b0 @action="cartridge_do", @agent="libra", @caller="cert=mcollective-public", @data= {:action=>"show-port", :process_results=>true, :args=>"'de83f8805a' 'jialiu' 'de83f8805abd40be9f0264bf26361919'", :cartridge=>"myapp"}, @sender="ip-10-62-7-53", @time=1334563486, @uniqid="b98f661eb25aa7874818e56a8ad37de3"> D, [2012-04-16T04:04:46.956035 #1015] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = myapp show-port 'de83f8805a' 'jialiu' 'de83f8805abd40be9f0264bf26361919' D, [2012-04-16T04:04:46.956579 #1015] DEBUG -- : aes_security.rb:142:in `serialize' Serializing using yaml <--snip--> Broker log: <--snip--> DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7f84cd2a7098> DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"myapp", :args=>"'de83f8805a' 'jialiu' 'de83f8805abd40be9f0264bf26361919'", :action=>"show-port"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7f84cda4e648 @results={:statusmsg=>"cartridge_do_action ERROR action 'show-port' not found.", :sender=>"ip-10-62-7-53", :statuscode=>1, :data=>{:exitcode=>127}}, @action="cartridge_do", @agent="libra">] Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. #<StickShift::NodeException: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.> <--snip--> The new error looks like a change somewhere in broker has the event delivered to the application rather than the cartridge.
From mcollective.log...
D, [2012-04-23T16:21:06.790061 #4379] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7ff1a78c69f0
@action="cartridge_do",
@agent="libra",
@caller="cert=mcollective-public",
@data=
{:cartridge=>"rmtest",
:action=>"show-port",
:args=>"'6b86e78bc0' 'rmillner0126' '6b86e78bc09c441ab0048ca1e8a216f6'",
:process_results=>true},
@sender="ip-10-114-49-240",
@time=1335212466,
@uniqid="7f231c2de8aa1b8a57a582654ff7baff">
Notice that the cartridge is the application name, not the application cartridge.
Found an additional issue, the hook is now called on every cartridge including haproxy which doesn't have one. Fixed in commits 7912787, 3a32e804, and 0ec053d. Retest this bug with devenv_1738, still failed.
broker log:
<--snip-->
#<ComponentInstance:0x7f8f95610f40 @changed_attributes={"parent_cart_group"=>nil, "parent_cart_name"=>nil, "name"=>nil, "group_instance_name"=>nil, "dependencies"=>nil, "parent_cart_profile"=>nil, "cart_data"=>[], "parent_component_name"=>nil, "exec_order"=>nil}, @parent_component_name="default", @cart_data=[""], @exec_order=[], @dependencies=[], @parent_cart_profile="default", @name="@@app/comp-proxy/cart-haproxy-1.4", @group_instance_name="@@app/comp-proxy/cart-haproxy-1.4", @parent_cart_name="haproxy-1.4", @parent_cart_group="default">
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7f8f95555088>
DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"show-port", :args=>"'myapp' 'jialiu' '46df010dc6a84e67a2c041844d6269c0'", :cartridge=>"haproxy-1.4"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7f8f95685520 @agent="libra", @results={:sender=>"ip-10-62-97-178", :statusmsg=>"cartridge_do_action ERROR action 'show-port' not found.", :statuscode=>1, :data=>{:exitcode=>127}}, @action="cartridge_do">]
Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.
#<StickShift::NodeException: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.>
<--snip-->
mcollective log:
<--snip-->
D, [2012-04-23T23:25:06.200108 #1014] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f2da7592e10
@action="cartridge_do",
@agent="libra",
@caller="cert=mcollective-public",
@data=
{:process_results=>true,
:cartridge=>"haproxy-1.4",
:args=>"'myapp' 'jialiu' '46df010dc6a84e67a2c041844d6269c0'",
:action=>"show-port"},
@sender="ip-10-62-97-178",
@time=1335237906,
@uniqid="4838f0c0a28a760e6d6d462ebcffc99e">
D, [2012-04-23T23:25:06.200596 #1014] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = haproxy-1.4 show-port 'myapp' 'jialiu' '46df010dc6a84e67a2c041844d6269c0'
<--snip-->
Look like broker is trying to call show-port hooks against haproxy-1.4
which doesn't have one.
Sorry about that - the commits missed devenv_1723 but are in devenv_1739 and later. $ devrun Launching latest DevEnv instance ami-ca66bda3 I, [2012-04-24T10:54:41.846568 #3166] INFO -- : Creating new instance... ... $ rhc app create -p [hidden] -a rmtest -t python-2.6 -s Creating application: rmtest in rmillner0127 ... Successfully created application: rmtest $ curl -k -X POST -H 'Accept: application/xml' -d event=show-port --user "rmillner:[hidden]" https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events <?xml version="1.0" encoding="UTF-8"?> <response> <version>1.0</version> <type>application</type> <data> <application> <name>rmtest</name> <uuid>e6561fd52fec491880a0ad975c531bfa</uuid> <git-url>ssh://e6561fd52fec491880a0ad975c531bfa.rhcloud.com/~/git/rmtest.git/</git-url> <aliases/> <app-url>http://rmtest-rmillner0127.dev.rhcloud.com/</app-url> <health-check-path>health</health-check-path> <scalable nil="true"></scalable> <embedded> <haproxy-1.4> <info></info> </haproxy-1.4> </embedded> <creation-time>2012-04-24T14:18:54-04:00</creation-time> <links> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>restart</valid-options> </param> </required-params> <optional-params/> <rel>Restart application</rel> </link> <link> <method>GET</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/cartridges</href> <required-params/> <optional-params/> <rel>List embedded cartridges</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>expose-port</valid-options> </param> </required-params> <optional-params/> <rel>Expose port</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>start</valid-options> </param> </required-params> <optional-params/> <rel>Start application</rel> </link> <link> <method>GET</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/gears</href> <required-params/> <optional-params/> <rel>Get application gears</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>show-port</valid-options> </param> </required-params> <optional-params/> <rel>Show port</rel> </link> <link> <method>DELETE</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest</href> <required-params/> <optional-params/> <rel>Delete application</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>scale-down</valid-options> </param> </required-params> <optional-params/> <rel>Scale down application</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>conceal-port</valid-options> </param> </required-params> <optional-params/> <rel>Conceal port</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>stop</valid-options> </param> </required-params> <optional-params/> <rel>Stop application</rel> </link> <link> <method>GET</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/descriptor</href> <required-params/> <optional-params/> <rel>Get application descriptor</rel> </link> <link> <method>GET</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest</href> <required-params/> <optional-params/> <rel>Get application</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/cartridges</href> <required-params> <param> <name>cartridge</name> <description>framework-type, e.g.: mysql-5.1</description> <type>string</type> <valid-options> <valid-option>mongodb-2.0</valid-option> <valid-option>cron-1.4</valid-option> <valid-option>mysql-5.1</valid-option> <valid-option>postgresql-8.4</valid-option> <valid-option>haproxy-1.4</valid-option> <valid-option>10gen-mms-agent-0.1</valid-option> <valid-option>phpmyadmin-3.4</valid-option> <valid-option>metrics-0.1</valid-option> <valid-option>phpmoadmin-1.0</valid-option> <valid-option>rockmongo-1.1</valid-option> <valid-option>jenkins-client-1.4</valid-option> </valid-options> </param> </required-params> <optional-params/> <rel>Add embedded cartridge</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>scale-up</valid-options> </param> </required-params> <optional-params/> <rel>Scale up application</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>add-alias</valid-options> </param> <param> <name>alias</name> <description>The server alias for the application</description> <type>string</type> <valid-options/> </param> </required-params> <optional-params/> <rel>Add application alias</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>remove-alias</valid-options> </param> <param> <name>alias</name> <description>The application alias to be removed</description> <type>string</type> <valid-options/> </param> </required-params> <optional-params/> <rel>Remove application alias</rel> </link> <link> <method>POST</method> <href>https://50.17.177.181/broker/rest/domains/rmillner0127/applications/rmtest/events</href> <required-params> <param> <name>event</name> <description>event</description> <type>string</type> <valid-options>force-stop</valid-options> </param> </required-params> <optional-params/> <rel>Force stop application</rel> </link> </links> <domain-id>rmillner0127</domain-id> <framework>python-2.6</framework> <gear-profile>small</gear-profile> </application> </data> <messages> <message> <exit-code nil="true"></exit-code> <text>Added show-port to application rmtest</text> <field nil="true"></field> <severity>INFO</severity> </message> </messages> <status>ok</status> <supported-api-versions> <supported-api-version>1.0</supported-api-version> </supported-api-versions> </response> Verified this bug on devenv_1741, and PASS. |