Bug 865282

Summary: [US2513][fork]Node execution failure when doing rhc threaddump for jbossEWS app
Product: OKD Reporter: joycezhang <jinzhang>
Component: ContainersAssignee: Bill DeCoste <wdecoste>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: jhonce
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-11-06 18:49:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
development log from boker none

Description joycezhang 2012-10-11 06:30:06 UTC
Description of problem:
Node execution failure when doing rhc threaddump for jbossews app as below:
# rhc threaddump app4
Password: *

Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.

Version-Release number of selected component (if applicable):
rhc client built from 
fork_ami_US2597_US2599_US2813_US2817_US2872_US2816_US2814_226
fork_ami_US2513-6_224 

How reproducible:
always

Steps to Reproduce:
1.Create an jbossews app
#rhc app create app jbossews-1.0
2.Do threaddump for this app
#rhc threaddump app

  
Actual results:
It's failed with node execution failure error.


Expected results:
The threaddump file should be available.

Additional info:
It can work well with jbosseap-6.0 app.

Comment 1 Bill DeCoste 2012-10-11 17:34:33 UTC
I can't recreate this from AMI US2513-6. Please see my logs below. Can you attach the errors you are seeing?

[root@ip-10-100-253-248 ~]# rhc app create -l bdecoste42 -p 123456 -a ews1 -t jbossews-1.0 -d
Submitting form:
rhlogin: bdecoste42
debug: true
Contacting https://localhost
Creating application: ews1 in bdecoste42b
Contacting https://localhost
Response from server:
DEBUG:


Exit Code: 0
api_c: placeholder
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
API version:    1.1.3
DEBUG: 'ews1' creation returned success.
Now your new domain name is being propagated worldwide (this might take a minute)...
Pulling new repo down
git clone  ssh://a3fea95f3d76415eaadadd14b11de6e6.rhcloud.com/~/git/ews1.git/ ews1
The authenticity of host 'ews1-bdecoste42b.dev.rhcloud.com (10.100.253.248)' can't be established.
RSA key fingerprint is 23:9d:5c:a9:83:bb:9e:fc:f6:4c:0b:c1:0e:e8:bc:7a.
Are you sure you want to continue connecting (yes/no)? yes
Checking if the application is available #1
Application ews1 is available at: http://ews1-bdecoste42b.dev.rhcloud.com/
  Git URL: ssh://a3fea95f3d76415eaadadd14b11de6e6.rhcloud.com/~/git/ews1.git/
To make changes to 'ews1', commit to ews1/.
Successfully created application: ews1
[root@ip-10-100-253-248 ~]# rhc app tail -l bdecoste42 -p 123456 -a ews1 -d
Submitting form:
debug: true
rhlogin: bdecoste42
Contacting https://localhost
Attempting to tail files: */log*/*
Use ctl + c to stop
ssh -t a3fea95f3d76415eaadadd14b11de6e6.rhcloud.com 'tail */log*/*'
==> jbossews-1.0/logs/localhost.2012-10-11.log <==

==> jbossews-1.0/logs/manager.2012-10-11.log <==

==> jbossews-1.0/logs/catalina.out <==
Oct 11, 2012 1:29:37 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Oct 11, 2012 1:29:37 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Oct 11, 2012 1:29:37 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive ROOT.war
Oct 11, 2012 1:29:38 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-127.0.250.129-8080
Oct 11, 2012 1:29:38 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1011 ms

==> jbossews-1.0/logs/catalina.2012-10-11.log <==
Oct 11, 2012 1:29:37 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Oct 11, 2012 1:29:37 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Oct 11, 2012 1:29:37 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive ROOT.war
Oct 11, 2012 1:29:38 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-127.0.250.129-8080
Oct 11, 2012 1:29:38 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1011 ms

==> jbossews-1.0/logs/host-manager.2012-10-11.log <==

Comment 2 joycezhang 2012-10-12 03:02:13 UTC
Created attachment 625790 [details]
development log from boker

Comment 3 joycezhang 2012-10-12 03:05:11 UTC
This problem is caused by rhc threaddump, it works fine with rhc tail separately.

Please refer to debug infor here from command line:

# rhc threaddump app1 -d
Password: *

D, [2012-10-12T10:53:58.865557 #2332] DEBUG -- : Connecting to https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/api
D, [2012-10-12T10:54:00.056587 #2332] DEBUG -- : Getting all domains
D, [2012-10-12T10:54:00.056827 #2332] DEBUG -- : Request: #<RestClient::Request:0x0000000236b788 @method="GET", @headers={:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, @url="https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains", @cookies={}, @payload="", @user=nil, @password=nil, @timeout=nil, @open_timeout=nil, @block_response=nil, @raw_response=false, @verify_ssl=false, @ssl_client_cert=nil, @ssl_client_key=nil, @ssl_ca_file=nil, @tf=nil, @max_redirects=10, @processed_headers={"Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)", "Content-Length"=>"0", "Content-Type"=>"application/x-www-form-urlencoded"}, @args={:url=>"https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains", :method=>"GET", :headers=>{:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, :payload=>{}, :timeout=>nil}>
D, [2012-10-12T10:54:01.299825 #2332] DEBUG -- : Finding domain joycedev3
D, [2012-10-12T10:54:01.299906 #2332] DEBUG -- : Getting all domains
D, [2012-10-12T10:54:01.300083 #2332] DEBUG -- : Request: #<RestClient::Request:0x00000001849828 @method="GET", @headers={:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, @url="https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains", @cookies={}, @payload="", @user=nil, @password=nil, @timeout=nil, @open_timeout=nil, @block_response=nil, @raw_response=false, @verify_ssl=false, @ssl_client_cert=nil, @ssl_client_key=nil, @ssl_ca_file=nil, @tf=nil, @max_redirects=10, @processed_headers={"Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)", "Content-Length"=>"0", "Content-Type"=>"application/x-www-form-urlencoded"}, @args={:url=>"https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains", :method=>"GET", :headers=>{:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, :payload=>{}, :timeout=>nil}>
D, [2012-10-12T10:54:02.637625 #2332] DEBUG -- : Finding application :name => app1, :framework => 
D, [2012-10-12T10:54:02.637717 #2332] DEBUG -- : Getting all applications for domain joycedev3
D, [2012-10-12T10:54:02.637883 #2332] DEBUG -- : Request: #<RestClient::Request:0x000000022b48d0 @method="GET", @headers={:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, @url="https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains/joycedev3/applications", @cookies={}, @payload="", @user=nil, @password=nil, @timeout=nil, @open_timeout=nil, @block_response=nil, @raw_response=false, @verify_ssl=false, @ssl_client_cert=nil, @ssl_client_key=nil, @ssl_ca_file=nil, @tf=nil, @max_redirects=10, @processed_headers={"Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)", "Content-Length"=>"0", "Content-Type"=>"application/x-www-form-urlencoded"}, @args={:url=>"https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains/joycedev3/applications", :method=>"GET", :headers=>{:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, :payload=>{}, :timeout=>nil}>
D, [2012-10-12T10:54:04.010336 #2332] DEBUG -- : Running thread dump for app1
D, [2012-10-12T10:54:04.010613 #2332] DEBUG -- : Request: #<RestClient::Request:0x00000002332848 @method="POST", @headers={:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, @url="https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains/joycedev3/applications/app1/events", @cookies={}, @payload="event=thread-dump", @user=nil, @password=nil, @timeout=nil, @open_timeout=nil, @block_response=nil, @raw_response=false, @verify_ssl=false, @ssl_client_cert=nil, @ssl_client_key=nil, @ssl_ca_file=nil, @tf=nil, @max_redirects=10, @processed_headers={"Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)", "Content-Length"=>"17", "Content-Type"=>"application/x-www-form-urlencoded"}, @args={:url=>"https://ec2-54-242-155-252.compute-1.amazonaws.com/broker/rest/domains/joycedev3/applications/app1/events", :method=>"POST", :headers=>{:accept=>:json, "Authorization"=>"Basic amluemhhbmcrMUByZWRoYXQuY29tOmE=", "User-Agent"=>"rhc/0.99.5 (ruby 1.9.3; x86_64-linux)"}, :payload=>{:event=>"thread-dump"}, :timeout=>nil}>
Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.

Comment 4 Bill DeCoste 2012-10-13 01:49:40 UTC
Sorry about that, misread the bug. threaddump is now fixed.

Comment 5 joycezhang 2012-10-15 04:55:04 UTC
This bug has been fixed after verifying on devenv_2327. The threaddump file is available while using rhc threaddump as below:


# rhc threaddump app2
Password: *

Success

The thread dump file will be available via: rhc app tail -a app2 -f jbossews-1.0/jbossews-1.0/logs/catalina.out -o '-n 250'