Bug 982992 - threaddump doesn't work if scale the app to 4 gears (from 3 gears)
threaddump doesn't work if scale the app to 4 gears (from 3 gears)
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
low Severity medium
: ---
: ---
Assigned To: Jhon Honce
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-10 05:18 EDT by Yan Du
Modified: 2015-05-14 19:23 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-17 09:27:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yan Du 2013-07-10 05:18:09 EDT
Description of problem:
create a app, scale up to 3 gears, the threaddump can be works normally, scale up again, it fail to threaddump and show "Unable to detect application PID. Check the application's availability by accessing http://r18s7-111.int.rhcloud.com"




Version-Release number of selected component (if applicable):
INT (devenv_3470)
rhc-1.11.3




How reproducible:
Always



Steps to Reproduce:

1. Create a scalable app, visit app in web and run rhc threaddump

2. scale up to 3 gears, visit app in web and run rhc threaddump

3. scale up to 4 gears, visit app in web and run rhc threaddump

4. check the app status and visit the app again in web and tun rhc threaddump




Actual results:

step1&2,threaddump works

step3, threaddump doesn't work and show "Unable to detect application PID. Check the application's availability by accessing
http://r18s7-111.int.rhcloud.com "

[root@localhost test]# rhc cartridge scale -a r18s7 ruby-1.8 --min 3
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x3 (minimum: 3, maximum: available) on small gears

[root@localhost test]# rhc threaddump r18s7
Success
The thread dump file will be available via: rhc tail r18s7 -f
/var/lib/openshift/51dd1d3d6cec0e0f6900018a/ruby//logs//error_log-20130710-* -o '-n 250'

[root@localhost test]# rhc cartridge scale -a r18s7 ruby-1.8 --min 4
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x4 (minimum: 4, maximum: available) on small gears
  
[root@localhost test]# rhc threaddump r18s7
 Unable to detect application PID. Check the application's availability by accessing
http://r18s7-111.int.rhcloud.com


step4: app still can be accessed and the status is started. 
run rhc threaddump still got the message 

[root@localhost test]# rhc threaddump r18s7
 Unable to detect application PID. Check the application's availability by accessing
http://r18s7-111.int.rhcloud.com



Expected results:
rhc threaddump works normally



Additional info:

if just scale up app to 4 gears for one time, threaddumo works:
 
[root@localhost test]# rhc scale-cartridge -a r18s2 ruby-1.8 --min 4
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x4 (minimum: 4, maximum: available) on small gears


[root@localhost test]# rhc threaddump r18s2
Success
The thread dump file will be available via: rhc tail r18s2 -f
/var/lib/openshift/51dd0c1e6cec0ee92a0006b7/ruby//logs//error_log-20130710-* -o '-n 250'
Comment 1 Jhon Honce 2013-09-23 14:15:52 EDT
This is an artifact of the Ruby cartridge's threaddump implementation. You can only successfully run threaddump for all gears, after the URL for each gear has been accessed.

To verify:

After running rhc threaddump <application>, ssh into failing gear(s) and run:
 curl http://$OPENSHIFT_RUBY_IP:8080

Now run rhc threaddump <application>, all gears will report success.

JBossAS cartridge does not exhibit this behavior.
Comment 3 Yan Du 2013-09-24 02:39:27 EDT
test devenv_3816

after got the failed threaddump information, ssh into the failed gear and run the command "http://$OPENSHIFT_RUBY_IP:8080" , then run rhc threaddump $app again, threaddump can work normally.


[root@Daphne test]# rhc threaddump r18s
Success

The thread dump file will be available via: rhc tail r18s -g 5241307c856551e68300048e -f
/var/lib/openshift/5241307c856551e68300048e/ruby//logs//error_log-20130924-* -o '-n 250'
Success

The thread dump file will be available via: rhc tail r18s -g 524131278565510ada000054 -f
/var/lib/openshift/524131278565510ada000054/ruby//logs//error_log-20130924-* -o '-n 250'
Success

The thread dump file will be available via: rhc tail r18s -g 9613f35824e211e3b85d12313d33824f -f
/var/lib/openshift/9613f35824e211e3b85d12313d33824f/ruby//logs//error_log-20130924-* -o '-n 250'

move bug to verified.
Success

The thread dump file will be available via: rhc tail r18s -g de306f7c24e211e3b85d12313d33824f -f
/var/lib/openshift/de306f7c24e211e3b85d12313d33824f/ruby//logs//error_log-20130924-* -o '-n 250'

Note You need to log in before you can comment on or make changes to this bug.