Bug 982992

Summary: threaddump doesn't work if scale the app to 4 gears (from 3 gears)
Product: OpenShift Online Reporter: Yan Du <yadu>
Component: ContainersAssignee: Jhon Honce <jhonce>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: low    
Version: 2.xCC: xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-17 13:27:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yan Du 2013-07-10 09:18:09 UTC 
Description of problem:
create a app, scale up to 3 gears, the threaddump can be works normally, scale up again, it fail to threaddump and show "Unable to detect application PID. Check the application's availability by accessing http://r18s7-111.int.rhcloud.com"




Version-Release number of selected component (if applicable):
INT (devenv_3470)
rhc-1.11.3




How reproducible:
Always



Steps to Reproduce:

1. Create a scalable app, visit app in web and run rhc threaddump

2. scale up to 3 gears, visit app in web and run rhc threaddump

3. scale up to 4 gears, visit app in web and run rhc threaddump

4. check the app status and visit the app again in web and tun rhc threaddump




Actual results:

step1&2,threaddump works

step3, threaddump doesn't work and show "Unable to detect application PID. Check the application's availability by accessing
http://r18s7-111.int.rhcloud.com "

[root@localhost test]# rhc cartridge scale -a r18s7 ruby-1.8 --min 3
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x3 (minimum: 3, maximum: available) on small gears

[root@localhost test]# rhc threaddump r18s7
Success
The thread dump file will be available via: rhc tail r18s7 -f
/var/lib/openshift/51dd1d3d6cec0e0f6900018a/ruby//logs//error_log-20130710-* -o '-n 250'

[root@localhost test]# rhc cartridge scale -a r18s7 ruby-1.8 --min 4
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x4 (minimum: 4, maximum: available) on small gears
  
[root@localhost test]# rhc threaddump r18s7
 Unable to detect application PID. Check the application's availability by accessing
http://r18s7-111.int.rhcloud.com


step4: app still can be accessed and the status is started. 
run rhc threaddump still got the message 

[root@localhost test]# rhc threaddump r18s7
 Unable to detect application PID. Check the application's availability by accessing
http://r18s7-111.int.rhcloud.com



Expected results:
rhc threaddump works normally



Additional info:

if just scale up app to 4 gears for one time, threaddumo works:
 
[root@localhost test]# rhc scale-cartridge -a r18s2 ruby-1.8 --min 4
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x4 (minimum: 4, maximum: available) on small gears


[root@localhost test]# rhc threaddump r18s2
Success
The thread dump file will be available via: rhc tail r18s2 -f
/var/lib/openshift/51dd0c1e6cec0ee92a0006b7/ruby//logs//error_log-20130710-* -o '-n 250'
Comment 1 Jhon Honce 2013-09-23 18:15:52 UTC 
This is an artifact of the Ruby cartridge's threaddump implementation. You can only successfully run threaddump for all gears, after the URL for each gear has been accessed.

To verify:

After running rhc threaddump <application>, ssh into failing gear(s) and run:
 curl http://$OPENSHIFT_RUBY_IP:8080

Now run rhc threaddump <application>, all gears will report success.

JBossAS cartridge does not exhibit this behavior.
Comment 3 Yan Du 2013-09-24 06:39:27 UTC 
test devenv_3816

after got the failed threaddump information, ssh into the failed gear and run the command "http://$OPENSHIFT_RUBY_IP:8080" , then run rhc threaddump $app again, threaddump can work normally.


[root@Daphne test]# rhc threaddump r18s
Success

The thread dump file will be available via: rhc tail r18s -g 5241307c856551e68300048e -f
/var/lib/openshift/5241307c856551e68300048e/ruby//logs//error_log-20130924-* -o '-n 250'
Success

The thread dump file will be available via: rhc tail r18s -g 524131278565510ada000054 -f
/var/lib/openshift/524131278565510ada000054/ruby//logs//error_log-20130924-* -o '-n 250'
Success

The thread dump file will be available via: rhc tail r18s -g 9613f35824e211e3b85d12313d33824f -f
/var/lib/openshift/9613f35824e211e3b85d12313d33824f/ruby//logs//error_log-20130924-* -o '-n 250'

move bug to verified.
Success

The thread dump file will be available via: rhc tail r18s -g de306f7c24e211e3b85d12313d33824f -f
/var/lib/openshift/de306f7c24e211e3b85d12313d33824f/ruby//logs//error_log-20130924-* -o '-n 250'