Bug 981547

Summary: Rack process of non-scaling ruby apps is killed after dumping its thread
Product: OpenShift Online Reporter: Lei Zhang <lzhang>
Component: ocAssignee: Clayton Coleman <ccoleman>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: ccoleman, mpatel, xtian
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-22 15:21:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lei Zhang 2013-07-05 06:43:04 UTC
Description of problem:
Given a non-scaling ruby app, its Rack process is killed after performing thread-dump. As a consequence, we need to access the app again in order to dump its thread a second time. On the other hand, there is no such problem for scaling ruby apps.

Version-Release number of selected component (if applicable):
devenv_3450

How reproducible:
always

Steps to Reproduce:
1. create a non-scaling ruby app (either ruby-1.8 or ruby-1.9)
rhc app create rb19 ruby-1.9
2. access this app to make sure it is available
3. SSH into this app, and check if its Rack process is running
ssh <ssh_url> "ps aux | grep Rack"
4. dump its thread via RHC
rhc threaddump rb19
5. check its Rack process again
6. dump the app's thread again via RHC
7. repeat Step 1 ~ 6 for a scaling ruby apps (either ruby-1.8 or ruby-1.9)

Actual results:
For the non-scaling ruby-1.9 app, we can see its Rack process is running in Step 2. However, the process is killed when checking in Step 5. Moreover, in Step 6, thread-dump fails with the warning:

Unable to detect application PID. Check the application's availability by accessing http://myrb19-rayzhang.dev.rhcloud.com

On the other hand, the Rack process of the scaling ruby app keeps running even after dumping its thread.

Expected results:
The rack process should be running after dumping the app's thread.

Additional info:

Comment 1 Hiro Asari 2013-07-05 15:27:52 UTC
This is by design due to the change in how threaddump is done. It's documented here: https://github.com/openshift/origin-server/tree/master/cartridges/openshift-origin-cartridge-ruby#threaddump-command

Comment 2 Lei Zhang 2013-07-08 02:37:37 UTC
The last sentence "Note also that scaled applications are not supported by the threaddump command" of the document https://github.com/openshift/origin-server/tree/master/cartridges/openshift-origin-cartridge-ruby#threaddump-command. My question is that since the scaled app are not suppported by the threaddump command, why the scaled ruby app still can do threaddump and Rack process won't be terminated. Is it a bug for the scaled ruby app?

Check on deven_3456

[rayzhang@rayzhang Work]$ rhc app create myruby19 ruby-1.9 -s 
Application Options
-------------------
  Namespace:  rayzhang
  Cartridges: ruby-1.9
  Gear Size:  default
  Scaling:    yes

Creating application 'myruby19' ... done

  Application myruby19 was created.
  HAProxy instance is started

Waiting for your DNS name to be available ... done

Initialized empty Git repository in /home/rayzhang/Work/myruby19/.git/

Your application 'myruby19' is now available.

  URL:        http://myruby19-rayzhang.dev.rhcloud.com/
  SSH to:     51da223d3265fa0c4c000001.rhcloud.com
  Git remote: ssh://51da223d3265fa0c4c000001.rhcloud.com/~/git/myruby19.git/
  Cloned to:  /home/rayzhang/Work/myruby19

Run 'rhc show-app myruby19' for more details about your app.

[rayzhang@rayzhang Work]$ rhc threaddump myruby19
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'

[rayzhang@rayzhang Work]$ rhc threaddump myruby19
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'

Comment 3 Hiro Asari 2013-07-08 03:21:19 UTC
I think CLI is displaying the message indiscriminately. It might be reasonable to expect CLI to warn the user, in that case.

Reassigning this ticket to CLI.

Comment 4 Xiaoli Tian 2013-07-08 10:16:18 UTC
Note: thread dump for scaling app has been updated to allow thread dump scaled up gears  by ssh to the gears:
https://bugzilla.redhat.com/show_bug.cgi?id=974933#c4
https://bugzilla.redhat.com/show_bug.cgi?id=974933#c7

Comment 5 Clayton Coleman 2013-07-09 04:41:32 UTC
Fixed in https://bugzilla.redhat.com/show_bug.cgi?id=981780

Comment 6 Lei Zhang 2013-07-10 07:41:24 UTC
Verified on INT(devenv_3470), for non-scalable ruby app, it's work by design. so move it verified.


[rayzhang@rayzhang Work]$ rhc threaddump myruby18
Success
The thread dump file will be available via: rhc tail myruby18 -f /var/lib/openshift/51dcf64b6cec0e6d6b0000c2/ruby//logs//error_log-20130710-* -o '-n 250'

[rayzhang@rayzhang Work]$ rhc threaddump myruby18
 Unable to detect application PID. Check the application's availability by accessing http://myruby18-rayzhang.int.rhcloud.com