Bug 981547 - Rack process of non-scaling ruby apps is killed after dumping its thread
Rack process of non-scaling ruby apps is killed after dumping its thread
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Command Line Interface (Show other bugs)
2.x
Unspecified Linux
medium Severity medium
: ---
: ---
Assigned To: Clayton Coleman
libra bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-05 02:43 EDT by Lei Zhang
Modified: 2015-05-14 22:24 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-22 11:21:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Lei Zhang 2013-07-05 02:43:04 EDT
Description of problem:
Given a non-scaling ruby app, its Rack process is killed after performing thread-dump. As a consequence, we need to access the app again in order to dump its thread a second time. On the other hand, there is no such problem for scaling ruby apps.

Version-Release number of selected component (if applicable):
devenv_3450

How reproducible:
always

Steps to Reproduce:
1. create a non-scaling ruby app (either ruby-1.8 or ruby-1.9)
rhc app create rb19 ruby-1.9
2. access this app to make sure it is available
3. SSH into this app, and check if its Rack process is running
ssh <ssh_url> "ps aux | grep Rack"
4. dump its thread via RHC
rhc threaddump rb19
5. check its Rack process again
6. dump the app's thread again via RHC
7. repeat Step 1 ~ 6 for a scaling ruby apps (either ruby-1.8 or ruby-1.9)

Actual results:
For the non-scaling ruby-1.9 app, we can see its Rack process is running in Step 2. However, the process is killed when checking in Step 5. Moreover, in Step 6, thread-dump fails with the warning:

Unable to detect application PID. Check the application's availability by accessing http://myrb19-rayzhang.dev.rhcloud.com

On the other hand, the Rack process of the scaling ruby app keeps running even after dumping its thread.

Expected results:
The rack process should be running after dumping the app's thread.

Additional info:
Comment 1 Hiro Asari 2013-07-05 11:27:52 EDT
This is by design due to the change in how threaddump is done. It's documented here: https://github.com/openshift/origin-server/tree/master/cartridges/openshift-origin-cartridge-ruby#threaddump-command
Comment 2 Lei Zhang 2013-07-07 22:37:37 EDT
The last sentence "Note also that scaled applications are not supported by the threaddump command" of the document https://github.com/openshift/origin-server/tree/master/cartridges/openshift-origin-cartridge-ruby#threaddump-command. My question is that since the scaled app are not suppported by the threaddump command, why the scaled ruby app still can do threaddump and Rack process won't be terminated. Is it a bug for the scaled ruby app?

Check on deven_3456

[rayzhang@rayzhang Work]$ rhc app create myruby19 ruby-1.9 -s 
Application Options
-------------------
  Namespace:  rayzhang
  Cartridges: ruby-1.9
  Gear Size:  default
  Scaling:    yes

Creating application 'myruby19' ... done

  Application myruby19 was created.
  HAProxy instance is started

Waiting for your DNS name to be available ... done

Initialized empty Git repository in /home/rayzhang/Work/myruby19/.git/

Your application 'myruby19' is now available.

  URL:        http://myruby19-rayzhang.dev.rhcloud.com/
  SSH to:     51da223d3265fa0c4c000001@myruby19-rayzhang.dev.rhcloud.com
  Git remote: ssh://51da223d3265fa0c4c000001@myruby19-rayzhang.dev.rhcloud.com/~/git/myruby19.git/
  Cloned to:  /home/rayzhang/Work/myruby19

Run 'rhc show-app myruby19' for more details about your app.

[rayzhang@rayzhang Work]$ rhc threaddump myruby19
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'

[rayzhang@rayzhang Work]$ rhc threaddump myruby19
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'
Success
The thread dump file will be available via: rhc tail myruby19 -f
/var/lib/openshift/51da223d3265fa0c4c000001/ruby//logs//error_log-20130708-* -o '-n 250'
Comment 3 Hiro Asari 2013-07-07 23:21:19 EDT
I think CLI is displaying the message indiscriminately. It might be reasonable to expect CLI to warn the user, in that case.

Reassigning this ticket to CLI.
Comment 4 Xiaoli Tian 2013-07-08 06:16:18 EDT
Note: thread dump for scaling app has been updated to allow thread dump scaled up gears  by ssh to the gears:
https://bugzilla.redhat.com/show_bug.cgi?id=974933#c4
https://bugzilla.redhat.com/show_bug.cgi?id=974933#c7
Comment 5 Clayton Coleman 2013-07-09 00:41:32 EDT
Fixed in https://bugzilla.redhat.com/show_bug.cgi?id=981780
Comment 6 Lei Zhang 2013-07-10 03:41:24 EDT
Verified on INT(devenv_3470), for non-scalable ruby app, it's work by design. so move it verified.


[rayzhang@rayzhang Work]$ rhc threaddump myruby18
Success
The thread dump file will be available via: rhc tail myruby18 -f /var/lib/openshift/51dcf64b6cec0e6d6b0000c2/ruby//logs//error_log-20130710-* -o '-n 250'

[rayzhang@rayzhang Work]$ rhc threaddump myruby18
 Unable to detect application PID. Check the application's availability by accessing http://myruby18-rayzhang.int.rhcloud.com

Note You need to log in before you can comment on or make changes to this bug.