Bug 974933 - Inconsistent message is shown when rhc threaddump for a scaled up app
Inconsistent message is shown when rhc threaddump for a scaled up app
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Image (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Jakub Hadvig
libra bugs
:
Depends On:
Blocks: 978890
  Show dependency treegraph
 
Reported: 2013-06-17 03:44 EDT by Yan Du
Modified: 2015-05-14 20:33 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-26 14:07:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yan Du 2013-06-17 03:44:58 EDT
Description of problem:
It will show both  "The threaddump command not supported" and " the threaddump log"  for while trying to threaddump an scaling app which is scaled up:
The threaddump command is not supported on scalable applications.
The threaddump command is not supported on scalable applications.
Success
The thread dump file will be available via: rhc tail ry19s -f
/var/lib/openshift/fe084542d71a11e2ac9512313d04ead9/ruby//logs//error_log-20130617-* -o '-n 250'


Version-Release number of selected component (if applicable):
devenv_3368
rhc-1.10.1 (build from devenv_3368)


How reproducible:
Always


Steps to Reproduce:
1.Create a ruby-1.9 scaleable app: rhc app create ry19s ruby-1.9 -s
2.Scale the app by rest api  and visit the app in website
3.rhc threaddump ry19s


Actual results:
[root@Daphne test]# rhc threaddump ry19s
The threaddump command is not supported on scalable applications.
The threaddump command is not supported on scalable applications.
Success
The thread dump file will be available via: rhc tail ry19s -f
/var/lib/openshift/fe084542d71a11e2ac9512313d04ead9/ruby//logs//error_log-20130617-* -o '-n 250'


Expected results:
should not show below error message:
The threaddump command is not supported on scalable applications.
The threaddump command is not supported on scalable applications.
 
 
Additional info:
rhc threaddump works normally for a no scale app
Comment 2 openshift-github-bot 2013-06-17 20:42:46 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/bca69b1e0ae1681bf78316212277a69a776aed1b
Bug 974933: Better handle threaddump for scaled up gears.
Comment 3 Yan Du 2013-06-18 02:13:55 EDT
Test on devenv_3375, issue still can be reproduced.

[root@Daphne ~]# rhc threaddump r19
The rhc threaddump command is not supported on scaled gear: 260602421948395181572096. ssh
into the gear and issue ~/ruby/bin/control threaddump.
The rhc threaddump command is not supported on scaled gear: 260602421948395181572096. ssh
into the gear and issue ~/ruby/bin/control threaddump.
Success
The thread dump file will be available via: rhc tail r19 -f
/var/lib/openshift/540121037306486711123968/ruby//logs//error_log-20130618-* -o '-n 250'
Comment 4 Mrunal Patel 2013-06-18 11:22:50 EDT
The behavior has been modified to still show the threaddupmp logs from the head gear, but request the user to ssh into scaled up gears to get the threaddump there, locally.
Comment 5 Yan Du 2013-06-18 23:29:26 EDT
The rhc threaddump command is not supported on scaled gear: 260602421948395181572096. ssh
into the gear and issue ~/ruby/bin/control threaddump.

according the below message, the rhc threaddump is supported on scaled gear,  user just need to ssh into the gear and control threaddump. maybe "The rhc threaddump command is not supported on scaled gear" is a little confused.
Comment 7 Yan Du 2013-06-27 05:27:56 EDT
test on STG(devenv-stage_381)

threaddump works with scalable app. user should ssh into the app and run  ~/ruby/bin/control threaddump. and the log can be tailed normally too. but the waining message is still a little confused.

steps:
1. create a scalable app
  rhc app create r18s ruby-1.8 -s
2. scale up the app by rest api

3. visit the app in web and run rhc threaddump  

OpenShift:test duyan$ rhc threaddump r18s
The rhc threaddump command is not supported on scaled gear: 51cc0253dbd93c925e00086b. ssh
into the gear and issue ~/ruby/bin/control threaddump.
The rhc threaddump command is not supported on scaled gear: 51cc0253dbd93c925e00086b. ssh
into the gear and issue ~/ruby/bin/control threaddump.
Success
The thread dump file will be available via: rhc tail r18s -f
/var/lib/openshift/51cc01c82587c880fa0002ba/ruby//logs//error_log-20130627-* -o '-n 250'

4. ssh into the app and run ~/ruby/bin/control threaddump
[r18s-test.stg.rhcloud.com 51cc01c82587c880fa0002ba]\> ~/ruby/bin/control threaddump
    CLIENT_RESULT: Success
    CLIENT_RESULT: The thread dump file will be available via: rhc tail r18s -f /var/lib/openshift/51cc01c82587c880fa0002ba/ruby//logs//error_log-20130627-* -o '-n 250'

5. rhc tail the log, log can be tailed successfully.
Comment 8 Meng Bo 2013-07-31 04:34:59 EDT
$ rhc threaddump jbeap1s
Success
The thread dump file will be available via: rhc tail jbeap1s -f /tmp/jbosseap.log -o '-n 250'
Success
The thread dump file will be available via: rhc tail jbeap1s -f /tmp/jbosseap.log -o '-n 250'


It shows duplicate message for scaled-up app.
Comment 9 Yan Du 2013-08-13 03:59:01 EDT
test devenv_3641, rhc threaddump a scale-up ruby-1.9 app got the below message:

Error occured while processing event 'thread-dump':The rhc threaddump command is not supported on scaled
gear: 7c345f2203ea11e3b4b522000a904a26. ssh into the gear and issue ~/ruby/bin/control threaddump. 

It is different with #Comment7
Comment 10 Jakub Hadvig 2013-11-05 11:08:44 EST
This seems to be fixed.
Below pasting the output of the  rhc threaddump. I've even tested the command on the gear and work as desired.


[jhadvig@jhadvig ~]$ rhc apps

fonendoskop @ http://fonendoskop-pankral.rhcloud.com/ (uuid: 524c1d224382ecbf200003b7)
--------------------------------------------------------------------------------------
  Domain:          pankral
  Created:         Oct 02  3:18 PM
  Gears:           2 (defaults to small)
  Git URL:         ssh://524c1d224382ecbf200003b7@fonendoskop-pankral.rhcloud.com/~/git/fonendoskop.git/
  Initial Git URL: git://github.com/jhadvig/fondak.git
  SSH:             524c1d224382ecbf200003b7@fonendoskop-pankral.rhcloud.com
  Aliases:         martinsky-fonendoskop.sk, www.martinsky-fonendoskop.sk

  mysql-5.1 (MySQL 5.1)
  ---------------------
    Gears:          1 small
    Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
    Database Name:  fonendoskop
    Password:       GgIT2SyKfIxx
    Username:       adminQCbAWz5

  ruby-1.9 (Ruby 1.9)
  -------------------
    Scaling: x1 (minimum: 1, maximum: available) on small gears

  haproxy-1.4 (Web Load Balancer)
  -------------------------------
    Gears: Located with ruby-1.9

You have 1 applications
[jhadvig@jhadvig ~]$ rhc threaddump fonendoskop
Success

The thread dump file will be available via: rhc tail fonendoskop -g 524c1d224382ecbf200003b7 -f
Comment 11 Yan Du 2013-11-05 21:55:15 EST
Test on devenv_3692

threaddump works well with both ruby-1.9 and jbossas-7 scalable app.

1. For ruby-1.9 app:

root@openshift-ubuntu:~/test# rhc threaddump r19s
Success
The thread dump file will be available via: rhc tail r19s -g 5279abeb0c7ba92271000007 -f
/var/lib/openshift/5279abeb0c7ba92271000007/ruby//logs//error_log-20131106-* -o '-n 250'
Success
The thread dump file will be available via: rhc tail r19s -g 5279ac6d0c7ba9077d000005 -f
/var/lib/openshift/5279ac6d0c7ba9077d000005/ruby//logs//error_log-20131106-* -o '-n 250'


2. For jbossas-7 app:

root@openshift-ubuntu:~/test# rhc threaddump j7s
Success

The thread dump file will be available via: rhc tail j7s -f /tmp/jbossas.log -o '-n 250'
Success

The thread dump file will be available via: rhc tail j7s -f /tmp/jbossas.log -o '-n 250'


Move to verified.
Comment 12 Yan Du 2013-11-05 22:00:23 EST
Sorry for my careless, For jboss app, still show duplicate message for scaled-up app. Will jboss app show different UUID in output for the scale-up gear as ruby app?
Comment 13 Jakub Hadvig 2013-11-06 09:30:33 EST
Yes, it will only right now the output of threaddump of the jbossas cartrigde doesnt show the gear UUID, thats why the rhc threaddump output appears as an duplicate message.

The below PR should fix this inconsistency for jbossas

https://github.com/openshift/origin-server/pull/4107
Comment 14 openshift-github-bot 2013-11-06 13:19:11 EST
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/0a932b9feae96ae46e4cfe1c5346f486919fddfe
Bug 974933 - Inconsistent message is shown when rhc threaddump for a scaled up app
Comment 15 Yan Du 2013-11-14 21:22:30 EST
Test on devenv_4038

Show correct message when rhc threaddump jbossas app. And the log can be tailed normally.

[root@localhost ~]# rhc threaddump j7s
Success
The thread dump file will be available via: rhc tail j7s -g 528581b24784d56a3e000001 -f /tmp/jbossas.log -o '-n
250'
Success
The thread dump file will be available via: rhc tail j7s -g fe20ebd24d9a11e3888722000ab938f0 -f /tmp/jbossas.log -o
'-n 250'


But that would be perfect if the gear UUID can be shown when rhc threaddump jbosseap-6/jbossews-1.0/jbossews-2.0 apps.

Thanks.
Comment 16 Jakub Hadvig 2014-01-22 09:31:35 EST
Could you please more specify the problem, I don't quite follow where is the problem.

Tried the 'rhc threaddump j7s' and get: 

[root@ip-10-225-11-229 ~]# rhc threaddump j7s
Success

The thread dump file will be available via: rhc tail j7s -g 52dfd30e0b4d4e268b0000aa -f /tmp/jbossas.log -o '-n 250'


Thanks
Comment 17 chunchen 2014-01-22 22:28:10 EST
@Jakub Hadvig, it works perfectly for the thread-dump of ruby/jbossas cartridges for now, but the rhc threaddump output still appears as an duplicate message for jbosseap/jbossews cartridges, so could you please help to check jbosseap and jbossews cartridges?

please refer to the following results:
1. Create a jbosseap/jbossews app
rhc app create ceap6s jbosseap-6 -s --no-git
2. Scale up this app
rhc cartridge-scale jbosseap-6 -a ceap6s --min 2
3. Thread-dump this app
$ rhc threaddump ceap6s
Success

The thread dump file will be available via: rhc tail ceap6s -f /tmp/jbosseap.log -o '-n 250'
Success

The thread dump file will be available via: rhc tail ceap6s -f /tmp/jbosseap.log -o '-n 250'

As above, the output messages do not show the gear UUID.
Comment 18 openshift-github-bot 2014-01-23 06:33:11 EST
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/e0e859b61e19817fb632b075884aa775641eb651
Bug 974933 - Inconsistent message is shown when rhc threaddump for a scaled up app
Comment 19 Jakub Hadvig 2014-01-23 06:59:06 EST
Thanks for explanation.
Fixed.

PR: https://github.com/openshift/origin-server/pull/4569
Comment 20 chunchen 2014-01-23 21:05:00 EST
It's fixed, verified on devenv_4267, please refer to the following results:

1. Create scaled jbosseap-6, jbossews-1.0 and jbossews-2.0 apps
rhc app create cews1s jbossews-1.0 -s --no-git
rhc app create cews2s jbossews-2.0 -s --no-git
rhc app create ceap6s jbosseap-6 -s --no-git

2. Scale up these apps
rhc cartridge-scale jbosseap-6 -a ceap6s --min 2
rhc cartridge-scale jbossews-1.0 -a cews1s --min 2
rhc cartridge-scale jbossews-2.0 -a cews2s --min 2

3. Thread-dump these apps
$ rhc threaddump ceap6s
Success

The thread dump file will be available via: rhc tail ceap6s -g 52e1c8191d7324a8710000ce -f /tmp/jbosseap.log -o '-n 250'
Success

The thread dump file will be available via: rhc tail ceap6s -g 412259622408311434903552 -f /tmp/jbosseap.log -o '-n 250'


$ rhc threaddump cews2s
Success

The thread dump file will be available via: rhc tail cews2s -g 52e1c8291d7324cd83000001 -f */logs/catalina.out -o '-n 250'
Success

The thread dump file will be available via: rhc tail cews2s -g 1d8de394849b11e3a9f812313907accd -f */logs/catalina.out -o '-n 250'

$ rhc threaddump cews1s
Success

The thread dump file will be available via: rhc tail cews1s -g 52e1c8591d732422c2000277 -f */logs/catalina.out -o '-n 250'
Success

The thread dump file will be available via: rhc tail cews1s -g 1597f8b4849b11e38b4c12313907accd -f */logs/catalina.out -o '-n 250'

Note You need to log in before you can comment on or make changes to this bug.