Bug 974933

https://github.com/openshift/origin-server/pull/2871

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/bca69b1e0ae1681bf78316212277a69a776aed1b
Bug 974933: Better handle threaddump for scaled up gears.

Test on devenv_3375, issue still can be reproduced.

[root@Daphne ~]# rhc threaddump r19
The rhc threaddump command is not supported on scaled gear: 260602421948395181572096. ssh
into the gear and issue ~/ruby/bin/control threaddump.
The rhc threaddump command is not supported on scaled gear: 260602421948395181572096. ssh
into the gear and issue ~/ruby/bin/control threaddump.
Success
The thread dump file will be available via: rhc tail r19 -f
/var/lib/openshift/540121037306486711123968/ruby//logs//error_log-20130618-* -o '-n 250'

The behavior has been modified to still show the threaddupmp logs from the head gear, but request the user to ssh into scaled up gears to get the threaddump there, locally.

The rhc threaddump command is not supported on scaled gear: 260602421948395181572096. ssh
into the gear and issue ~/ruby/bin/control threaddump.

according the below message, the rhc threaddump is supported on scaled gear,  user just need to ssh into the gear and control threaddump. maybe "The rhc threaddump command is not supported on scaled gear" is a little confused.

test on STG(devenv-stage_381)

threaddump works with scalable app. user should ssh into the app and run  ~/ruby/bin/control threaddump. and the log can be tailed normally too. but the waining message is still a little confused.

steps:
1. create a scalable app
  rhc app create r18s ruby-1.8 -s
2. scale up the app by rest api

3. visit the app in web and run rhc threaddump  

OpenShift:test duyan$ rhc threaddump r18s
The rhc threaddump command is not supported on scaled gear: 51cc0253dbd93c925e00086b. ssh
into the gear and issue ~/ruby/bin/control threaddump.
The rhc threaddump command is not supported on scaled gear: 51cc0253dbd93c925e00086b. ssh
into the gear and issue ~/ruby/bin/control threaddump.
Success
The thread dump file will be available via: rhc tail r18s -f
/var/lib/openshift/51cc01c82587c880fa0002ba/ruby//logs//error_log-20130627-* -o '-n 250'

4. ssh into the app and run ~/ruby/bin/control threaddump
[r18s-test.stg.rhcloud.com 51cc01c82587c880fa0002ba]\> ~/ruby/bin/control threaddump
    CLIENT_RESULT: Success
    CLIENT_RESULT: The thread dump file will be available via: rhc tail r18s -f /var/lib/openshift/51cc01c82587c880fa0002ba/ruby//logs//error_log-20130627-* -o '-n 250'

5. rhc tail the log, log can be tailed successfully.

$ rhc threaddump jbeap1s
Success
The thread dump file will be available via: rhc tail jbeap1s -f /tmp/jbosseap.log -o '-n 250'
Success
The thread dump file will be available via: rhc tail jbeap1s -f /tmp/jbosseap.log -o '-n 250'


It shows duplicate message for scaled-up app.

test devenv_3641, rhc threaddump a scale-up ruby-1.9 app got the below message:

Error occured while processing event 'thread-dump':The rhc threaddump command is not supported on scaled
gear: 7c345f2203ea11e3b4b522000a904a26. ssh into the gear and issue ~/ruby/bin/control threaddump. 

It is different with #Comment7

This seems to be fixed.
Below pasting the output of the  rhc threaddump. I've even tested the command on the gear and work as desired.


[jhadvig@jhadvig ~]$ rhc apps

fonendoskop @ http://fonendoskop-pankral.rhcloud.com/ (uuid: 524c1d224382ecbf200003b7)
--------------------------------------------------------------------------------------
  Domain:          pankral
  Created:         Oct 02  3:18 PM
  Gears:           2 (defaults to small)
  Git URL:         ssh://524c1d224382ecbf200003b7.com/~/git/fonendoskop.git/
  Initial Git URL: git://github.com/jhadvig/fondak.git
  SSH:             524c1d224382ecbf200003b7.com
  Aliases:         martinsky-fonendoskop.sk, www.martinsky-fonendoskop.sk

  mysql-5.1 (MySQL 5.1)
  ---------------------
    Gears:          1 small
    Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
    Database Name:  fonendoskop
    Password:       GgIT2SyKfIxx
    Username:       adminQCbAWz5

  ruby-1.9 (Ruby 1.9)
  -------------------
    Scaling: x1 (minimum: 1, maximum: available) on small gears

  haproxy-1.4 (Web Load Balancer)
  -------------------------------
    Gears: Located with ruby-1.9

You have 1 applications
[jhadvig@jhadvig ~]$ rhc threaddump fonendoskop
Success

The thread dump file will be available via: rhc tail fonendoskop -g 524c1d224382ecbf200003b7 -f

Test on devenv_3692

threaddump works well with both ruby-1.9 and jbossas-7 scalable app.

1. For ruby-1.9 app:

root@openshift-ubuntu:~/test# rhc threaddump r19s
Success
The thread dump file will be available via: rhc tail r19s -g 5279abeb0c7ba92271000007 -f
/var/lib/openshift/5279abeb0c7ba92271000007/ruby//logs//error_log-20131106-* -o '-n 250'
Success
The thread dump file will be available via: rhc tail r19s -g 5279ac6d0c7ba9077d000005 -f
/var/lib/openshift/5279ac6d0c7ba9077d000005/ruby//logs//error_log-20131106-* -o '-n 250'


2. For jbossas-7 app:

root@openshift-ubuntu:~/test# rhc threaddump j7s
Success

The thread dump file will be available via: rhc tail j7s -f /tmp/jbossas.log -o '-n 250'
Success

The thread dump file will be available via: rhc tail j7s -f /tmp/jbossas.log -o '-n 250'


Move to verified.

Sorry for my careless， For jboss app, still show duplicate message for scaled-up app. Will jboss app show different UUID in output for the scale-up gear as ruby app?

Yes, it will only right now the output of threaddump of the jbossas cartrigde doesnt show the gear UUID, thats why the rhc threaddump output appears as an duplicate message.

The below PR should fix this inconsistency for jbossas

https://github.com/openshift/origin-server/pull/4107

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/0a932b9feae96ae46e4cfe1c5346f486919fddfe
Bug 974933 - Inconsistent message is shown when rhc threaddump for a scaled up app

Test on devenv_4038

Show correct message when rhc threaddump jbossas app. And the log can be tailed normally.

[root@localhost ~]# rhc threaddump j7s
Success
The thread dump file will be available via: rhc tail j7s -g 528581b24784d56a3e000001 -f /tmp/jbossas.log -o '-n
250'
Success
The thread dump file will be available via: rhc tail j7s -g fe20ebd24d9a11e3888722000ab938f0 -f /tmp/jbossas.log -o
'-n 250'


But that would be perfect if the gear UUID can be shown when rhc threaddump jbosseap-6/jbossews-1.0/jbossews-2.0 apps.

Thanks.

Could you please more specify the problem, I don't quite follow where is the problem.

Tried the 'rhc threaddump j7s' and get: 

[root@ip-10-225-11-229 ~]# rhc threaddump j7s
Success

The thread dump file will be available via: rhc tail j7s -g 52dfd30e0b4d4e268b0000aa -f /tmp/jbossas.log -o '-n 250'


Thanks

@Jakub Hadvig, it works perfectly for the thread-dump of ruby/jbossas cartridges for now, but the rhc threaddump output still appears as an duplicate message for jbosseap/jbossews cartridges, so could you please help to check jbosseap and jbossews cartridges?

please refer to the following results:
1. Create a jbosseap/jbossews app
rhc app create ceap6s jbosseap-6 -s --no-git
2. Scale up this app
rhc cartridge-scale jbosseap-6 -a ceap6s --min 2
3. Thread-dump this app
$ rhc threaddump ceap6s
Success

The thread dump file will be available via: rhc tail ceap6s -f /tmp/jbosseap.log -o '-n 250'
Success

The thread dump file will be available via: rhc tail ceap6s -f /tmp/jbosseap.log -o '-n 250'

As above, the output messages do not show the gear UUID.

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/e0e859b61e19817fb632b075884aa775641eb651
Bug 974933 - Inconsistent message is shown when rhc threaddump for a scaled up app

Thanks for explanation.
Fixed.

PR: https://github.com/openshift/origin-server/pull/4569

It's fixed, verified on devenv_4267, please refer to the following results:

1. Create scaled jbosseap-6, jbossews-1.0 and jbossews-2.0 apps
rhc app create cews1s jbossews-1.0 -s --no-git
rhc app create cews2s jbossews-2.0 -s --no-git
rhc app create ceap6s jbosseap-6 -s --no-git

2. Scale up these apps
rhc cartridge-scale jbosseap-6 -a ceap6s --min 2
rhc cartridge-scale jbossews-1.0 -a cews1s --min 2
rhc cartridge-scale jbossews-2.0 -a cews2s --min 2

3. Thread-dump these apps
$ rhc threaddump ceap6s
Success

The thread dump file will be available via: rhc tail ceap6s -g 52e1c8191d7324a8710000ce -f /tmp/jbosseap.log -o '-n 250'
Success

The thread dump file will be available via: rhc tail ceap6s -g 412259622408311434903552 -f /tmp/jbosseap.log -o '-n 250'


$ rhc threaddump cews2s
Success

The thread dump file will be available via: rhc tail cews2s -g 52e1c8291d7324cd83000001 -f */logs/catalina.out -o '-n 250'
Success

The thread dump file will be available via: rhc tail cews2s -g 1d8de394849b11e3a9f812313907accd -f */logs/catalina.out -o '-n 250'

$ rhc threaddump cews1s
Success

The thread dump file will be available via: rhc tail cews1s -g 52e1c8591d732422c2000277 -f */logs/catalina.out -o '-n 250'
Success

The thread dump file will be available via: rhc tail cews1s -g 1597f8b4849b11e38b4c12313907accd -f */logs/catalina.out -o '-n 250'