Bug 967426 - Threaddump is not working for both Ruby and JBoss applications
Threaddump is not working for both Ruby and JBoss applications
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-27 02:07 EDT by Zhe Wang
Modified: 2015-05-14 20:17 EDT (History)
3 users (show)

See Also:
Fixed In Version: devenv_3282
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 00:11:35 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Zhe Wang 2013-05-27 02:07:58 EDT
Description of problem:
Given a Ruby or JBoss application, generating a thread dump fails via both RHC and REST API.

Version-Release number of selected component (if applicable):
devenv_3277

How reproducible:
always

Steps to Reproduce:
1. create a Ruby or JBoss application

2. access this app before generating a thread dump

3. generate a thread dump of this app via RHC
e.g.,
rhc threaddump rb9

4. generate a thread dump via REST API
e.g.,
curl -k -H "Accept:application/json" --user zhewang+1@redhat.com:redhat https://ec2-54-234-163-159.compute-1.amazonaws.com/broker/rest/domains/dev3277tst/applications/as/events -d event=thread-dump -X POST | python -m json.tool

Actual results:

Generating a thread dump via either RHC or REST API fails with the same error:

1) RHC:

The threaddump command is not available for this application

2) REST API:

"messages": [
        {
            "exit_code": 180, 
            "field": null, 
            "severity": "error", 
            "text": "The threaddump command is not available for this application"
        }
    ], 

Expected results:
Generating a thread dump of a Ruby or JBoss application should be successful.

Additional info:
Comment 1 Jhon Honce 2013-05-27 15:26:15 EDT
The manifests for the given cartridges have the necessary elements:

Additional-Control-Actions:
    - threaddump
Comment 2 Lili Nader 2013-05-28 17:13:51 EDT
Works for me with the latest code.
Comment 3 Abhishek Gupta 2013-05-28 18:33:09 EDT
I was able to reproduce this bug and have added the fix in this pull request --> https://github.com/openshift/origin-server/pull/2664
Comment 4 Abhishek Gupta 2013-05-28 18:33:58 EDT
Moving it back from ON_QA to MODIFIED and assigning it to myself. Will mark it ON_QA once the fix is merged.
Comment 6 Zhe Wang 2013-05-29 01:45:59 EDT
Verified in devenv_3282.

The verification covered Ruby-1.9 and JBossAS applications, and dumping the apps' thread succeeded.

For the Ruby-1.9 app:

[zhe@fedora run80]$ rhc threaddump rb9
Success
The thread dump file will be available via: rhc tail rb9 -f /var/lib/openshift/583185735959884328861696/ruby//logs//error_log-20130529-* -o '-n 250'
[zhe@fedora run80]$ rhc tail rb9 -f /var/lib/openshift/583185735959884328861696/ruby//logs//error_log-20130529-* -o '-n 250'
[Wed May 29 01:38:33 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c0,c501
[Wed May 29 01:38:33 2013] [error] (2)No such file or directory: mod_mime_magic: can't read magic file /var/lib/openshift/583185735959884328861696/ruby/configuration/magic
[Wed May 29 01:38:33 2013] [error] (2)No such file or directory: mod_mime_magic: can't read magic file /var/lib/openshift/583185735959884328861696/ruby/configuration/magic
[Wed May 29 01:38:33 2013] [notice] Digest: generating secret for digest authentication ...
[Wed May 29 01:38:33 2013] [notice] Digest: done
[Wed May 29 01:38:33 2013] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.17 configured -- resuming normal operations
*** Exception SignalException in Class (SIGABRT) (process 5951, thread #<Thread:0x000000019f6108>):
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:444:in `block in install_useful_signal_handlers'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:459:in `call'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:459:in `select'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:459:in `accept_and_process_next_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/utils.rb:470:in `safe_fork'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/passenger-spawn-server:102:in `<main>'

For the JBossAS app:
[zhe@fedora run80]$  rhc tail as -f */logs/server.log -o '-n 250'
2013/05/29 01:42:48,956 INFO  [org.jboss.jaxr] (MSC service thread 1-1) JBAS014000: Started JAXR subsystem, binding JAXR connection factory into JNDI as: java:jboss/jaxr/ConnectionFactory
2013/05/29 01:42:49,046 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default]
2013/05/29 01:42:49,093 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 38) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
2013/05/29 01:42:49,190 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 38) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
2013/05/29 01:42:49,485 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-1) Starting Coyote HTTP/1.1 on http--127.0.252.1-8080
2013/05/29 01:42:49,942 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-2) JBoss Web Services - Stack CXF Server 4.0.2.GA
2013/05/29 01:42:50,767 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) JBAS016330: CORBA ORB Service started
2013/05/29 01:42:50,881 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) JBAS015012: Started FileSystemDeploymentService for directory /var/lib/openshift/51a59528b1ab7b0713000001/jbossas/standalone/deployments
2013/05/29 01:42:51,188 WARN  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011600: AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
2013/05/29 01:42:52,982 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/var/lib/openshift/51a59528b1ab7b0713000001/jbossas/standalone/data/messagingjournal,bindingsDirectory=/var/lib/openshift/51a59528b1ab7b0713000001/jbossas/standalone/data/messagingbindings,largeMessagesDirectory=/var/lib/openshift/51a59528b1ab7b0713000001/jbossas/standalone/data/messaginglargemessages,pagingDirectory=/var/lib/openshift/51a59528b1ab7b0713000001/jbossas/standalone/data/messagingpaging)
2013/05/29 01:42:53,074 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) Waiting to obtain live lock
2013/05/29 01:42:53,390 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-1) Using NIO Journal
2013/05/29 01:42:53,784 INFO  [org.jboss.as.jacorb] (MSC service thread 1-2) JBAS016328: CORBA Naming Service started
2013/05/29 01:42:53,791 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
2013/05/29 01:42:54,121 INFO  [org.hornetq.core.server.impl.FileLockNodeManager] (MSC service thread 1-1) Waiting to obtain live lock
2013/05/29 01:42:54,122 INFO  [org.hornetq.core.server.impl.FileLockNodeManager] (MSC service thread 1-1) Live Server Obtained live lock
2013/05/29 01:42:55,422 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-1) Started Netty Acceptor version 3.2.5.Final-a96d88c 127.0.252.1:5455 for CORE protocol
2013/05/29 01:42:55,476 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-1) Started Netty Acceptor version 3.2.5.Final-a96d88c 127.0.252.1:5445 for CORE protocol
2013/05/29 01:42:55,485 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) Server is now live
2013/05/29 01:42:55,486 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) HornetQ Server version 2.2.13.Final (HQ_2_2_13_FINAL_AS7, 122) [9b049c22-c822-11e2-a0b6-22000a98a5f2]) started
2013/05/29 01:42:55,515 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) trying to deploy queue jms.queue.testQueue
2013/05/29 01:42:56,122 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/queue/test
2013/05/29 01:42:56,174 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
2013/05/29 01:42:56,184 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) trying to deploy queue jms.topic.testTopic
2013/05/29 01:42:56,377 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/topic/test
2013/05/29 01:42:56,382 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
2013/05/29 01:42:56,587 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
2013/05/29 01:42:56,592 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
2013/05/29 01:42:56,597 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory

2013/05/29 01:42:57,078 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-1) JBAS010406: Registered connection factory java:/JmsXA
2013/05/29 01:42:57,182 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-1) HornetQ resource adaptor started
2013/05/29 01:42:57,186 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) IJ020002: Deployed: file://RaActivatorhornetq-ra
2013/05/29 01:42:57,192 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
2013/05/29 01:42:57,515 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "ROOT.war"
2013/05/29 01:43:02,848 INFO  [org.jboss.web] (MSC service thread 1-1) JBAS018210: Registering web context: 
2013/05/29 01:43:02,895 INFO  [org.jboss.as] (MSC service thread 1-1) JBAS015951: Admin console listening on http://127.0.252.1:9990
2013/05/29 01:43:02,897 INFO  [org.jboss.as] (MSC service thread 1-1) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 26996ms - Started 212 of 331 services (116 services are passive or on-demand)
2013/05/29 01:43:03,504 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "ROOT.war"

Note You need to log in before you can comment on or make changes to this bug.