Bug 982298

Summary:	jbossas app is unable to start (as of devenv_3458)
Product:	OpenShift Online	Reporter:	Hiro Asari <hasari>
Component:	Containers	Assignee:	Rob Millner <rmillner>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.x	CC:	bmeng, jkeck, mfisher, mpatel, zhewang
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-22 15:22:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hiro Asari 2013-07-08 15:35:30 UTC

Description of problem:
Creation of a jbossas-7 app fails.

Version-Release number of selected component (if applicable):
devenv_3458

How reproducible:
almost always


Steps to Reproduce:
1. bx bin/rhc app create j7 jbossas-7 # this should succeed
2. cd j7; make a small edit, add and commit the change
3. git push

Actual results:
mvn artifacts are downloaded, then…

remote: Starting jbossas cart
remote: /var/lib/openshift/5de76e1ae7e311e2b76c12313d1fc210/jbossas/bin/control: fork: retry: Resource temporarily unavailable
remote: /var/lib/openshift/5de76e1ae7e311e2b76c12313d1fc210/jbossas/bin/control: fork: retry: Resource temporarily unavailable
remote: /var/lib/openshift/5de76e1ae7e311e2b76c12313d1fc210/jbossas/bin/control: fork: retry: Resource temporarily unavailable
remote: /var/lib/openshift/5de76e1ae7e311e2b76c12313d1fc210/jbossas/bin/control: fork: retry: Resource temporarily unavailable
remote: Timed out waiting for http listening port
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: Failed to execute: 'control start' for /var/lib/openshift/5de76e1ae7e311e2b76c12313d1fc210/jbossas


Expected results:
Success

Additional info:

Comment 1 Zhe Wang 2013-07-09 02:29:14 UTC

I was able to reproduce this bug in devenv_3462.

Another symptom of this bug is that JBossAS or JBossEAP apps are inaccessible after creation, i.e., homepages are blank and returning 404 errors in the HTTP response headers. The logs in server.log shows that it fails to start poa service:

2013/07/08 21:53:59,441 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC00001: Failed to start service jboss.jacorb.poa-service.rootpoa: org.jboss.msc.service.StartException in service jboss.jacorb.poa-service.rootpoa: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
        at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]
        at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]
        at org.jacorb.orb.etf.ListenerBase.listen(ListenerBase.java:117)
        at org.jacorb.orb.iiop.IIOPListener.listen(IIOPListener.java:183)
        at org.jacorb.orb.BasicAdapter.configure(BasicAdapter.java:110)
        at org.jacorb.orb.ORB.getRootPOA(ORB.java:1036)
        at org.jacorb.orb.ORB.resolve_initial_references(ORB.java:1307)
        at org.jboss.as.jacorb.service.CorbaPOAService.start(CorbaPOAService.java:153)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        ... 3 more

2013/07/08 21:54:00,045 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC00001: Failed to start service jboss.txn.ArjunaTransactionManager: org.jboss.msc.service.StartException in service jboss.txn.ArjunaTransactionManager: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
        at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]
        at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]
        at com.arjuna.ats.arjuna.coordinator.TransactionReaper.instantiate(TransactionReaper.java:868)
        at com.arjuna.ats.arjuna.coordinator.TransactionReaper.transactionReaper(TransactionReaper.java:883)
        at com.arjuna.ats.jbossatx.jta.TransactionManagerService.create(TransactionManagerService.java:68)
        at org.jboss.as.txn.service.ArjunaTransactionManagerService.start(ArjunaTransactionManagerService.java:122)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        ... 3 more

2013/07/08 21:54:00,246 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847)) Exception in thread "Thread-1 (HornetQ-scheduled-threads-8411847)" java.lang.OutOfMemoryError: unable to create new native thread
2013/07/08 21:54:00,248 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.lang.Thread.start0(Native Method)
2013/07/08 21:54:00,252 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.lang.Thread.start(Thread.java:693)
2013/07/08 21:54:00,258 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
2013/07/08 21:54:00,338 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
2013/07/08 21:54:00,339 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
2013/07/08 21:54:00,349 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2013/07/08 21:54:00,360 ERROR [stderr] (Thread-1 (HornetQ-scheduled-threads-8411847))   at java.lang.Thread.run(Thread.java:724)

Comment 2 Rob Millner 2013-07-09 03:06:35 UTC

There was a bug in the new pam code.

Pull request:
https://github.com/openshift/origin-server/pull/3001

Comment 3 Meng Bo 2013-07-09 08:07:11 UTC

Checked on devenv_3463, jbossas/eap app can be started successfully.

remote: Deploying JBoss
remote: Starting jbossas cart
remote: Found 127.1.2.129:8080 listening port
To ssh://5ff7edf8e86811e2bf6c22000a9aa086.rhcloud.com/~/git/jbas1.git/
   b23e5c3..992fdba  master -> master

Move bug to verified.