Bug 1038592

Summary: Error 503: Service Temporarily Unavailable
Product: OpenShift Online Reporter: Anton Kirzyk <kirzyka>
Component: ImageAssignee: Dan Mace <dmace>
Status: CLOSED NOTABUG QA Contact: libra bugs <libra-bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.xCC: bparees, dmcphers, kirzyka
Target Milestone: ---Keywords: SupportQuestion
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-08 10:17:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Anton Kirzyk 2013-12-05 12:51:39 UTC 
Description of problem:
Error 503: Service Temporarily Unavailable

Version-Release number of selected component (if applicable):


How reproducible:
start application ( JBoss Enterprise Application Platform 6.1. )

Steps to Reproduce:
1. rhc app start o restart
2.
3.

Actual results:
503 Error

Expected results:
No Error

Additional info:
nyanya-cyberlogic.rhcloud.com
Comment 1 Dan Mace 2013-12-05 15:34:03 UTC 
Anton,

Can you please provide the contents of your JBoss server.log from a start/restart attempt? Without some sort of error details, there's no way to even try and attribute the problem to OpenShift, your application/code, or other external factors.

Thanks.
Comment 2 Anton Kirzyk 2013-12-05 16:09:12 UTC 
==> mysql/log/mysql_error.log <==
131205 11:04:11 [Note] /usr/libexec/mysqld: Normal shutdown

131205 11:04:11 [Note] Event Scheduler: Purging the queue. 0 events
131205 11:04:11
  InnoDB: Starting shutdown...
131205 11:04:12  InnoDB: Shutdown completed; log sequence number 0 44566
131205 11:04:12 [Note] /usr/libexec/mysqld: Shutdown complete

131205 11:04:12 mysqld_safe mysqld from pid file /var/lib/openshift/5263d8f05004468d670000b0/mysql//pid/mysql.pid ended
131205 11:04:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/5263d8f05004468d670000b0/mysql//data/
131205 11:04:15  InnoDB: Initializing buffer pool, size = 32.0M
131205 11:04:15  InnoDB: Completed initialization of buffer pool
131205 11:04:15  InnoDB: Started; log sequence number 0 44566
131205 11:04:15 [Note] Event Scheduler: Loaded 0 events
131205 11:04:15 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/openshift/5263d8f05004468d670000b0/mysql//socket/mysql.sock'  port: 3306  Source distribution

==> phpmyadmin/logs/error_log-20131205-000000-EST <==
[Thu Dec 05 11:04:17 2013] [notice] Digest: generating secret for digest authentication ...
[Thu Dec 05 11:04:17 2013] [notice] Digest: done
[Thu Dec 05 11:04:17 2013] [notice] Apache/2.2.22 (Unix) PHP/5.3.3 configured -- resuming normal operations

==> jbosseap/logs/server.log <==
2013/12/05 11:05:09,574 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.2.Final-redhat-1
2013/12/05 11:05:12,080 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA-redhat-1
2013/12/05 11:05:12,503 INFO  [org.jboss.as] (MSC service thread 1-1) JBAS015899: JBoss EAP 6.1.1.GA (AS 7.2.1.Final-redhat-10) starting
2013/12/05 11:05:18,982 ERROR [org.jboss.as.server] (Controller Boot Thread) JBAS015956: Caught exception during boot: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
        at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:141) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.server.ServerService.boot(ServerService.java:308) [jboss-as-server.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:188) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
Caused by: javax.xml.stream.XMLStreamException: JBAS014674: Failed to load module org.jboss.as.jpa
        at org.jboss.as.controller.parsing.ExtensionXml.parseExtensions(ExtensionXml.java:154) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.server.parsing.StandaloneXml.readServerElement_1_1(StandaloneXml.java:315) [jboss-as-server.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:133) [jboss-as-server.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:103) [jboss-as-server.jar:7.2.1.Final-redhat-10]
        at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper.jar:1.1.0.Final-redhat-2]
        at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69) [staxmapper.jar:1.1.0.Final-redhat-2]
        at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:133) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        ... 3 more
Caused by: java.util.concurrent.ExecutionException: javax.xml.stream.XMLStreamException: JBAS014674: Failed to load module
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_45]
        at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_45]
        at org.jboss.as.controller.parsing.ExtensionXml.parseExtensions(ExtensionXml.java:146) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        ... 9 more
Caused by: javax.xml.stream.XMLStreamException: JBAS014674: Failed to load module
        at org.jboss.as.controller.parsing.ExtensionXml.loadModule(ExtensionXml.java:195) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.controller.parsing.ExtensionXml.access$000(ExtensionXml.java:68) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.controller.parsing.ExtensionXml$1.call(ExtensionXml.java:126) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.controller.parsing.ExtensionXml$1.call(ExtensionXml.java:123) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads.jar:2.1.0.Final-redhat-1]
Caused by: org.jboss.modules.ModuleLoadException: Error loading module from /var/lib/openshift/5263d8f05004468d670000b0/jbosseap/modules/system/layers/base/org/jboss/as/jpa/main/module.xml
        at org.jboss.modules.ModuleXmlParser.parseModuleXml(ModuleXmlParser.java:292) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleXmlParser.parseModuleXml(ModuleXmlParser.java:256) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.LocalModuleFinder$1.run(LocalModuleFinder.java:144) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.LocalModuleFinder$1.run(LocalModuleFinder.java:138) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_45]
        at org.jboss.modules.LocalModuleFinder.fi
ndModule(LocalModuleFinder.java:138) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleLoader.findModule(ModuleLoader.java:389) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleLoader.loadModuleLocal(ModuleLoader.java:293) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleLoader.preloadModule(ModuleLoader.java:240) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:222) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.as.controller.parsing.ExtensionXml.loadModule(ExtensionXml.java:177) [jboss-as-controller.jar:7.2.1.Final-redhat-10]
        ... 8 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[31,49]
Message: Failed to add resource root 'jboss-as-jpa.jar' at path 'jboss-as-jpa.jar'
        at org.jboss.modules.ModuleXmlParser.parseResourceRoot(ModuleXmlParser.java:756) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleXmlParser.parseResources(ModuleXmlParser.java:712) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleXmlParser.parseModuleContents(ModuleXmlParser.java:538) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleXmlParser.parseDocument(ModuleXmlParser.java:369) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        at org.jboss.modules.ModuleXmlParser.parseModuleXml(ModuleXmlParser.java:287) [jboss-modules-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1]
        ... 18 more

2013/12/05 11:05:19,032 FATAL [org.jboss.as.server] (Controller Boot Thread) JBAS015957: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
2013/12/05 11:05:19,683 INFO  [org.jboss.as] (MSC service thread 1-4) JBAS015950: JBoss EAP 6.1.1.GA (AS 7.2.1.Final-redhat-10) stopped in 172ms

==> mysql/log/mysql_error.log <==
131205 11:07:27 [Note] /usr/libexec/mysqld: Normal shutdown

131205 11:07:27 [Note] Event Scheduler: Purging the queue. 0 events
131205 11:07:27  InnoDB: Starting shutdown...
131205 11:07:30  InnoDB: Shutdown completed; log sequence number 0 44566
131205 11:07:30 [Note] /usr/libexec/mysqld: Shutdown complete

131205 11:07:30 mysqld_safe mysqld from pid file /var/lib/openshift/5263d8f05004468d670000b0/mysql//pid/mysql.pid ended
131205 11:07:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/5263d8f05004468d670000b0/mysql//data/
131205 11:07:32  InnoDB: Initializing buffer pool, size = 32.0M
131205 11:07:32  InnoDB: Completed initialization of buffer pool
131205 11:07:32  InnoDB: Started; log sequence number 0 44566
131205 11:07:32 [Note] Event Scheduler: Loaded 0 events
131205 11:07:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/openshift/5263d8f05004468d670000b0/mysql//socket/mysql.sock'  port: 3306  Source distribution

==> phpmyadmin/logs/error_log-20131205-000000-EST <==
[Thu Dec 05 11:07:34 2013] [notice] SIGHUP received.  Attempting to restart
[Thu Dec 05 11:07:34 2013] [notice] Digest: generating secret for digest authentication ...
[Thu Dec 05 11:07:34 2013] [notice] Digest: done
[Thu Dec 05 11:07:34 2013] [notice] Apache/2.2.22 (Unix) PHP/5.3.3 configured -- resuming normal operations
Comment 3 Anton Kirzyk 2013-12-05 17:10:59 UTC 
rhc app tidy -a nyanya
DL is deprecated, please use Fiddle
Server returned an unexpected error code: 504
Comment 4 Anton Kirzyk 2013-12-05 19:27:17 UTC 
[nyanya-cyberlogic.rhcloud.com 5263d8f05004468d670000b0]\> gear restart --trace
Cart to restart?
1. mysql-5.1
2. jbosseap-6
3. phpmyadmin-4
?  2
Stopping jbosseap cartridge
Sending SIGTERM to jboss:400525 ...
Starting jbosseap cartridge
Timed out waiting for http listening port
An error occurred executing 'gear restart' (exit code: 1)
Error message: Failed to execute: 'control restart' for /var/lib/openshift/5263d8f05004468d670000b0/jbosseap

/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/v2_cart_model.rb:1262:in `block in do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/v2_cart_model.rb:1063:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/v2_cart_model.rb:1225:in `do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/v2_cart_model.rb:1085:in `do_control'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/v2_cart_model.rb:1447:in `start_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1200:in `restart_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1158:in `block in restart'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1105:in `rotate_and_yield'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.17.8/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:1069:in `block in with_gear_rotation'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:345:in `call_with_index'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:188:in `block (3 levels) in work_in_threads'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:352:in `with_instrumentation'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:186:in `block (2 levels) in work_in_threads'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:180:in `block in work_in_threads'
/opt/rh/ruby193/root/usr/share/gems/gems/parallel-0.8.0/lib/parallel.rb:65:in `block (2 levels) in in_threads'
Comment 5 Ben Parees 2014-01-06 22:51:03 UTC 
Are you still facing this issue?  Did you modify the application's standalone.xml as part of your development?  If so, can you attach it?