Bug 984765 - Corrupt instance w/ incorrect Tomcat data dir, thus Tomcat hangs during startup.
Summary: Corrupt instance w/ incorrect Tomcat data dir, thus Tomcat hangs during startup.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Dan Mace
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 990564
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-15 22:25 UTC by Vasco Veloso
Modified: 2015-05-14 23:23 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-08-29 12:48:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 1 Dan Mace 2013-07-16 15:37:27 UTC
Marking UpcomingRelease; evidence indicates this is an isolated incident potentially related to an old migration. I'll follow up with ops to try and repair the gear.

Comment 2 Dan Mace 2013-07-29 19:41:11 UTC
Upon revisiting this issue, I was unable to determine the root cause of the app startup failure. I've inspected the environment variables on the gear, and everything looks fine. The logs provided by the user contain the following:

  WARNING: Problem with directory [/var/lib/openshift/512f77595973ca795c000303/data], exists: [false], isDirectory: [false], canRead: [false]

It's unclear whether the warning is related. The path in the warning refers to a non-existent directory, and the configuration which led to the warning is in the user's control.

Vasco, are you willing to give me permission to inspect the log files, application code, and other internals of your application which is currently deployed? If so, perhaps we can get more insight into the specific failure which is occurring.

Comment 3 Vasco Veloso 2013-07-29 22:23:43 UTC
Dan, you are authorized to access and/or inspect whatever you may find necessary in the application. I hope that it will help you understand what may be causing this specific failure.

Comment 4 Dan Mace 2013-07-31 13:23:02 UTC
This application is suffering from a wedge state due to an upstream openjdk regression (bug 976693). I've opened a new bug to track updating openjdk on OpenShift to a fixed version (bug 990564).

Once the new openjdk is deployed to OpenShift, this particular application will also need to update to log4j 2.0beta7 to get around a bug introduced in 2.0beta3 (something I happened to notice while debugging). The custom appender will fail to initialize due to an uncaught empty stack exception in this log4j code:

http://grepcode.com/file_/repo1.maven.org/maven2/org.apache.logging.log4j/log4j-core/2.0-beta7/org/apache/logging/log4j/core/impl/ThrowableProxy.java/?v=diff&id2=2.0-beta4#339

Comment 5 Dan Mace 2013-08-21 19:08:28 UTC
The OpenJDK package used by OpenShift has been updated to java-1.7.0-openjdk-1.7.0.25-2.4.1.1.el6.i686 which contains the regression fix.

The new package will be rolled out this release.

Comment 6 Meng Bo 2013-08-22 10:04:29 UTC
Checked on latest devenv_3684, the openjdk version is 1.7.0.25-2.4 now 

Move bug to verified.

[root@ip-10-179-21-14 ~]# rpm -qa|grep openjdk
java-1.7.0-openjdk-1.7.0.25-2.4.1.1.el6.i686
java-1.6.0-openjdk-devel-1.6.0.0-1.62.1.11.11.90.el6_4.i686
java-1.6.0-openjdk-1.6.0.0-1.62.1.11.11.90.el6_4.i686
java-1.7.0-openjdk-devel-1.7.0.25-2.4.1.1.el6.i686


To Vasco, if you meet this issue again after the update pushed out, please reopen this bug.
Usually, this release should be online on Mon or Tue next week.

Thanks.


Note You need to log in before you can comment on or make changes to this bug.