Class loading breakage regression sound like a serious issue. If you could narrow it down that would be great!

Assigning to Andrew to track.

Do you have an example stack trace from this error?

Here is the stack trace for the one mentioned.  This would be obviously an application issue, except the class is found in previous versions (and has been found since installation in 2013) and is found again by simply downgrading java and restarting the app.

java.lang.NoClassDefFoundError: Could not initialize class org.jfree.chart.JFreeChart
	at org.jfree.chart.ChartFactory.createLineChart3D(ChartFactory.java:1296)
	at org.adempiere.apps.graph.GraphBuilder.createLineChart(GraphBuilder.java:170)
	at org.adempiere.apps.graph.GraphBuilder.createChart(GraphBuilder.java:75)
	at org.adempiere.webui.apps.graph.WGraph.render(WGraph.java:215)
	at org.adempiere.webui.apps.graph.WGraph.<init>(WGraph.java:153)
	at org.adempiere.webui.desktop.DefaultDesktop.createHomeTab(DefaultDesktop.java:386)
	at org.adempiere.webui.desktop.DefaultDesktop.doCreatePart(DefaultDesktop.java:165)
	at org.adempiere.webui.part.AbstractUIPart.createPart(AbstractUIPart.java:41)
	at org.adempiere.webui.AdempiereWebUI.loginCompleted(AdempiereWebUI.java:266)
	at org.adempiere.webui.AdempiereWebUI.onCreate(AdempiereWebUI.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:622)
	at org.zkoss.zk.ui.impl.EventProcessor.process0(EventProcessor.java:209)
	at org.zkoss.zk.ui.impl.EventProcessor.process(EventProcessor.java:141)
	at org.zkoss.zk.ui.impl.EventProcessingThreadImpl.process0(EventProcessingThreadImpl.java:519)
	at org.zkoss.zk.ui.impl.EventProcessingThreadImpl.run(EventProcessingThreadImpl.java:446)

Strangely, the app mostly works.  It is only plugin stuff that no longer loads in the updated java.  Hence my admittedly unscientific hunch that a class loader somewhere is broken.  It is clearly a problem with the update, however, since the whole point of maintaining 1.6 is compatibility with legacy apps.  I am working on migrating this app to a newer version than runs on java 1.8, but it will be a year or so at least.

Let me try to manually find this class and see what bytecode version it is.

Apparently, there is a 1:1.6.0.39-1.13.11.1.el6_8 in CR.  Is it worth me trying to get that and see if it has the same problem?

java-1.6.0-openjdk.i686 1:1.6.0.39-1.13.11.1.el6_8 from CR is also broken.

Here is a stack trace:
>>java.lang.Error: can't load javax.swing.plaf.metal.MetalLookAndFeel
>>      at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1326)
>>      at javax.swing.UIManager.initialize(UIManager.java:1407)
>>      at javax.swing.UIManager.maybeInitialize(UIManager.java:1395)
>>      at javax.swing.UIManager.getDefaults(UIManager.java:644)
>>      at javax.swing.UIManager.getColor(UIManager.java:686)
>>      at org.jfree.chart.JFreeChart.<clinit>(JFreeChart.java:261)
>>      at org.jfree.chart.ChartFactory.createLineChart3D(ChartFactory.java:1296)
>>      at org.adempiere.apps.graph.GraphBuilder.createLineChart(GraphBuilder.java:170)
>>      at org.adempiere.apps.graph.GraphBuilder.createChart(GraphBuilder.java:75)
>>      at org.adempiere.webui.apps.graph.WGraph.render(WGraph.java:215)
>>      at org.adempiere.webui.apps.graph.WGraph.<init>(WGraph.java:153)
>>...

Weird.  MetalLookAndFeel is clearly present, right in rt.jar.  Well, I have to get some work done, so putting back the working version for now.  Will look at it some more tomorrow.

The two builds are largely the same. The 6.8 version has the additional fixes for bug 1217131 (disable RC4) and bug 1195203 (add mime.types symlink) which were added as part of the RHEL 6.8 release, but shouldn't make any difference to this.

Looking at the code, that MetalLookAndFeel error is swallowing the actual exception that's being thrown. Some basic testing of the method:

import javax.swing.UIManager;
import java.util.Arrays;

public class Test
{
  public static void main(String[] args)
  {
    System.err.println(Arrays.toString(UIManager.getInstalledLookAndFeels()));
  }
}

works fine, so I do suspect it's something to do with the more complex class loading setup in that context. 

Assuming the above short fragment also works fine for you, can you reproduce the Metal LnF failure easily? If so, we could try with a JDK build that doesn't swallow the underlying exception so we can see what's really going on there.

Created attachment 1159207 [details]
hs_err_pid24513.log

Well, I did something else - I upgraded java on our backup server which has the same java software and database, but is 64-bit.  Again, it fails with the upgrade, but works again after downgrading.  BUT, this time we get a java diagnostic dump!  Maybe that will be a clue.

Well, this isn't my code, but it looks like JFreeChart brings up swing with a virtual screen to draw the graph, and that is what is really broken with the update (at least on x86_64 - but I suspect i686 just presents differently).

I think this may well be a latent bug awakened by the updates between 6.12 and 6.13 rather than anything to do with the actual updates themselves. As Andrew Hughes pointed out they are relatively innocuous.

I am suspicious that it relates to a bug found in JDK7 some while back by James Livingstone where a NPE happened because of a bad argument in a native call whose effect was somewhat arbitrary. I'll look further into it but I think it would be worth checking if that patch was needed in but has not gone into JDK6. The relevant bugzilla is 
https://bugzilla.redhat.com/show_bug.cgi?id=1176718

(In reply to Andrew Dinn from comment #12)
> I think this may well be a latent bug awakened by the updates between 6.12
> and 6.13 rather than anything to do with the actual updates themselves. As
> Andrew Hughes pointed out they are relatively innocuous.
> 

I was referring to the difference between 1.13.11 in RHEL 6.7 and RHEL 6.8 rather than the differences between 1.13.10 and 1.13.11.

OpenJDK 6 is an odd case in that, unlike OpenJDK 7 & 8, it was never maintained by Sun/Oracle or used as a basis for their proprietary releases. It was forked from early OpenJDK 7 and then fixed to be compatible with 6 as best as possible. This means that OpenJDK 6 & the proprietary 6 are substantially different beasts, whereas OpenJDK 7 & the proprietary 7 should be much closer.

We encounter this when we come to backporting security patches because it's often the case that there are dependent fixes that went in the proprietary JDK 6 and then were eventually forward-ported to OpenJDK 7, but never made it to OpenJDK 6. We don't have access to what was in the proprietary 6 obviously, so we have to work with backporting things from 7 as best as possible.

When stuck between the choices of backporting something from 7 that a security patch depends on, and trying to alter the security fix to fit the existing 6, I tend to go with the former as the safer option. Neither is ideal, but it's easier to diagnose any compatibility issues introduced by the backport than it is to risk a security fix not doing what it says on the tin, when we have basically no information about them.

> I am suspicious that it relates to a bug found in JDK7 some while back by
> James Livingstone where a NPE happened because of a bad argument in a native
> call whose effect was somewhat arbitrary. I'll look further into it but I
> think it would be worth checking if that patch was needed in but has not
> gone into JDK6. The relevant bugzilla is 
> https://bugzilla.redhat.com/show_bug.cgi?id=1176718

This was fixed in 1.13.8: http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=2510

I was wondering if this was related to bug 1334465 which is why I was trying to get more of a stack trace to see what's really going on. Judging by the HotSpot crash we have now, I think the class loading exceptions may be a misnomer.

Stuart, it looks likely they are both the same issue, it's just that, on x86, it doesn't crash the VM, so failure is reported back up to the Swing code which throws an exception.

Did you manage to try the fragment in comment #9 on your setup? These font bugs are usually related to the exact set of installed fonts on the system, so the code may succeed here, but fail on your installation.

Created attachment 1160245 [details]
Crash from test snippet

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f0a89647aa1, pid=25438, tid=139683560376064
#
# JRE version: 6.0_39-b39
# Java VM: OpenJDK 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.13.11
# Distribution: CentOS release 6.7 (Final), package rhel-1.13.11.0.el6_7-x86_64
# Problematic frame:
# C  [libfontconfig.so.1+0xaaa1]  FcConfigUptoDate+0x191
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/stuart/hs_err_pid25438.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted

The X86_64 version does a much better job of actually crashing, so I'll keep running tests there.

fonctconfig version installed:

fontconfig-2.8.0-5.el6.x86_64

Also broken on el7 (CentOS7):

[root@c7dev stuart]# /java6/bin/java Test
Exception in thread "main" java.lang.Error: can't load javax.swing.plaf.metal.MetalLookAndFeel
	at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1326)
	at javax.swing.UIManager.initialize(UIManager.java:1407)
	at javax.swing.UIManager.maybeInitialize(UIManager.java:1395)
	at javax.swing.UIManager.getInstalledLookAndFeels(UIManager.java:410)
	at Test.main(Test.java:8)

Using java-1.7 instead:

[root@c7dev stuart]# java Test
[javax.swing.UIManager$LookAndFeelInfo[Metal javax.swing.plaf.metal.MetalLookAndFeel], javax.swing.UIManager$LookAndFeelInfo[Nimbus javax.swing.plaf.nimbus.NimbusLookAndFeel], javax.swing.UIManager$LookAndFeelInfo[CDE/Motif com.sun.java.swing.plaf.motif.MotifLookAndFeel], javax.swing.UIManager$LookAndFeelInfo[GTK+ com.sun.java.swing.plaf.gtk.GTKLookAndFeel]]

And, also checked that previous version does work on CentOS-7:

[stuart@c7dev ~]$ /java6/bin/java Test
[javax.swing.UIManager$LookAndFeelInfo[Metal javax.swing.plaf.metal.MetalLookAndFeel], javax.swing.UIManager$LookAndFeelInfo[Nimbus com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel], javax.swing.UIManager$LookAndFeelInfo[CDE/Motif com.sun.java.swing.plaf.motif.MotifLookAndFeel], javax.swing.UIManager$LookAndFeelInfo[GTK+ com.sun.java.swing.plaf.gtk.GTKLookAndFeel]]
[stuart@c7dev ~]$ rpm -q java-1.6.0-openjdk
java-1.6.0-openjdk-1.6.0.38-1.13.10.0.el7_2.x86_64

Should I open another bugzilla for EL7?

Maybe, but let's try and diagnose the actual issue on EL6 yet as it's likely to be the same or similar (the same version of OpenJDK 6 & IcedTea is used on both).

By the sound of it, we now have a simple reproducer for the failure on both EL6 & 7. Sadly, I tried it on both and it worked, with the same version on fontconfig on EL6.

Could you try running the test on EL6 with debug output turned on e.g.

$ /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -Dsun.java2d.debugfonts=true Test

Created attachment 1164788 [details]
thread dump from crash with font debug

$ java -Dsun.java2d.debugfonts=true Test
Jun 4, 2016 11:09:35 AM sun.awt.X11GraphicsEnvironment registerFontDir
INFO: ParseFontDir /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/lib/fonts
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2e18d9eaa1, pid=5022, tid=139836186896128
#
# JRE version: 6.0_39-b39
# Java VM: OpenJDK 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.13.11
# Distribution: CentOS release 6.7 (Final), package rhel-1.13.11.0.el6_7-x86_64
# Problematic frame:
# C  [libfontconfig.so.1+0xaaa1]  FcConfigUptoDate+0x191
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/stuart/hs_err_pid5022.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted

The /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/lib/fonts directory does not exist.  I created an empty directory there (with matching selinux attributes), but that doesn't change the error.  I also tried symlinking to /usr/share/fonts, no change with that either.

issue : I encountered the same kind problem and it occurred only in my QA servers but it worked fine in DEV & Prod. I started comparing all JAVA version, JRE libs ,setup and everything for two days. finally I found  some difference in QA  like some of the (font related)packages were not installed in QA but those were installed in  DEV & Prod.

Command to detect available font packages which I used : rpm -qa | grep font | sort.
output.
fontconfig-2.8.0-5.el6.i686
fontconfig-2.8.0-5.el6.x86_64
fontconfig-devel-2.8.0-5.el6.x86_64
fontpackages-filesystem-1.41-1.1.el6.noarch
ghostscript-fonts-5.50-23.2.el6.noarch
liberation-fonts-common-1.05.1.20090721-5.el6.noarch
liberation-sans-fonts-1.05.1.20090721-5.el6.noarch
libfontenc-1.1.2-3.el6.x86_64
libfontenc-devel-1.1.2-3.el6.x86_64
libXfont-1.5.1-2.el6.x86_64
libXfont-devel-1.5.1-2.el6.x86_64
texlive-texmf-errata-fonts-2007-7.1.el6.noarch
texlive-texmf-fonts-2007-39.el6_7.noarch
urw-fonts-2.4-11.el6.noarch
xorg-x11-fonts-ISO8859-1-75dpi-7.2-11.el6.noarch
xorg-x11-font-utils-7.2-11.el6.x86_64

FIX ( installed the following missed font packages solved the problem
chkfontpath-1.10.1-1.x86_64
xorg-x11-fonts-Type1-7.2-11.el6.noarch
msttcorefonts-2.5-1.noarch

For your case make sure all the above fonts packages installed.

One more verification after I installed the packages, it will create 

it will create a  directory & a file in the location /etc/X11
 fs- directory
 config- file.

Reference Link http://www.city-fan.org/tips/SunJava5OnFedora

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000364100aaa1, pid=8308, tid=139884383495936
#
# JRE version: 6.0_39-b39
# Java VM: OpenJDK 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.13.11
# Distribution: CentOS release 6.7 (Final), package rhel-1.13.11.1.el6_8-x86_64
# Problematic frame:
# C  [libfontconfig.so.1+0xaaa1]  chmod+0xaaa1
#
# Core dump written. Default location: //core or core.8308
#
# An error report file with more information is saved as:
# /tmp/java_error_8308.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future
Listening for transport dt_socket at address: 8016
Jul 20, 2016 5:10:30 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/../lib/amd64::/opt/dialogic/IMG/GateControlEMS/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
                                                                                                                  

C  [libfontconfig.so.1+0xaaa1]  chmod+0xaaa1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sun.font.FontManager.getFontConfig(Ljava/lang/String;Lsun/font/FontManager$FontConfigInfo;[Lsun/font/FontManager$FcCompFont;Z)V+0
j  sun.font.FontManager.initFontConfigFonts(Z)V+165

.............

Fix -installed the following missed font packages solved the problem

chkfontpath-1.10.1-1.x86_64
xorg-x11-fonts-Type1-7.2-11.el6.noarch
msttcorefonts-2.5-1.noarch

For your case make sure all the above fonts packages installed.

One more verification after I installed the packages, it will create 

it will create a  directory & a file in the location /etc/X11
 fs- directory
 config- file.

check my previous comment 324 which has detail information.

That explains why I couldn't replicate it. Thanks.

Maybe a simple solution is to add those font packages as dependencies of the java-1.6.0-openjdk package.

I already had the xorg fonts installed.  chkfontpath and msttcorefonts are not available in CentOS6.  Are they in RHEL6?

I believe the RPMs used to require more fonts and we got complaints. It's difficult with OpenJDK 6 because it doesn't have a headless package, so any font requirement would be imposed on every JDK users, including those who don't make use of any graphical functionality.

This installation is headless also - fonts are needed for printing, not just for awt/swing.

A README listing the fonts needed for java printing and/or gui would be just fine for an ancient package like this one.  A subpackage with no content, but dependencies on the necessary fonts would also be good.

However, I don't think that is the problem.  I did not uninstall any fonts.  This starts happening with .39 with everything else staying the same.  Go back to .38, works.  Upgrade to .39, broken again.

There were changes into how fonts are configured in b39, that's why.

java-1.6.0-openjdk-1.6.0.40-1.13.12.6.el6_8 is now available. Can you see if this update is any better? It includes the fix for bug 1334465.

That does indeed fix the problem on x86_64 test system.  Awesome!  I'm about to test on 32-bit, then continue rolling it out.  (How did we manage without virtual machines?)  I'll let you know if there are any problems.

And 32-bit testing is also good.

Closing w.r.t. comment #33 and #34.