Few details: as we can see, we start JVM with lot of extra flags, some could be interesting here:

1) We run tests with code coverage provided by jacoco agent: javaagent:/opt/hp93000rt/el7/x86_64/jacoco-0.7.x/0.8.5/lib/jacocoagent.jar. I assume that one transforms classes on load, but I'm not sure if the class in question was transformed or not.
2) We have pretty big workstation here (12 core Xeons with 64 GB RAM) and so high heap size and increased stack size in the JVM: -Xmx24G -Xms1G -Xss5m 
3) We don't use default G1 GC because for our workload the Parallel is faster: -XX:+UseParallelGC
4) Application in question is Eclipse IDE with lot of parallel tasks
5) The JVM was built by us from RH provided 11.0.7 SDK source rpm (https://access.redhat.com/downloads/content/java-11-openjdk-src/11.0.7.10-4.el7_8/x86_64/fd431d51/package), with only one patch from bug 1751985 comment 81 applied on top.

What would be really helpful for us to know: 

1) If this is a regression in 11.0.7 compared to 11.0.5, or of the same problem (looking on the trace) is most likely also present in 11.0.5?
2) Is there a "safe" workaround (something like -XX:-UseLoopPredicate or excluding code in question from compilation)?

Created attachment 1682817 [details]
Another crash dump with PhaseIdealLoop::build_loop_late_post(Node*)

Unfortunately we saw another crash in same code but now in a different test, with similar crash dump, see attachment.

I fear either we see a regression between 11.0.5.10-0.el7_7 build we've used before and 11.0.7.10-4.el7 we have now, or the build 11.0.7.10-4.el7 we've created on RHEL 7.4 is somehow broken (wrong gcc version used etc).

We see the same crash on RHEL 11.0.5, with the patch from https://bugzilla.redhat.com/show_bug.cgi?id=1751985#c81. So its highly likely building OpenJDK 11.0.5+ on RHEL 7.4 doesn't work properly.

We'll try to build the JDK without the patch and see if the crash persists. Also we recently changed our GC strategy to parallel GC, so maybe this is also a problem (seems unlikely but who knows). We'll run tests also with the OpenJDK 11.0.5 RPM from RHEL, to be sure the parallel GC strategy is not the problem.

FWIW, from 4 crashes we've seen so far, 2 were crashed trying to optimize this code below and crash stack was almost same:

org.eclipse.dltk.core.PriorityDLTKExtensionManager.findScriptNature(IProject)

public String findScriptNature(IProject project) {
	try {
		if (!project.isAccessible()) {
			return null;
		}
		String[] natureIds = project.getDescription().getNatureIds();
		for (int i = 0; i < natureIds.length; i++) {
			String natureId = natureIds[i];

			if (getElementInfo(natureId) != null) {
				return natureId;
			}
		}
	} catch (CoreException e) {
		return null;
	}

	return null;
}

Crash stack 1:

Current CompileTask:
C2: 151818 41725   !   4       org.eclipse.dltk.core.PriorityDLTKExtensionManager::findScriptNature (120 bytes)

Stack: [0x00007fff955e4000,0x00007fff956e5000],  sp=0x00007fff956e0610,  free space=1009k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xac3319]  PhaseIdealLoop::build_loop_late_post(Node*)+0x159
V  [libjvm.so+0xac37eb]  PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x13b
V  [libjvm.so+0xac6af4]  PhaseIdealLoop::build_and_optimize(LoopOptsMode)+0x844
V  [libjvm.so+0x67b440]  Compile::Optimize()+0x780
V  [libjvm.so+0x67d1dc]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0x10ec
V  [libjvm.so+0x5a17d9]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x109
V  [libjvm.so+0x686005]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x485
V  [libjvm.so+0x687868]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xeafc12]  JavaThread::thread_main_inner()+0x232
V  [libjvm.so+0xeac30a]  Thread::call_run()+0x16a
V  [libjvm.so+0xc216a8]  thread_native_entry(Thread*)+0xf8


Crash stack 2:

Current CompileTask:
C2: 537162 50342   !   4       org.eclipse.dltk.core.PriorityDLTKExtensionManager::findScriptNature (120 bytes)

Stack: [0x00007fff94a11000,0x00007fff94b12000],  sp=0x00007fff94b0d510,  free space=1009k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xabdca1]  PhaseIdealLoop::build_loop_late_post(Node*, bool)+0x151
V  [libjvm.so+0xabe13e]  PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&, bool)+0x13e
V  [libjvm.so+0xac1426]  PhaseIdealLoop::build_and_optimize(LoopOptsMode)+0x826
V  [libjvm.so+0x678257]  Compile::Optimize()+0x797
V  [libjvm.so+0x679eec]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0xffc
V  [libjvm.so+0x59f15e]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0xfe
V  [libjvm.so+0x682985]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x485
V  [libjvm.so+0x6841e8]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xe94822]  JavaThread::thread_main_inner()+0x232
V  [libjvm.so+0xe94b68]  JavaThread::run()+0x308
V  [libjvm.so+0xe90f65]  Thread::call_run()+0x155
V  [libjvm.so+0xc1a7f8]  thread_native_entry(Thread*)+0xf8

Created attachment 1683221 [details]
Crash with 11.0.5 (patched, build locally)

The JVM should have dumped replay_pid*.log files where you found the hs_err_pid*.log files. Can you check whether you have those files and attach them to the bug report if you do?

Here is the code that was optimized from the first crash (from different Eclipse plug-in), interestingly it also uses IProject.getDescription() API that throws CoreException and in both cases the loop code is inside the try/catch block.

public static boolean hasBuilder(IProject project) {
	Preconditions.checkNotNull(project);
	if (project.isAccessible()) {
		try {
			for (ICommand command : project.getDescription().getBuildSpec()) {
				if (BUILDER_ID.equals(command.getBuilderName())) {
					return true;
				}
			}
		} catch (CoreException e) {
			log.error("Can't build due to an exception.", e);
		}
	}
	return false;
}

We see the crash with 11.0.5 unpatched, when built on RHEL 7.4. So I imagine this can be closed as a user error. Andrey?

Created attachment 1683590 [details]
Replay log file from JDK crash.

Created attachment 1683591 [details]
Crash with OpenJDK 11.0.5 built on RHEL 7.4 and parallel GC strategy.

(In reply to Simeon Andreev from comment #11)
> We see the crash with 11.0.5 unpatched, when built on RHEL 7.4. So I imagine
> this can be closed as a user error. Andrey?

Nope. I've now run same test with "official" 11.0.5 build, same crash at same place.

@Simeon: please change bug title to 
"OpenJDK 11.0.5 crash in PhaseIdealLoop::build_loop_late_post(Node*)+0x159 with XX:+UseParallelGC"

I assume the crash is specific on workstations with large amount of *physical* cores and memory, where parallel GC threads probably interfere with C2 compiler threads.

We don't see the crash in our CI environment that uses KVM virtual machines with ~8 virtual cores and 24 GB RAM, where JVM uses 12 GB max heap.
We see the crash sporadically on Xeon E5-1650 workstations with 12 real cores and 64+ GB RAM, where JVM uses 24+ GB max heap.

This is now really pity, because we've enabled UseParallelGC on Java 11 because G1 caused significant performance degradation compared to Java 8 (10%) for our product in specific task types (batch compilation). So now we have a choice - either Java 11 is slow it is crashing :-(.

Created attachment 1684037 [details]
Crash with "official" OpenJDK 11.0.5 build and parallel GC

Created attachment 1684544 [details]
Another crash with official 11.0.5, replay file coming next

Created attachment 1684545 [details]
replay file for previous crash log

Thanks for the replay files. With a replay file, it's possible to re-run the the failed compilation and it's often the easiest way to reproduce a bug that occurs in JIT compilation. For that I need all classes that are referred by the compilation process (all classes mentioned in the replay file). Maybe the easiest way for me to get that would be to run the failing test(s) locally. Is it doable? If yes, how would I do that?

(In reply to Roland Westrelin from comment #19)
> Thanks for the replay files. With a replay file, it's possible to re-run the
> the failed compilation and it's often the easiest way to reproduce a bug
> that occurs in JIT compilation. For that I need all classes that are
> referred by the compilation process (all classes mentioned in the replay
> file). 

We will try to collect & attach them. Is this a problem that we've used jacoco agent - the classes itself as coming from eclipse.org will be *not* transformed by jacoco?
Or should we try to dump transformed classes from a running VM? Or should we try to reproduce without jacoco?

Our problem so far is, that we only see it crashing inside our ~10 hours long test suite (that uses jacoco to get coverage data), but not inside a single test started by hand. We will try to work on something isolated. I guess increasing number of parallel GC threads should do the trick.

> Maybe the easiest way for me to get that would be to run the failing
> test(s) locally. Is it doable? If yes, how would I do that?

Not at all, unfortunately. If you would have access to Advantest hardware, it would be possible, but I guess you aren't testing semiconductors :-)

(In reply to Andrey Loskutov from comment #20)

> We will try to collect & attach them. Is this a problem that we've used
> jacoco agent - the classes itself as coming from eclipse.org will be *not*
> transformed by jacoco?

jacoco could be a problem. Replaying compilations only works with classes that are on disk in a jar or class file. It doesn't work for transformed or generated classes.
I think it's worth trying replaying the compilation first, anyway. I see in the replay files that a lot of classes are from eclipse. If it reproduces that way, fixing it will be mostly straightforward.
FYI, if you want to give it a try yourself:

java -XX:+ReplayCompiles -XX:ReplayDataFile=replay_pidxxx.log -XX:+ReplayIgnoreInitErrors

is the way to rerun the failed compilation. You have to pass it the exact same command line options that were passed to the JVM that crashes (-XX:+UseParallelGC etc.) and a classpath that includes all classes that are mentioned in the replay file.

> Or should we try to dump transformed classes from a running VM? Or should we
> try to reproduce without jacoco?

Dumping classes wouldn't help AFAICT unless we could save them to .class files. 

> Our problem so far is, that we only see it crashing inside our ~10 hours
> long test suite (that uses jacoco to get coverage data), but not inside a
> single test started by hand. We will try to work on something isolated. I
> guess increasing number of parallel GC threads should do the trick.

Ok. Thanks.

Is there some code you would like added to the JDK for extra logging? Reproducing on a smaller scale won't be easy on first glance and maybe more logging in JDK would help identify the issue.

We'll try to reproduce without jacoco, if we can we'll see about attaching the .class files from the replay log.

Created attachment 1684876 [details]
dump extra information to help diagnose the issue

(In reply to Simeon Andreev from comment #22)
> Is there some code you would like added to the JDK for extra logging?
> Reproducing on a smaller scale won't be easy on first glance and maybe more
> logging in JDK would help identify the issue.

I attached a patch that should provide extra details when the crash happens (appended to the hs_err file). Can you give a try?

> We'll try to reproduce without jacoco, if we can we'll see about attaching
> the .class files from the replay log.

I think if it makes sense to give replay a try even if jacoco is enabled.

(In reply to Roland Westrelin from comment #23)
> Created attachment 1684876 [details]
> dump extra information to help diagnose the issue

When I try to build OpenJDK 11.0.7 with the patch, there is a compile error:


=== Output from failing command(s) repeated here ===
make/Init.gmk:339: Building on-failure 
/usr/bin/printf "* For target hotspot_variant-server_libjvm_objs_vmError.o:\n" 
+ /usr/bin/printf '* For target hotspot_variant-server_libjvm_objs_vmError.o:\n'
* For target hotspot_variant-server_libjvm_objs_vmError.o:
make/Init.gmk:339: Building on-failure 
(/usr/bin/grep -v -e "^Note: including file:" <  /data/rpmbuild/BUILD/java-11-openjdk-11.0.7.10-4.el7.x86_64/openjdk/build/make-support/failure-logs/hotspot_variant-server_libjvm_objs_vmError.o.log || true) | /usr/bin/head -n 15 
+ /usr/bin/grep -v -e '^Note: including file:'
+ /usr/bin/head -n 15
./src/hotspot/share/utilities/vmError.cpp: In static member function 'static void VMError::report(outputStream*, bool)':
./src/hotspot/share/utilities/vmError.cpp:1031:57: error: 'class Node' has no member named 'dump'
           st->print("XXX"); C->_build_loop_late_post_n->dump(st);
                                                         ^
./src/hotspot/share/utilities/vmError.cpp:1032:22: error: 'class RootNode' has no member named 'dump'
           C->root()->dump(999, st);


We got the source RPM from here: https://access.cdn.redhat.com/content/origin/rpms/java-11-openjdk/11.0.7.10/4.el7_8/fd431d51/java-11-openjdk-11.0.7.10-4.el7_8.src.rpm?user=a380cb0addc6df66d00ea8567e0d5e4e&_auth_=1587566542_4d702dd872d223bca72d7f22616fe06a

Created attachment 1685110 [details]
dump extra information to help diagnose the issue

(In reply to Simeon Andreev from comment #25)
> (In reply to Roland Westrelin from comment #23)
> > Created attachment 1684876 [details]
> > dump extra information to help diagnose the issue
> 
> When I try to build OpenJDK 11.0.7 with the patch, there is a compile error:
> 
> 
> === Output from failing command(s) repeated here ===
> make/Init.gmk:339: Building on-failure 
> /usr/bin/printf "* For target
> hotspot_variant-server_libjvm_objs_vmError.o:\n" 
> + /usr/bin/printf '* For target
> hotspot_variant-server_libjvm_objs_vmError.o:\n'
> * For target hotspot_variant-server_libjvm_objs_vmError.o:
> make/Init.gmk:339: Building on-failure 
> (/usr/bin/grep -v -e "^Note: including file:" < 
> /data/rpmbuild/BUILD/java-11-openjdk-11.0.7.10-4.el7.x86_64/openjdk/build/
> make-support/failure-logs/hotspot_variant-server_libjvm_objs_vmError.o.log
> || true) | /usr/bin/head -n 15 
> + /usr/bin/grep -v -e '^Note: including file:'
> + /usr/bin/head -n 15
> ./src/hotspot/share/utilities/vmError.cpp: In static member function 'static
> void VMError::report(outputStream*, bool)':
> ./src/hotspot/share/utilities/vmError.cpp:1031:57: error: 'class Node' has
> no member named 'dump'
>            st->print("XXX"); C->_build_loop_late_post_n->dump(st);
>                                                          ^
> ./src/hotspot/share/utilities/vmError.cpp:1032:22: error: 'class RootNode'
> has no member named 'dump'
>            C->root()->dump(999, st);
> 
> 
> We got the source RPM from here:
> https://access.cdn.redhat.com/content/origin/rpms/java-11-openjdk/11.0.7.10/
> 4.el7_8/fd431d51/java-11-openjdk-11.0.7.10-4.el7_8.src.
> rpm?user=a380cb0addc6df66d00ea8567e0d5e4e&_auth_=1587566542_4d702dd872d223bca
> 72d7f22616fe06a

I attached an updated patch.
Actually, backtracking a bit, the first thing for you to try would be to run with a fastdebug build (for some reason I thought it was already the case).
Do you know how to produce one?

(In reply to Roland Westrelin from comment #27)
> Actually, backtracking a bit, the first thing for you to try would be to run
> with a fastdebug build (for some reason I thought it was already the case).
> Do you know how to produce one?

Only with the OpenJDK build from mercurial. How to do this from a source RPM?

The header of the spec file only mentions release and slow debug, not fast debug:

# RPM conditionals so as to be able to dynamically produce
# slowdebug/release builds. See:
# http://rpm.org/user_doc/conditional_builds.html
#
# Examples:
#
# Produce release *and* slowdebug builds on x86_64 (default):
# $ rpmbuild -ba java-1.8.0-openjdk.spec
#
# Produce only release builds (no slowdebug builds) on x86_64:
# $ rpmbuild -ba java-1.8.0-openjdk.spec --without slowdebug
#
# Only produce a release build on x86_64:
# $ fedpkg mockbuild --without slowdebug
#
# Only produce a debug build on x86_64:
# $ fedpkg local --without release

I've changed the debug level specified in the .spec file to hard-code fastdebug, and will call the build with "--without slowdebug". Would this be enough?

Unfortunately, the fastdebug build I did results in:

[9.993s][warning][malloc,free] ## nof_mallocs = 1247804, nof_frees = 565839
[9.993s][warning][malloc,free] ## memory stomp:
[9.993s][warning][malloc,free] GuardedMemory(0x00007ffff45fb1a0) base_addr=0x00007ffff2f57620 tag=0x0000000000000000 user_size=1 user_data=0x00007ffff2f57640
[9.993s][warning][malloc,free]   Header guard @0x00007ffff2f57620 is OK
[9.993s][warning][malloc,free]   Trailer guard @0x00007ffff2f57641 is BROKEN
[9.993s][warning][malloc,free]   User data appears to be in use
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/os.cpp:638
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/data/rpmbuild/BUILD/java-11-openjdk-11.0.7.10-5.el7.x86_64/openjdk/src/hotspot/share/runtime/os.cpp:638), pid=18203, tid=18209
#  fatal error: memory stomping error
#
# JRE version: OpenJDK Runtime Environment (11.0.7+10) (fastdebug build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 11.0.7+10-LTS, mixed mode, sharing, tiered, compressed oops, parallel gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /workspaces/socbm775/sandreev/ws-java11/opt/93000/src/segments/eclipse/workspace/com.advantest.itee.ippanel.core.tests/hs_err_pid18203.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

I guess its not as simple as changing this in the spec:

    --with-debug-level=$debugbuild \

To:

    --with-debug-level=fastdebug \

Created attachment 1685171 [details]
spec file patch to produce fastdebug build over slowdebug

(In reply to Simeon Andreev from comment #29)
> I've changed the debug level specified in the .spec file to hard-code
> fastdebug, and will call the build with "--without slowdebug". Would this be
> enough?

Simeon, I've attached a patch for the java-11-openjdk.spec in comment 31 which should produce a fastdebug build (over slowdebug) for the -debug subpackages once you rebuild with something like:

$ rpmbuild -ba java-11-openjdk.spec

or just the fastdebug build:

$ rpmbuild -ba java-11-openjdk.spec --without release

Verify you've got a fastdebug build:

$ rpm -qa | grep java-11-openjdk
java-11-openjdk-devel-debug-11.0.7.10-4.el7_8.x86_64
java-11-openjdk-headless-debug-11.0.7.10-4.el7_8.x86_64
java-11-openjdk-debug-11.0.7.10-4.el7_8.x86_64

$ /usr/lib/jvm/java-11-openjdk/bin/java -version
openjdk version "11.0.7" 2020-04-14 LTS
OpenJDK Runtime Environment 18.9 (fastdebug build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (fastdebug build 11.0.7+10-LTS, mixed mode, sharing)

Note the "fastdebug" in the -version output.

socbm775:/data/rpmbuild$ rpmbuild -ba SPECS/java-11-openjdk.spec --without release        
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.PjkaiD
+ umask 022
+ cd /data/rpmbuild//BUILD
+ '[' 0 -eq 0 -o 0 -eq 1 ']'
+ echo 'include_normal_build is 0'
include_normal_build is 0
+ '[' 1 -eq 0 -o 1 -eq 1 ']'
+ echo 'include_debug_build is 1'
include_debug_build is 1
+ '[' 1 -eq 0 -a 0 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ echo 'You have disabled the normal build, but this is required to provide docs for the debug build.'
You have disabled the normal build, but this is required to provide docs for the debug build.
+ exit 14
error: Bad exit status from /var/tmp/rpm-tmp.PjkaiD (%prep)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.PjkaiD (%prep)

I've built release and fastdebug, this goes through. But how do I install only the fastdebug? Do I need to manually remove all OpenJDK RPMs I have and only install the debug ones?

socbm775:/home/sandreev$ rpm -qa | grep java-11-openjdk
java-11-openjdk-javadoc-11.0.7.10-4.el7.x86_64
java-11-openjdk-jmods-debug-11.0.7.10-4.el7.x86_64
java-11-openjdk-devel-debug-11.0.7.10-4.el7.x86_64
java-11-openjdk-11.0.7.10-4.el7.x86_64
java-11-openjdk-src-11.0.7.10-4.el7.x86_64
java-11-openjdk-devel-11.0.7.10-4.el7.x86_64
java-11-openjdk-src-debug-11.0.7.10-4.el7.x86_64
java-11-openjdk-headless-debug-11.0.7.10-4.el7.x86_64
java-11-openjdk-javadoc-zip-11.0.7.10-4.el7.x86_64
java-11-openjdk-headless-11.0.7.10-4.el7.x86_64
java-11-openjdk-jmods-11.0.7.10-4.el7.x86_64
java-11-openjdk-debug-11.0.7.10-4.el7.x86_64

(In reply to Simeon Andreev from comment #33)
> socbm775:/data/rpmbuild$ rpmbuild -ba SPECS/java-11-openjdk.spec --without
> release        
> Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.PjkaiD
> + umask 022
> + cd /data/rpmbuild//BUILD
> + '[' 0 -eq 0 -o 0 -eq 1 ']'
> + echo 'include_normal_build is 0'
> include_normal_build is 0
> + '[' 1 -eq 0 -o 1 -eq 1 ']'
> + echo 'include_debug_build is 1'
> include_debug_build is 1
> + '[' 1 -eq 0 -a 0 -eq 0 ']'
> + '[' 0 -eq 0 ']'
> + echo 'You have disabled the normal build, but this is required to provide
> docs for the debug build.'
> You have disabled the normal build, but this is required to provide docs for
> the debug build.
> + exit 14
> error: Bad exit status from /var/tmp/rpm-tmp.PjkaiD (%prep)
> 
> 
> RPM build errors:
>     Bad exit status from /var/tmp/rpm-tmp.PjkaiD (%prep)

Sorry about that. I didn't know about this (and yes, I didn't test this config). Sorry again.

> I've built release and fastdebug, this goes through. But how do I install
> only the fastdebug? Do I need to manually remove all OpenJDK RPMs I have and
> only install the debug ones?
> 
> socbm775:/home/sandreev$ rpm -qa | grep java-11-openjdk
> java-11-openjdk-javadoc-11.0.7.10-4.el7.x86_64
> java-11-openjdk-jmods-debug-11.0.7.10-4.el7.x86_64
> java-11-openjdk-devel-debug-11.0.7.10-4.el7.x86_64
> java-11-openjdk-11.0.7.10-4.el7.x86_64
> java-11-openjdk-src-11.0.7.10-4.el7.x86_64
> java-11-openjdk-devel-11.0.7.10-4.el7.x86_64
> java-11-openjdk-src-debug-11.0.7.10-4.el7.x86_64
> java-11-openjdk-headless-debug-11.0.7.10-4.el7.x86_64
> java-11-openjdk-javadoc-zip-11.0.7.10-4.el7.x86_64
> java-11-openjdk-headless-11.0.7.10-4.el7.x86_64
> java-11-openjdk-jmods-11.0.7.10-4.el7.x86_64
> java-11-openjdk-debug-11.0.7.10-4.el7.x86_64

Yes, remove any package not containing '-debug' in the name. In your case:

java-11-openjdk-javadoc
java-11-openjdk
java-11-openjdk-src
java-11-openjdk-devel
java-11-openjdk-javadoc-zip
java-11-openjdk-headless
java-11-openjdk-jmods

I'm still getting a crash when using the JDK, same error as before:

CRASH
6792
[exec] # fatal error: memory stomping error

(In reply to Simeon Andreev from comment #35)
> I'm still getting a crash when using the JDK, same error as before:
> 
> CRASH
> 6792
> [exec] # fatal error: memory stomping error

Are some patches applied to jdk for the build you're using?

Created attachment 1685557 [details]
Crash when specifying G1 99% time in non-GC.

(In reply to Roland Westrelin from comment #36)
> Are some patches applied to jdk for the build you're using?

The one you suggested, plus a performance patch for class unloading (I've mentioned this in comment 6). Also there are patches in the RHEL OpenJDK 11 source RPM.

I can run with the release build just fine, its the fast debug build that crashes (right on Eclipse start-up I think).

Anyway, we are trying to tune the G1 GC, since the parallel GC causes crashes in our case. We added VM option: -XX:GCTimeRatio=99

Now we see another crash:

Stack: [0x00007fffb70a4000,0x00007fffb71a4000],  sp=0x00007fffb71a28f8,  free space=1018k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libpthread.so.0+0xce90]  pthread_getcpuclockid+0x0
V  [libjvm.so+0xe8a751]  Thread::print_on(outputStream*, bool) const+0x51
V  [libjvm.so+0xe8cfc6]  JavaThread::print_on(outputStream*, bool) const+0xe6
V  [libjvm.so+0xe8fd18]  Threads::print_on(outputStream*, bool, bool, bool, bool)+0x668
V  [libjvm.so+0xf04cd0]  VM_Operation::evaluate()+0xe0
V  [libjvm.so+0xf0269f]  VMThread::evaluate_operation(VM_Operation*)+0x11f
V  [libjvm.so+0xf02af5]  VMThread::loop()+0x265
V  [libjvm.so+0xf0302c]  VMThread::run()+0x7c
V  [libjvm.so+0xe90fe5]  Thread::call_run()+0x155
V  [libjvm.so+0xc1a878]  thread_native_entry(Thread*)+0xf8

Note that in comment 37 we are using official 11.0.5-10 release from RHEL (albeit on RHEL 7.4). No patches, standard build.

Upon closer inspection, -XX:GCTimeRatio=99 was not used during the crash listed in comment 37.

I've reported bug 1832121. Andrey already ran into this problem with our 11.0.7 build but assumed it was due to the build itself. Now we observed the problem also with the 11.0.5 official build.

> The one you suggested, plus a performance patch for class unloading (I've
> mentioned this in comment 6). Also there are patches in the RHEL OpenJDK 11
> source RPM.

Can you try building and running fastdebug without extra patches (that is only the srpm patches)? Do you still see the memory stomping error?

Same error without any patches, when using the fastdebug build for 11.0.5 (built with the spec change from comment 32).

(In reply to Simeon Andreev from comment #42)
> Same error without any patches, when using the fastdebug build for 11.0.5
> (built with the spec change from comment 32).

Thanks for giving this a try. Do you think there's a chance I can reproduce the memory stomping error locally?
Would you be willing to give the replay file a try (that is collect class files for all classes mentioned in the replay file and share them with me)?

/usr/lib/jvm/java-11-openjdk-11.0.5.10-1.el7.x86_64-debug/bin/java -XX:+ReplayCompiles -XX:ReplayDataFile=/home/sandreev/replay_pid55147.log -XX:+ReplayIgnoreInitErrors -Dosgi.dataAreaRequiresExplicitInit=true -Xmx2048m --add-modules=ALL-SYSTEM -D93k.eclipseid=/93k_workspace -DXOC_SYSTEM=/workspaces/socbm775/sandreev/ws-java11/vobs/zenith/workspace/system -D93k.isOnline=false -Xmx24G -Xms1G -Xss5m -XX:ReservedCodeCacheSize=256m -XX:+UseParallelGC -DmaxCompiledUnitsAtOnce=0 -Dxtext.qn.interning=true -Dxtext.skipSecondaryTypes=true -Dosgi.bundlefile.limit=100 -Declipse.log.size.max=20000 -Declipse.log.backup.max=50 -Dorg.eclipse.swt.internal.gtk.noThemingFixes -Dorg.eclipse.core.resources.disable_advanced_recursive_link_checks -Dorg.eclipse.wst.validation.ui.disable.validation.context.menu=true -Dfile.encoding=UTF8 -Dorg.eclipse.tips.startup.default=dialog -Dlog4jloc=/workspaces/socbm775/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/logs/log4j/log4jlog.txt -javaagent:/opt/hp93000rt/el7/x86_64/jacoco-0.7.x/0.8.5/lib/jacocoagent.jar=destfile=/workspaces/socbm775/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/reports/coverage/jacoco.exec -XX:ErrorFile=/tmp/sandreev_ws-dev-andreev-2//eclipse_vm_crash_%p.log /workspaces/socbm775/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar -os linux -ws gtk -arch x86_64 -launcher /workspaces/socbm775/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest/eclipse -name Eclipse --launcher.library /workspaces/socbm775/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.1100.v20190907-0426/eclipse_1801.so -startup /workspaces/socbm775/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar --launcher.appendVmargs -exitdata 6bea806e -showLocation -product com.verigy.itee.ui.ewcProduct -application com.verigy.itee.testframework.ewc_test_framework -configuration /opt/hp93000/soc64_IworkspacesIsocbm275IsandreevIws-dev-andreev-2/segments/eclipse/eclipseForTest/configuration -data /tmp/sandreev_ws-dev-andreev-2/ -clean -vm /usr/lib/jvm/java-11/bin/java -vmargs -Dosgi.dataAreaRequiresExplicitInit=true -Xmx2048m --add-modules=ALL-SYSTEM -D93k.eclipseid=/93k_workspace -DXOC_SYSTEM=/workspaces/socbm775/sandreev/ws-java11/vobs/zenith/workspace/system -D93k.isOnline=false -Xmx24G -Xms1G -Xss5m -XX:ReservedCodeCacheSize=256m -XX:+UseParallelGC -DmaxCompiledUnitsAtOnce=0 -Dxtext.qn.interning=true -Dxtext.skipSecondaryTypes=true -Dosgi.bundlefile.limit=100 -Declipse.log.size.max=20000 -Declipse.log.backup.max=50 -Dorg.eclipse.swt.internal.gtk.noThemingFixes -Dorg.eclipse.core.resources.disable_advanced_recursive_link_checks -Dorg.eclipse.wst.validation.ui.disable.validation.context.menu=true -Dfile.encoding=UTF8 -Dorg.eclipse.tips.startup.default=dialog -Dlog4jloc=/workspaces/socbm775/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/logs/log4j/log4jlog.txt -javaagent:/opt/hp93000rt/el7/x86_64/jacoco-0.7.x/0.8.5/lib/jacocoagent.jar=destfile=/workspaces/socbm775/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/reports/coverage/jacoco.exec -XX:ErrorFile=/tmp/sandreev_ws-dev-andreev-2//eclipse_vm_crash_%p.log -jar /workspaces/socbm775/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.NullPointerException
        at org.jacoco.agent.rt.internal_43f5073.core.runtime.AgentOptions.<init>(AgentOptions.java:215)
        at org.jacoco.agent.rt.internal_43f5073.PreMain.premain(PreMain.java:48)
        ... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at  line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
Current thread is 56454
Dumping core ...
Aborted

I guess next stop will be attaching our class files. I'll discuss this with Andrey.

Maybe the fastdebug version is not being built correctly on our RHEL 7.4. Could you provide the build and we'll try it?

We'll collect the classes from the replay file and attach them today.

(In reply to Simeon Andreev from comment #45)
> Maybe the fastdebug version is not being built correctly on our RHEL 7.4.
> Could you provide the build and we'll try it?

Severin noticed he could reproduce a memory stomping error simply running eclipse with a fastdebug JVM. I ran it in a debugger and I see some eclipse native code corrupts memory. That could lead to any weird crash.

Most likely a build we would provide would make no difference.

Created attachment 1686239 [details]
classes listed in replay_pid55147.log

I hope the classes are in directories that reflect their packages. Classes present in the JDK are not included.

I've reported an Eclipse bug for the crash: https://bugs.eclipse.org/bugs/show_bug.cgi?id=562951

Looks like its caused by: https://bugs.eclipse.org/bugs/show_bug.cgi?id=549733

We'll try a workaround to not run into the bug and see how running our tests with the fastdebug build goes.

(In reply to Simeon Andreev from comment #48)
> I've reported an Eclipse bug for the crash:
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=562951

This bug is fixed now, new build with the fix is available: https://download.eclipse.org/eclipse/downloads/drops4/I20200509-0600/.

Thanks for the fixing the eclipse bug. I had no luck with the replay file (some classes don't seem to match the data encoded in the replay file).

We are running our tests with the fastdebug build, so far we didn't run into a crash. I'll keep running the test throughout the week.

(In reply to Simeon Andreev from comment #51)
> We are running our tests with the fastdebug build, so far we didn't run into
> a crash. I'll keep running the test throughout the week.

Ok. Thanks. Do you still see the crash with a JDK release build and the fixed eclipse build?

(In reply to Roland Westrelin from comment #52)
> Do you still see the crash with a JDK release build and the
> fixed eclipse build?

I'll need some time to get to this. Andrey will build the updated SDK soon, then I'll have to find workstation time to run our tests enough times.

So far (~1 week of running our tests) we've not seen the crash with the fastdebug build. I'll keep running our tests next week as well.

(In reply to Simeon Andreev from comment #54)
> So far (~1 week of running our tests) we've not seen the crash with the
> fastdebug build. I'll keep running our tests next week as well.

Thanks for the update. So:

- either the bug doesn't reproduce with a fastdebug JDK build for some reason

- or the eclipse memory corruption was the root cause (eclipse overwrote the JVM memory causing a crash). You haven't reproduced this with a JDK release build and the new eclipse build at this point, right?

- or some other change in the new eclipse build hides the problem. I suppose the new eclipse build has all upstream changes that happened between the previous build and the memory corruption fix, right?

Right now I'm testing with the Adwaita-dark theme, as the -dark prefix ensures the piece of native code that causes memory corruption does not run. I hope this is a minimal change that does not impact reproduction itself. I've not had the chance to run our tests with a fixed 4.15, but the build we'll use should have minimal changes on top of the current build we have.

My assumption is that the crash does not occur due to different timings with the debug build (as we've only observed the crash with parallel GC). It seems like a slim chance that a bit of corrupted memory would cause a crash, only with parallel GC, minutes later on (and after many tests). In particular in code that has crashed before (e.g. https://bugs.openjdk.java.net/browse/JDK-8181070). But we'll see; if we can't reproduce the crash with fixed https://bugs.eclipse.org/bugs/show_bug.cgi?id=562951, we can blame that bug I guess.

(In reply to Simeon Andreev from comment #56)
> It
> seems like a slim chance that a bit of corrupted memory would cause a crash,
> only with parallel GC, minutes later on (and after many tests).

Are we sure eclipse doesn't run the broken code multiple times in the course of the execution? That it's caught early by a fastdebug build doesn't mean it doesn't occur again once or multiple times as execution proceeds with a release build, right?

Anyway, I agree with everything you said in your comment.

(In reply to Roland Westrelin from comment #57) 
> Are we sure eclipse doesn't run the broken code multiple times in the course
> of the execution?

This is a very good point. In the same test plug-in with the crash, I found (surprisingly) creation of SWT Display objects (running the memory corrupting code). Normally an Eclipse application should have only 1 Display, so I assumed our tests are not different. I'll look into this.

We still see the same crash, with updated Eclipse 4.15 platform (that fixes the crash with the fastdebug build).

Stack: [0x00007fffb2ae2000,0x00007fffb2be3000],  sp=0x00007fffb2bde610,  free space=1009k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xac3319]  PhaseIdealLoop::build_loop_late_post(Node*)+0x159
V  [libjvm.so+0xac37eb]  PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x13b
V  [libjvm.so+0xac6af4]  PhaseIdealLoop::build_and_optimize(LoopOptsMode)+0x844
V  [libjvm.so+0x67b440]  Compile::Optimize()+0x780
V  [libjvm.so+0x67d1dc]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0x10ec
V  [libjvm.so+0x5a17d9]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x109
V  [libjvm.so+0x686005]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x485
V  [libjvm.so+0x687868]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xeafc12]  JavaThread::thread_main_inner()+0x232
V  [libjvm.so+0xeac30a]  Thread::call_run()+0x16a
V  [libjvm.so+0xc216a8]  thread_native_entry(Thread*)+0xf8

So I guess its something else (this is with parallel GC).

Since I can't reproduce this crash with the fastdebug build, maybe some patch can enable debug outputs for this specific crash only? Keeping the rest of the code in non-debug mode.

> Since I can't reproduce this crash with the fastdebug build, maybe some
> patch can enable debug outputs for this specific crash only? Keeping the
> rest of the code in non-debug mode.

Thanks for the update. I'm working on a patch to help diagnose the issue in a release build.

Created attachment 1690091 [details]
dump extra information to help diagnose the issue with a release build

Can you try the attached patch? It needs to be run with:

-XX:CompileCommand=option,org.eclipse.dltk.core.PriorityDLTKExtensionManager::findScriptNature,intx,IGVPrintLevel,2 -XX:+PrintIdealGraph -XX:PrintIdealGraphFile=graph.xml

This will cause graph*.xml files to be dumped. I'll need them as well as the hs_err file and recent process output.

Created attachment 1690514 [details]
Crash log with patch from comment 61.

Created attachment 1690516 [details]
graph.xml file with patch from comment 61.

Created attachment 1690519 [details]
Replay log file with patch from comment 61.

Created attachment 1690978 [details]
new patch for debugging

Thanks for the logs. I'd also need the last lines of output of the java process (there should be a line that starts with XXX and lists five integer: I need that one).

I attached a new patch that replaces the previous one. Could you run with that one and provide the logs + VM output?

From the last crash the last line is:

XXX 3827 5932 5932 267 801

I'll run tests with the updated patch.

Created attachment 1691459 [details]
Crash files with patch from comment 66.

Last output line of the JDK: XXX 4043 6085 6085 309 858

Unfortunately I see nothing in the graph*.log files? They all seem to be empty. I'll run more tests, hopefully we'll collect a non-empty graph*.xml file...

> Unfortunately I see nothing in the graph*.log files? They all seem to be
> empty. I'll run more tests, hopefully we'll collect a non-empty graph*.xml
> file...

Maybe the crash occured when compiling a different method this time (hs_err would tell)? The command line causes only the compilation of 
org.eclipse.dltk.core.PriorityDLTKExtensionManager::findScriptNature to be traced.

Created attachment 1691479 [details]
possible fix

I went over the log files from the previous crash again, and I think there's a good chance the attached patch fixes it.

Created attachment 1691694 [details]
Crash files with patch from comment 66.

Here is a crash with a non-empty graph.xml file. Last line from the JDK is:

XXX 5415 7712 7712 409 1072

(In reply to Roland Westrelin from comment #70)
> Created attachment 1691479 [details]
> possible fix
> 
> I went over the log files from the previous crash again, and I think there's
> a good chance the attached patch fixes it.

OK, I'll give it a shot.

(In reply to Simeon Andreev from comment #73)
> OK, I'll give it a shot.

Thanks. I took a look at the last log you attached and AFAICT it confirms the fix should work.

I still see a crash with the patch from comment 70:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ffff66df339, pid=68644, tid=68651
#
# JRE version: OpenJDK Runtime Environment (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, sharing, tiered, compressed oops, parallel gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xac3339]  PhaseIdealLoop::build_loop_late_post(Node*)+0x159
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Dosgi.dataAreaRequiresExplicitInit=true -Xmx2048m --add-modules=ALL-SYSTEM -D93k.eclipseid=/93k_workspace -DXOC_SYSTEM=/workspaces/socbm275/sandreev/ws-java11/vobs/zenith/workspace/system -D93k.isOnline=false -Xmx24G -Xms1G -Xss5m -XX:ReservedCodeCacheSize=256m -XX:+UseParallelGC -DmaxCompiledUnitsAtOnce=0 -Dxtext.qn.interning=true -Dxtext.skipSecondaryTypes=true -Dosgi.bundlefile.limit=100 -Declipse.log.size.max=20000 -Declipse.log.backup.max=50 -Dorg.eclipse.swt.internal.gtk.noThemingFixes -Dorg.eclipse.core.resources.disable_advanced_recursive_link_checks -Dorg.eclipse.wst.validation.ui.disable.validation.context.menu=true -Dfile.encoding=UTF8 -Dorg.eclipse.tips.startup.default=dialog -Dlog4jloc=/workspaces/socbm275/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/logs/log4j/log4jlog.txt -javaagent:/opt/hp93000rt/el7/x86_64/jacoco-0.7.x/0.8.5/lib/jacocoagent.jar=destfile=/workspaces/socbm275/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/reports/coverage/jacoco.exec -XX:ErrorFile=/tmp/sandreev_ws-java11//eclipse_vm_crash_%p.log /workspaces/socbm275/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar -os linux -ws gtk -arch x86_64 -launcher /workspaces/socbm275/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest/eclipse -name Eclipse --launcher.library /workspaces/socbm275/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.1100.v20190907-0426/eclipse_1801.so -startup /workspaces/socbm275/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar --launcher.appendVmargs -exitdata b6d80a8 -showLocation -product com.verigy.itee.ui.ewcProduct -application com.verigy.itee.testframework.ewc_test_framework -configuration /opt/hp93000/soc64_IworkspacesIsocbm275IsandreevIws-java11/segments/eclipse/eclipseForTest/configuration -data /tmp/sandreev_ws-java11/ -clean -vm /usr/lib/jvm/java-11/bin/java -vmargs -Dosgi.dataAreaRequiresExplicitInit=true -Xmx2048m --add-modules=ALL-SYSTEM -D93k.eclipseid=/93k_workspace -DXOC_SYSTEM=/workspaces/socbm275/sandreev/ws-java11/vobs/zenith/workspace/system -D93k.isOnline=false -Xmx24G -Xms1G -Xss5m -XX:ReservedCodeCacheSize=256m -XX:+UseParallelGC -DmaxCompiledUnitsAtOnce=0 -Dxtext.qn.interning=true -Dxtext.skipSecondaryTypes=true -Dosgi.bundlefile.limit=100 -Declipse.log.size.max=20000 -Declipse.log.backup.max=50 -Dorg.eclipse.swt.internal.gtk.noThemingFixes -Dorg.eclipse.core.resources.disable_advanced_recursive_link_checks -Dorg.eclipse.wst.validation.ui.disable.validation.context.menu=true -Dfile.encoding=UTF8 -Dorg.eclipse.tips.startup.default=dialog -Dlog4jloc=/workspaces/socbm275/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/logs/log4j/log4jlog.txt -javaagent:/opt/hp93000rt/el7/x86_64/jacoco-0.7.x/0.8.5/lib/jacocoagent.jar=destfile=/workspaces/socbm275/sandreev/ws-java11/opt/93000/src/segments/eclipse/test_framework/test_results/INTEGRATION/reports/coverage/jacoco.exec -XX:ErrorFile=/tmp/sandreev_ws-java11//eclipse_vm_crash_%p.log -jar /workspaces/socbm275/sandreev/ws-java11/opt/hp93000/soc/segments/eclipse/eclipseForTest//plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar

Host: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz, 12 cores, 62G, Red Hat Enterprise Linux Workstation release 7.4 (Maipo)
Time: Tue May 26 14:40:34 2020 CEST elapsed time: 205 seconds (0d 0h 3m 25s)

---------------  T H R E A D  ---------------

Current thread (0x00007ffff0238000):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=68651, stack(0x00007fff955e4000,0x00007fff956e5000)]


Current CompileTask:
C2: 205352 46760   !   4       org.eclipse.xtext.ui.XtextProjectHelper::hasBuilder (151 bytes)

Stack: [0x00007fff955e4000,0x00007fff956e5000],  sp=0x00007fff956e0610,  free space=1009k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xac3339]  PhaseIdealLoop::build_loop_late_post(Node*)+0x159
V  [libjvm.so+0xac380b]  PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x13b
V  [libjvm.so+0xac6b14]  PhaseIdealLoop::build_and_optimize(LoopOptsMode)+0x844
V  [libjvm.so+0x67b440]  Compile::Optimize()+0x780
V  [libjvm.so+0x67d1dc]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0x10ec
V  [libjvm.so+0x5a17d9]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x109
V  [libjvm.so+0x686005]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x485
V  [libjvm.so+0x687868]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xeafc52]  JavaThread::thread_main_inner()+0x232
V  [libjvm.so+0xeac34a]  Thread::call_run()+0x16a
V  [libjvm.so+0xc216e8]  thread_native_entry(Thread*)+0xf8

How do I double check that "loopPredicate.cpp" is patched? The source RPM doesn't seem to contain native sources?

I extracted the sources from the updated openjdk source RPM, it has no *.cpp files:

/usr/lib/jvm/java-11-openjdk-11.0.7.10-8.el7.x86_64/lib/src.zip (the path is listed by: rpm -ql java-11-openjdk-src-11.0.7.10-8.el7.x86_64, the "10" is my bump for the version, to avoid downgrading the RPM before installing the newly patched one)

The patch that I used for the JDK build from the source RPM:

diff --git openjdk/src/hotspot/share/opto/loopPredicate.cpp openjdk/src/hotspot/share/opto/loopPredicate.cpp
--- openjdk/src/hotspot/share/opto/loopPredicate.cpp
+++ openjdk/src/hotspot/share/opto/loopPredicate.cpp
@@ -120,6 +120,7 @@
       Node* n = uncommon_proj->fast_out(i);
       if (n->is_Load() || n->is_Store()) {
         _igvn.replace_input_of(n, 0, rgn);
+        set_ctrl(n, rgn);
         --i; --imax;
       }
     }

In the .spec file, this patch is applied with:

[..]
Patch2080: jep2080.patch
[..]
%patch2080
[..]

The JDK and RPM build was successful with this, though double-checking the patch is included wouldn't hurt.

> I still see a crash with the patch from comment 70:

It doesn't crash compiling the same method. Maybe it's a different bug.
FYI, the loopPredicate.cpp patch fixes an actual issue that I can trigger with a simple test case. It also affects the current jdk development version and is in the process of being fixed there (https://bugs.openjdk.java.net/browse/JDK-8245714 and https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-May/038321.html).

> How do I double check that "loopPredicate.cpp" is patched? The source RPM
> doesn't seem to contain native sources?

I don't know that. I will ask around.

Is that crash reproducible? Does it always occur compiling the same method?
Can you run with both patches (the fix + the one that produces the graph xml files) and change the command line to:

-XX:CompileCommand=option,org.eclipse.xtext.ui.XtextProjectHelper::hasBuilder,intx,IGVPrintLevel,2 -XX:+PrintIdealGraph -XX:PrintIdealGraphFile=graph.xml

(In reply to Simeon Andreev from comment #77)
> The patch that I used for the JDK build from the source RPM:
> 
> diff --git openjdk/src/hotspot/share/opto/loopPredicate.cpp
> openjdk/src/hotspot/share/opto/loopPredicate.cpp
> --- openjdk/src/hotspot/share/opto/loopPredicate.cpp
> +++ openjdk/src/hotspot/share/opto/loopPredicate.cpp
> @@ -120,6 +120,7 @@
>        Node* n = uncommon_proj->fast_out(i);
>        if (n->is_Load() || n->is_Store()) {
>          _igvn.replace_input_of(n, 0, rgn);
> +        set_ctrl(n, rgn);
>          --i; --imax;
>        }
>      }
> 
> In the .spec file, this patch is applied with:
> 
> [..]
> Patch2080: jep2080.patch
> [..]
> %patch2080
> [..]
> 
> The JDK and RPM build was successful with this, though double-checking the
> patch is included wouldn't hurt.

This looks right to me, but I can provide a test build if needed. Just let me know the RHEL version and patches required (as we did for the JDWP bug)

I'll try with the debug patch from comment 66 and the fix from comment 70, all changes in the same patch file. That way it should be clear the patch is applied, and there should be more output for the crash itself. Reproducing might take a day or two, the issue is seen once roughly in 1 out of 3 test runs (and each test run takes about 8 hours).

Created attachment 1692934 [details]
Crash logs with patches from comment 66 and comment 70.

Here is a graph*.xml file with the patches from comments 66 and 70, in the same method as before. Extra command line arguments:

-XX:CompileCommand=option,org.eclipse.dltk.core.PriorityDLTKExtensionManager::findScriptNature,intx,IGVPrintLevel,2 -XX:+PrintIdealGraph -XX:PrintIdealGraphFile=graph.xml

Last line from the JDK:

XXX 4134 6325 6325 287 881

Created attachment 1692984 [details]
revised fix, replaces previous attempt

(In reply to Simeon Andreev from comment #81)
> Created attachment 1692934 [details]
> Crash logs with patches from comment 66 and comment 70.

Thanks. It looks like another variant of the previous bug. I attached a new fix that replaces the previous one. Can you give it a try?

(In reply to Roland Westrelin from comment #83)
> It looks like another variant of the previous bug. I attached a new
> fix that replaces the previous one. Can you give it a try?

Tests are running, I'll let you know when we have notable results.

No crashes over the weekend with the patch from comment 83. I'll keep running the tests this week, just to be certain.

Created attachment 1694731 [details]
upstream fix

(In reply to Simeon Andreev from comment #85)
> No crashes over the weekend with the patch from comment 83. I'll keep
> running the tests this week, just to be certain.

I attached the patch that was pushed upstream. It's different from the one you've been testing (should be more robust). Would you be willing to verify that one?

(In reply to Roland Westrelin from comment #86)
> Created attachment 1694731 [details]
> upstream fix

OK, I'll run our tests with this fix.

No crash in our tests since my last comment, so I assume the crash is fixed. Many thanks!

Which version of OpenJDK 11 would have the fix? 11.0.8 or maybe 11.0.9?

(In reply to Simeon Andreev from comment #89)
> No crash in our tests since my last comment, so I assume the crash is fixed.
> Many thanks!

Thanks for verifying the fix. It's already integrated upstream for jdk 15. I will request backport to 11.

> Which version of OpenJDK 11 would have the fix? 11.0.8 or maybe 11.0.9?

It's too late for 11.0.8. It should get in 11.0.9

Patch is in 11.0.9, so we can get this in the RPMs for RHEL 7.9.

OK, TYVM!

Andrew, I see that fixed in version is set to "java-11-openjdk-11.0.8.2-0.3.ea.el7", we have "java-11-openjdk-11.0.8.10-0.el7.x86_64" - is the fix inside or not?

Asking because we've got the package before the "fixed in" flag was set here.

(In reply to Andrey Loskutov from comment #98)
> Andrew, I see that fixed in version is set to
> "java-11-openjdk-11.0.8.2-0.3.ea.el7", we have
> "java-11-openjdk-11.0.8.10-0.el7.x86_64" - is the fix inside or not?
> 
> Asking because we've got the package before the "fixed in" flag was set here.

It's fixed in java-11-openjdk-11.0.8.10-0.el7.x86_64 as well. It'll also be fixed in upstream 11.0.9 and better.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (java-11-openjdk bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3919

Created attachment 1720839 [details]
JDK crash using parallel GC with java-11-openjdk-11.0.8.10-0.el7.x86_64

While testing with JDK 1.0.8.10-0 and parallel GC we ran into a crash:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ffff673e359, pid=38428, tid=38435
#
# JRE version: OpenJDK Runtime Environment 18.9 (11.0.8+10) (build 11.0.8+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.8+10-LTS, mixed mode, sharing, tiered, compressed oops, parallel gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xac2359]  PhaseIdealLoop::build_loop_late_post(Node*)+0x159
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h" (or dumping to /tmp/sandreev_ws-dev-andreev-2_devices/74act299/core.38428)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

Is this the same crash as the one here, or another one?

Created attachment 1720885 [details]
Replay log file from crash with JDK 11.0.8.10-0

(In reply to Severin Gehwolf from comment #99)
> (In reply to Andrey Loskutov from comment #98)
> > Andrew, I see that fixed in version is set to
> > "java-11-openjdk-11.0.8.2-0.3.ea.el7", we have
> > "java-11-openjdk-11.0.8.10-0.el7.x86_64" - is the fix inside or not?
> > 
> > Asking because we've got the package before the "fixed in" flag was set here.
> 
> It's fixed in java-11-openjdk-11.0.8.10-0.el7.x86_64 as well. It'll also be
> fixed in upstream 11.0.9 and better.

Sorry, this should be have been java-11-openjdk-11.0.8.10-1.el7 from RHEL 7.9. In particular from this erratum:
https://access.redhat.com/errata/RHBA-2020:3919

This makes me wonder where you got java-11-openjdk-11.0.8.10-0.el7.x86_64 from? If that's a rebuild from rhel 7.8, specifically, java-11-openjdk-11.0.8.10-0.el7_8, then this build would NOT have the patch it. This will align once 11.0.9 comes out in about a weeks time.

Yes, we got the RPMs from e.g. here: https://access.redhat.com/downloads/content/java-11-openjdk/11.0.8.10-0.el7_8/x86_64/fd431d51/package (so for RHEL 7.8)

(In reply to Simeon Andreev from comment #104)
> Yes, we got the RPMs from e.g. here:
> https://access.redhat.com/downloads/content/java-11-openjdk/11.0.8.10-0.
> el7_8/x86_64/fd431d51/package (so for RHEL 7.8)

OK. That one wouldn't have the fix for this issue.

You'd need this one:
https://access.redhat.com/downloads/content/java-11-openjdk/11.0.8.10-1.el7/x86_64/fd431d51/package

OK, we'll check with those RPMs.