We have two intermediate patches (attached) which gets us further. However with those it fails in java TestCryptoLevel with:

+ /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.x86_64/openjdk/build/jdk8.build/images/j2sdk-image/bin/javac -d
 . /builddir/build/SOURCES/TestCryptoLevel.java
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f6b25ed736b, pid=5973, tid=140097488013056
#
# JRE version: OpenJDK Runtime Environment (8.0_72-b15) (build 1.8.0_72-b15)
# Java VM: OpenJDK 64-Bit Server VM (25.72-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x88636b]  Node::Node(unsigned int)+0x2b
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting \
Java again
#
# An error report file with more information is saved as:
# /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.x86_64/hs_err_pid5973.log
[thread 140097475966720 also had an error]
[thread 140097474914048 also had an error]
[thread 140097473861376 also had an error]
[thread 140097477019392 also had an error]
[error occurred during error reporting , id 0xb]

Building a fastdebug hotspot that asserts with:

#  Internal Error (/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.i386/openjdk/hotspot/src/share/vm/opto/node.cpp:298), pid=13094, tid=2730490688
#  assert(Compile::current() == C) failed: must use operator new(Compile*)

See:
http://koji.fedoraproject.org/koji/getfile?taskID=12930370&name=build.log&offset=-4000

Created attachment 1123106 [details]
Use older C/C++ standard with GCC6.

Created attachment 1123107 [details]
configure work-around

This is starting to look like an issue with hotspot + GCC6. We are still investigating.

A better trace of the fastdebug assertion is:

+ /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.i386/openjdk/build/jdk8.build-debug/images/j2sdk-image/bin/javac -d . /builddir/build/SOURCES/TestCryptoLevel.java
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/node.cpp:298
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.i386/openjdk/hotspot/src/share/vm/opto/node.cpp:298), pid=13094, tid=2730490688
#  assert(Compile::current() == C) failed: must use operator new(Compile*)
#
# JRE version: OpenJDK Runtime Environment (8.0_72-b15) (build 1.8.0_72-fastdebug-b15)
# Java VM: OpenJDK Server VM (25.72-b15-fastdebug mixed mode linux-x86 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.i386/hs_err_pid13094.log
[thread -1565533376 also had an error]
[error occurred during error reporting , id 0xb]
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

Adding a compiler barrier, as shown here, fixes the segfault, which is caused by Node._out not being initialized.  I strongly suspect that GCC does not initialize _out because Compile is not compatible with its type.  While this is, strictly speaking, correct, we are compiling with -fno-strict-aliasing and GCC should respect that.

I very strongly suspect that -fno-strict-aliasing is broken in this version of GCC.

inline void* operator new( size_t x, Compile* C) throw() {
   Node* n = (Node*)C->node_arena()->Amalloc_D(x);
#ifdef ASSERT
   n->_in = (Node**)n; // magic cookie for assertion check
#endif
   n->_out = (Node**)C;
   __asm__ volatile("nop" : : : "memory");
   return (void*)n;
}

Have you tried the code with -fsanitize=undefined?  Does -fno-delete-null-pointer-checks help?  Does building with -O0 help?  If yes, can you bisect it to a problematic *.o file, attach it preprocessed here, including all gcc options?

(In reply to Jakub Jelinek from comment #7)
> Does building with -O0 help?

Yes. We've done a slowdebug build which seems to work. That's -O0 I believe.

> If yes, can you bisect it to a problematic *.o file, attach it preprocessed
> here, including all gcc options?

It's going to take a while, but it's possible.

Well, you shouldn't need to rebuild everything each time, just keep around copies of an optimized and non-optimized builds and then in a third copy always copy in + touch always some set of optimized *.o files and rest of unoptimized *.o files, relink, retest, and using binary search it should take just log2 of number of *.o files.  If multiple shared libraries or binaries are involved, first step can be to bisect among the shared libraries/binaries to find the problematic one.

(In reply to Jakub Jelinek from comment #7)
> Have you tried the code with -fsanitize=undefined? 

No.  -fsanitize=undefined is fairly problematic with HotSpot, given that it warns about several things which are well-defined in GCC and everybody uses all the time, e.g. left shifts of negative integers.  It'd be nice to have a sensible set of options for building with fsanitize=undefined.

> Does
> -fno-delete-null-pointer-checks help?  Does building with -O0 help?  If yes,
> can you bisect it to a problematic *.o file, attach it preprocessed here,
> including all gcc options?

OK.  It's node.o.  I'll get you the preprocessed file.

Andrew.

(In reply to Andrew Haley from comment #10)
> (In reply to Jakub Jelinek from comment #7)
> > Have you tried the code with -fsanitize=undefined? 
> 
> No.  -fsanitize=undefined is fairly problematic with HotSpot, given that it
> warns about several things which are well-defined in GCC and everybody uses
> all the time, e.g. left shifts of negative integers.  It'd be nice to have a
> sensible set of options for building with fsanitize=undefined.

You can try using -fsanitize=null to see if the code isn't calling a member function on a null pointer or similar.

(In reply to Jakub Jelinek from comment #9)
> Well, you shouldn't need to rebuild everything each time, just keep around
> copies of an optimized and non-optimized builds and then in a third copy
> always copy in + touch always some set of optimized *.o files and rest of
> unoptimized *.o files, relink, retest, and using binary search it should
> take just log2 of number of *.o files.  If multiple shared libraries or
> binaries are involved, first step can be to bisect among the shared
> libraries/binaries to find the problematic one.

Understood. Still getting a local setup with a non-optimized build, an optimized build and then the script run would have taken a while. Anyway, a moot point now.

-fno-guess-branch-probability is the wrong flag to use because it hits too much other stuff.

The real bug is in due to an object field being accessed before the object is constructed, in breach of C++98 [class.cdtor]:

  For an object of non-POD class type ... before the constructor begins execution ... referring to any non-static member or base class of the object results in undefined behavior

Instead of -fno-guess-branch-probability we should use -fno-lifetime-dse:

  -fno-lifetime-dse

  In C++ the value of an object is only affected by changes within its
  lifetime: when the constructor begins, the object has an indeterminate
  value, and any changes during the lifetime of the object are dead when
  the object is destroyed.  Normally dead store elimination will take
  advantage of this; if your code relies on the value of the object
  storage persisting beyond the lifetime of the object, you can use this
  flag to disable this optimization.

Please try this flag instead.

Scratch build with -fno-lifetime-dse instead of -fno-guess-branch-probability:

http://koji.fedoraproject.org/koji/taskinfo?taskID=13009074

FWIW, reproducer works fine for me for a fastdebug hotspot-only build using -fno-lifetime-dse.

Created attachment 1127635 [details]
spec file patch changing flag to -fno-lifetime-dse

Task which properly boot-cycled on i686 with -fno-lifetime-dse:
http://koji.fedoraproject.org/koji/taskinfo?taskID=13009566

This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase

There is a work-around for this issue. Use -fno-lifetime-dse and -fno-delete-null-pointer-checks. It's unlikely to get fixed in upstream JDK 8. I'm closing this bug.

(In reply to Severin Gehwolf from comment #55)
> It's unlikely to get fixed in upstream JDK 8.

That is, the underlying issue of undefined behaviour being present in JDK 8 hotspot.