Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1813550 - java segmentation faults in GCC10-built java-1.8.0-openjdk
Summary: java segmentation faults in GCC10-built java-1.8.0-openjdk
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: java-1.8.0-openjdk
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Andrew Haley
QA Contact: Fedora Extras Quality Assurance
URL: https://bugs.openjdk.java.net/browse/...
Whiteboard:
Depends On: 1795268
Blocks: 1791951
TreeView+ depends on / blocked
 
Reported: 2020-03-14 11:18 UTC by Fabio Valentini
Modified: 2020-10-04 21:46 UTC (History)
15 users (show)

Fixed In Version: java-1.8.0-openjdk-1.8.0.242.b08-1.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-24 20:40:07 UTC
Type: Bug
decathorpe: fedora_prioritized_bug?


Attachments (Terms of Use)
example of JVM crash (from a local junit mock build on fedora-rawhide-x86_64) (130.17 KB, text/plain)
2020-03-15 14:39 UTC, Fabio Valentini
no flags Details
hserr pid from 17.3.2020 (189.73 KB, text/plain)
2020-03-17 16:37 UTC, jiri vanek
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github dogtagpki pki issues 3247 0 None None None 2020-10-04 21:46:27 UTC
openjdk bug system JDK-8241296 0 None None None 2020-03-19 17:32:26 UTC

Description Fabio Valentini 2020-03-14 11:18:18 UTC
Since the update of java-1.8.0-openjdk
from 1:1.8.0.242.b06-0.0.ea.fc32
to   1:1.8.0.242.b08-0.fc33

(which should have fixed FTBFS on fedora 32+ and with GCC 10)

I'm seeing Java segmentation faults during java package builds.

This is starting to affect a lot of packages:

https://koschei.fedoraproject.org/affected-by/java-1.8.0-openjdk-headless?epoch1=1&version1=1.8.0.242.b06&release1=0.0.ea.fc32&epoch2=1&version2=1.8.0.242.b08&release2=0.fc33&collection=f33

So I think there's some serious issue with this build ...

I'm not sure if it also affects the corresponding f32 update:

https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6

Comment 1 Fabio Valentini 2020-03-15 14:39:51 UTC
Created attachment 1670312 [details]
example of JVM crash (from a local junit mock build on fedora-rawhide-x86_64)

Attached a locally produced crash when building junit in mock with "-r fedora-rawhide-x86_64".

Comment 2 Raphael Groner 2020-03-15 14:54:26 UTC
The log indicates a bug in surefire plugin. Better assign this bug to there, or generally mockito executed by maven.
Wouldn't wonder if surefire isn't able to parse any character(s) got from the maven xml. Just a guessing because there are reported issues (partly fixed) with surefire and unicode (umlauts) or other special characters.

Comment 3 Raphael Groner 2020-03-15 15:12:30 UTC
Shorten backtrace for the relevant thread (0x00007f2eb031b000) only:

 1285       4       java.lang.CharacterDataLatin1::toLowerCase (39 bytes)
 nmethod 1285 0x00007f2ea15e0450 code [0x00007f2ea15e05a0, 0x00007f2ea15e0678]
 1289       4       org.apache.maven.surefire.util.internal.StringUtils::escapeToPrintable (152 bytes)
 nmethod 1289 0x00007f2ea15e7350 code [0x00007f2ea15e7520, 0x00007f2ea15e7b60]
 1314       4       sun.reflect.ClassFileAssembler::emitConstantPoolUTF8 (50 bytes)
 nmethod 1314 0x00007f2ea160c190 code [0x00007f2ea160c3a0, 0x00007f2ea160ced0]
 1349       4       java.lang.Class::getName (21 bytes)
 nmethod 1349 0x00007f2ea15c76d0 code [0x00007f2ea15c7820, 0x00007f2ea15c78c8]
 1322       4       java.lang.String::indexOf (7 bytes)
 nmethod 1322 0x00007f2ea1608750 code [0x00007f2ea16088a0, 0x00007f2ea1608a18]


That's clearly a bug with unicode. Can you assign the complete maven build file?

Comment 4 Raphael Groner 2020-03-15 15:27:00 UTC
openjdk itself FTBFS in F32, so I tend to say this bug depends on bug #1795268.

Comment 5 Alex Scheel 2020-03-15 16:19:23 UTC
We're also seeing this when running an existing (already built) Java package in Fedora Rawhide. See:

https://pagure.io/dogtagpki/issue/3130

Comment 6 Fabio Valentini 2020-03-15 16:28:19 UTC
This seems to be the "common core" of all crashes I've looked at:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f82a66cdfbe, pid=120, tid=0x00007f82a48f8700
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x701fbe]  JNIHandleBlock::oops_do(OopClosure*)+0xae

Everything else seems to not be reproducible.

During package builds, this crash affects mostly tests, probably just because those exercises the JVM most.
But it seems to be triggered randomly during any execution of Java code.

Comment 7 Raphael Groner 2020-03-15 18:52:20 UTC
(In reply to Raphael Groner from comment #3)
…
> That's clearly a bug with unicode. Can you assign the complete maven build
> file?

Can you please answer my question?

Comment 8 Fabio Valentini 2020-03-15 20:05:08 UTC
After some investigation, it looks like these crashes originate somewhere in memory allocation / Garbage Collection, for example (from the junit crash):

Current thread (0x00007f2eb0066000):  GCTaskThread [stack: 0x00007f2e9fa35000,0x00007f2e9fb35000] [id=190]

Stack: [0x00007f2e9fa35000,0x00007f2e9fb35000],  sp=0x00007f2e9fb33560,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x701fbe]  JNIHandleBlock::oops_do(OopClosure*)+0xae
V  [libjvm.so+0xae8b6c]  JavaThread::oops_do(OopClosure*, CLDClosure*, CodeBlobClosure*)+0x3c
V  [libjvm.so+0x9a633a]  ThreadRootsTask::do_it(GCTaskManager*, unsigned int)+0x7a
V  [libjvm.so+0x5d8274]  GCTaskThread::run()+0x1d4
V  [libjvm.so+0x92fd78]  java_start(Thread*)+0x1c8

Comment 9 Fabio Valentini 2020-03-15 20:14:40 UTC
And I've just been able to reproduce this crash with the pending fedora 32 update as well:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6

Comment 10 Raphael Groner 2020-03-16 17:04:49 UTC
(In reply to Fabio Valentini from comment #9)
> And I've just been able to reproduce this crash with the pending fedora 32
> update as well:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6

OK, so this means I fail to update my packages based on java as long it's jdk8, especially jpype (see bug #1791951 and bug #1808625).

Comment 11 Fabio Valentini 2020-03-16 17:13:45 UTC
(In reply to Raphael Groner from comment #10)
> (In reply to Fabio Valentini from comment #9)
> > And I've just been able to reproduce this crash with the pending fedora 32
> > update as well:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6
> 
> OK, so this means I fail to update my packages based on java as long it's
> jdk8, especially jpype (see bug #1791951 and bug #1808625).

Yep, this is blocking any package updates for builds that are affected by this crash.
Which is pretty bad, given how often Java packages get CVEs or need other critical fixes.

For now, you should still be able to build things on fedora 32, though.
The broken f32 update has not been pushed to stable yet, and there's no buildroot override for it.

Comment 12 Deepak Bhole 2020-03-16 19:50:14 UTC
Not sure if this is due to the GCC10 fix or something else. Assigning to aph to take a look.

Comment 13 Fabio Valentini 2020-03-17 16:12:33 UTC
Proposing this as a prioritized bug, since it's blocking a lot of updates in rawhide, and is blocking inclusion of future OpenJDK 8 security updates in fedora 32+.

Comment 14 jiri vanek 2020-03-17 16:37:27 UTC
Created attachment 1670862 [details]
hserr pid from 17.3.2020

 while true ; do mock -r fedora-rawhide-x86_64 java-runtime-decompiler-3.0-6.fc33.src.rpm   ; done

Failed 3 times in 10 builds Attached the trace.

Comment 15 Raphael Groner 2020-03-17 16:41:29 UTC
We could try to propose as a blocker for F32 Final.

Comment 16 Andrew Haley 2020-03-18 12:32:01 UTC
I have reproduced this problem and I'm working on it.

Comment 17 Adam Williamson 2020-03-19 16:23:48 UTC
The crash itself can't be a blocker for F32, as the broken Java did not make it through updates-testing (I've just unpushed it). But 'we need a newer Java for security fixes' could be, if any of the security issues is 'important' or higher on the Red Hat scale:

https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Security_bugs

it would be best to have a bug that is actually for such a security issue and propose *that* as the blocker, though (then have this block that).

Comment 18 Raphael Groner 2020-03-19 16:54:20 UTC
(In reply to Adam Williamson from comment #17)
…
> it would be best to have a bug that is actually for such a security issue
> and propose *that* as the blocker, though (then have this block that).

My packages are not security risky, obviously. Maybe take a look generally in (unfixed) jdk8 CVE?

Comment 19 Andrew John Hughes 2020-03-19 17:11:45 UTC
(In reply to Adam Williamson from comment #17)
> The crash itself can't be a blocker for F32, as the broken Java did not make
> it through updates-testing (I've just unpushed it). But 'we need a newer
> Java for security fixes' could be, if any of the security issues is
> 'important' or higher on the Red Hat scale:
> 
> https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Security_bugs
> 
> it would be best to have a bug that is actually for such a security issue
> and propose *that* as the blocker, though (then have this block that).

Sorry, can you be more specific as to what was 'unpushed'?

The bug here isn't due to a change in the OpenJDK package. It's due to the way GCC has started compiling code that has been in OpenJDK since its inception.

Andrew Haley has located the bug and a fix: https://bugs.openjdk.java.net/browse/JDK-8241296

If it doesn't show up on older builds, it's because they were built before GCC 10 was added. Any further rebuilds will need this fix to avoid this bug.

I'm going to work on getting it into rawhide now.

Comment 20 Fabio Valentini 2020-03-19 17:32:15 UTC
(In reply to Andrew John Hughes from comment #19)
> Sorry, can you be more specific as to what was 'unpushed'?

The fedora 32 update for java-1.8.0-openjdk which has this crasher bug has been "unpushed":
https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6#comment-1286568

Which means it cannot be "pushed" into stable repositories now.

> The bug here isn't due to a change in the OpenJDK package. It's due to the
> way GCC has started compiling code that has been in OpenJDK since its
> inception.
> 
> Andrew Haley has located the bug and a fix:
> https://bugs.openjdk.java.net/browse/JDK-8241296
> 
> If it doesn't show up on older builds, it's because they were built before
> GCC 10 was added. Any further rebuilds will need this fix to avoid this bug.
> 
> I'm going to work on getting it into rawhide now.

That's great news. Thank you!

Once it lands in rawhide, I'll check it out and confirm whether it fixes the issues I've been seeing.
If it indeed does, then the update should probably go into fedora 32 as well (since it has currently an outdated openjdk8 version).

Comment 21 Raphael Groner 2020-03-19 17:32:27 UTC
Thanks. Maybe try to file a pull request.

Comment 22 Adam Williamson 2020-03-19 17:55:31 UTC
Andrew Haley is also the downstream packager, so I would just relax and expect that he's perfectly capable of bringing the fix downstream once he's ready :) If it's not done in a day or two we can start worrying about it I guess.

Comment 23 Adam Williamson 2020-03-19 17:57:07 UTC
sorry, I meant both the Andrews work on the downstream package, so between them they're certainly capable of getting the fix landed.

Comment 24 Andrew Haley 2020-03-19 18:50:21 UTC
(In reply to Andrew John Hughes from comment #19)
> The bug here isn't due to a change in the OpenJDK package. It's due to the
> way GCC has started compiling code that has been in OpenJDK since its
> inception.
> 
> Andrew Haley has located the bug and a fix:
> https://bugs.openjdk.java.net/browse/JDK-8241296

And it's now double-approved upstream, so we're good to go. I'll also
create some backport requests for 8u and 11u tomorrow.

Comment 25 Andrew John Hughes 2020-03-20 15:53:29 UTC
(In reply to Fabio Valentini from comment #20)
> (In reply to Andrew John Hughes from comment #19)
> > Sorry, can you be more specific as to what was 'unpushed'?
> 
> The fedora 32 update for java-1.8.0-openjdk which has this crasher bug has
> been "unpushed":
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6#comment-
> 1286568
> 
> Which means it cannot be "pushed" into stable repositories now.
> 
> > The bug here isn't due to a change in the OpenJDK package. It's due to the
> > way GCC has started compiling code that has been in OpenJDK since its
> > inception.
> > 
> > Andrew Haley has located the bug and a fix:
> > https://bugs.openjdk.java.net/browse/JDK-8241296
> > 
> > If it doesn't show up on older builds, it's because they were built before
> > GCC 10 was added. Any further rebuilds will need this fix to avoid this bug.
> > 
> > I'm going to work on getting it into rawhide now.
> 
> That's great news. Thank you!
> 
> Once it lands in rawhide, I'll check it out and confirm whether it fixes the
> issues I've been seeing.
> If it indeed does, then the update should probably go into fedora 32 as well
> (since it has currently an outdated openjdk8 version).

Scratch build for rawhide is underway now: https://koji.fedoraproject.org/koji/taskinfo?taskID=42635480

Changes are in my local fork and I'll post a PR once that build succeeds: https://src.fedoraproject.org/fork/ahughes/rpms/java-1.8.0-openjdk/c/1b81e9374ae1c3d473461f20737119f489aa4b4c?branch=master

Also already completed the same for java-11-openjdk: https://koji.fedoraproject.org/koji/taskinfo?taskID=42633444

That old version of OpenJDK 8 is what I was worried about. We need to get an updated build into Fedora 32. The b06 version currently there is an early access build and doesn't include the last batch of security fixes from January (see the "Security Fixes" section of http://bitly.com/oj8u242). We were blocked from doing the later updates in what was then rawhide once GCC 10 went in (see bug #1795268). This would mean that Fedora 32 would regress from the version in Fedora 31.

If I need to file an additional blocker bug to make that happen, just let me know.

Comment 26 Fabio Valentini 2020-03-20 15:58:56 UTC
Nope, just merge the master branch into f32 after rawhide is done, and then build and submit a bodhi update as usual. there are no freezes in effect for feodra 32 right now, so there's nothing else you need to do (except wait a few days or for positive feedback, before it can be pushed to stable).

Comment 27 Adam Williamson 2020-03-20 16:31:07 UTC
Final freeze is scheduled for 2020-04-07; as long as you get the update submitted for stable before then it should be OK.

Comment 28 Andrew John Hughes 2020-03-20 19:07:47 UTC
If all goes well, I should be able to submit the update today.

Comment 29 Adam Williamson 2020-03-20 19:31:52 UTC
If you send a build to Rawhide at the same time, we'll find out with the next Rawhide compose whether things are OK (openQA runs on each Rawhide compose, and this bug always causes the FreeIPA tests to fail).

Comment 30 Andrew John Hughes 2020-03-20 21:32:18 UTC
I was about to send the build to rawhide, but it seems my previous build on rawhide, and my scratch build of this change, both failed on s390x:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42635501
https://koji.fedoraproject.org/koji/taskinfo?taskID=42634995

It seems the current rawhide JDK is not stable enough to build OpenJDK on s390x. I'm not sure if this is the same bug as this or something else. It doesn't seem obviously the same:

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_zero.cpp:261), pid=3765835, tid=0x000003ffa4bd2900
#  fatal error: caught unhandled signal 11

but that could be because this is the zero assembler port. armv7hl uses that too, incidentally, and has built.

It looks like the OpenJDK bootstrap mechanism does not perform what I would describe as a bootstrap. It's more of a minimal test build.

I would expect a bootstrap to do a minimal build with the system compiler, and then a full build using that build. That places minimum pressure on the system install and fully tests the one being built.

Instead, OpenJDK's bootstrapping just uses the just-built JDK to build a minimal image again, and then goes back to using the system JDK to build docs and demos, which is where it fails on s390x.

I'm going to look at changing the RPM to do a more usual bootstrap, as we used to do with java-1.{6,7}.0-openjdk. This should tell us a) if this fix resolves the problem and b) make it less likely from broken JDKs to end up being used to bootstrap in future.

Is there a way we can roll back rawhide to an earlier JDK as with F32? The one used by F32 seems to be the common ancestor that worked before GCC 10 builds were used.

Comment 31 Adam Williamson 2020-03-20 21:38:38 UTC
Well. Technically yes, but practically no. =) By policy Rawhide doesn't go backwards, once a build has made it into a successful compose we do not go backwards from it. In theory we *could* (it's just a matter of untagging all broken builds, then the next compose will include the older working build), but we don't do that. (It also wouldn't fix installed Rawhide systems on uodate ordinarily; users doing 'dnf update' would not be downgraded. You'd only be downgraded if you did 'dnf distro-sync'.)

If you can't figure out a way to get an s390x build through, we'd be left with only the options of leaving it broken or excluding s390x from the build (which would fix it on other arches, but mean s390x didn't have the package *at all* any more, until we got it building again).

Comment 32 Andrew John Hughes 2020-03-23 12:22:13 UTC
Looks like my changes to the bootstrapping process worked. It's built on rawhide:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42694611

f32 build seems to have stalled on s390x. I'm going to kick off another to see if that has more luck:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42704865

Comment 33 Andrew John Hughes 2020-03-23 12:24:27 UTC
New f32 build: https://koji.fedoraproject.org/koji/taskinfo?taskID=42716235

Comment 34 Fabio Valentini 2020-03-23 12:25:33 UTC
Thanks! I can confirm that the crashes I saw on rawhide are now gone.

Comment 35 Andrew John Hughes 2020-03-23 12:57:44 UTC
Great!
Now just hope this f32 build completes: https://koji.fedoraproject.org/koji/taskinfo?taskID=42716912

Comment 36 Andrew John Hughes 2020-03-23 21:41:27 UTC
Should be fixed now.

Comment 37 Adam Williamson 2020-03-23 22:26:44 UTC
You could edit the new build into the existing update and then mark it as fixing this bug.

Comment 38 Fedora Update System 2020-03-24 03:30:44 UTC
FEDORA-2020-42e79d74f1 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-42e79d74f1

Comment 39 Adam Williamson 2020-03-24 19:07:09 UTC
I've triggered the openQA tests on this update, we'll see how they go.

Comment 40 Fedora Update System 2020-03-24 20:40:07 UTC
FEDORA-2020-42e79d74f1 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 41 Raphael Groner 2020-03-25 14:07:16 UTC
(In reply to Fedora Update System from comment #40)
> FEDORA-2020-42e79d74f1 has been pushed to the Fedora 32 stable repository.
> If problem still persists, please make note of it in this bug report.

Thanks for the great work. Could you additionally fix those errors (marked as red in taskotron( given about rpmlint[¹] and rpmgrill[²]? Sorry for my late response after received enough positive karma for the autopush. Should I open a new bug for that?

¹ https://taskotron.fedoraproject.org/artifacts/all/a271f662-6d3e-11ea-9431-525400364adf/tests.yml/java-1.8.0-openjdk-1.8.0.242.b08-1.fc32.log

² https://taskotron.fedoraproject.org/artifacts/all/a2bc370e-6d3e-11ea-a1da-525400364adf/tests.yml/rpmgrill.json

Comment 42 Mat Booth 2020-03-27 11:12:38 UTC
I beleive I am still seeing the seg fault in the java-1.8.0-openjdk-aarch32 package -- is there still an update pending?

For example as seen in this task: https://koji.fedoraproject.org/koji/taskinfo?taskID=42782568

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xb5582e7c, pid=19031, tid=0xa25e3460
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b07) (build 1.8.0_242-b07)
# Java VM: OpenJDK Client VM (25.242-b07 mixed mode linux-aarch32 )
# Problematic frame:
# V  [libjvm.so+0x330e7c]  JNIHandleBlock::oops_do(OopClosure*)+0xa8
#
# Core dump written. Default location: /builddir/build/BUILD/netty-netty-4.1.13.Final/core or core.19031
#
# An error report file with more information is saved as:
# /builddir/build/BUILD/netty-netty-4.1.13.Final/hs_err_pid19031.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

Comment 43 Severin Gehwolf 2020-03-27 14:16:40 UTC
(In reply to Mat Booth from comment #42)
> I beleive I am still seeing the seg fault in the java-1.8.0-openjdk-aarch32
> package -- is there still an update pending?

Yes. It's a different package[1]. Please file a new bug for it, component java-1.8.0-openjdk-aarch32.

[1] https://src.fedoraproject.org/rpms/java-1.8.0-openjdk-aarch32

Comment 44 Mat Booth 2020-03-27 15:48:42 UTC
Please see bug 1818078

Comment 45 Andrew John Hughes 2020-03-29 23:15:09 UTC
(In reply to Raphael Groner from comment #41)
> (In reply to Fedora Update System from comment #40)
> > FEDORA-2020-42e79d74f1 has been pushed to the Fedora 32 stable repository.
> > If problem still persists, please make note of it in this bug report.
> 
> Thanks for the great work. Could you additionally fix those errors (marked
> as red in taskotron( given about rpmlint[¹] and rpmgrill[²]? Sorry for my
> late response after received enough positive karma for the autopush. Should
> I open a new bug for that?
> 
> ¹
> https://taskotron.fedoraproject.org/artifacts/all/a271f662-6d3e-11ea-9431-
> 525400364adf/tests.yml/java-1.8.0-openjdk-1.8.0.242.b08-1.fc32.log
> 
> ²
> https://taskotron.fedoraproject.org/artifacts/all/a2bc370e-6d3e-11ea-a1da-
> 525400364adf/tests.yml/rpmgrill.json

Most of these are long-term and are deliberate. They probably need whitelisting in some way. If I "fixed" them, the result would be a completely broken JDK.  Specifically, the $ORIGIN issues are required for the JDK to find its libraries in its directory tree, which is not in a standard location indexed by the linker (and can't be while supporting multiple JDKs)

The symlinks are resolved by other packages e.g. to use system copies of timezone data & security certificates.

The explicit dependencies are due to OpenJDK dynamically loading some libraries at runtime rather than linking against them (e.g. CUPS, libXcomposite)

So only the description & typos are worthy of actually being fixed.

Yes, this warrants a separate bug as it's not related to this crash. It probably also applies to the other OpenJDK packages (java-11-openjdk, java-latest-openjdk)


Note You need to log in before you can comment on or make changes to this bug.