Bug 1415137 - java-1.8.0-openjdk: NSS 3.28 update causes core dump
Summary: java-1.8.0-openjdk: NSS 3.28 update causes core dump
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: java-1.8.0-openjdk
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Andrew John Hughes
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1411116 1416121 1416173 1416525 1416561 1416574 1417092 1417295 1417317 1417350 1417544 (view as bug list)
Depends On:
Blocks: 1417207
TreeView+ depends on / blocked
 
Reported: 2017-01-20 11:25 UTC by Mikolaj Izdebski
Modified: 2017-04-04 07:59 UTC (History)
46 users (show)

Fixed In Version: java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 java-1.8.0-openjdk-1.8.0.121-8.b14.fc24 java-1.8.0-openjdk-1.8.0.121-8.b14.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-03 21:18:34 UTC
Type: Bug


Attachments (Terms of Use)
hs log (43.29 KB, text/plain)
2017-01-20 11:26 UTC, Mikolaj Izdebski
no flags Details
working java.security file (no segfault) (22.40 KB, text/plain)
2017-01-20 11:52 UTC, Mikolaj Izdebski
no flags Details
non-working java.security file (segfault) (26.28 KB, text/plain)
2017-01-20 11:53 UTC, Mikolaj Izdebski
no flags Details
diff between working and non-working java.security (6.04 KB, patch)
2017-01-20 11:54 UTC, Mikolaj Izdebski
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Gentoo 605430 None None None 2017-01-24 22:45:47 UTC

Description Mikolaj Izdebski 2017-01-20 11:25:36 UTC
Description of problem:
OpenJDK crashes when trying to connect to SSL socket.
This happens only with new java.security. The problem can be fixed by reverting to old config file.

Version-Release number of selected component (if applicable):
java-1.8.0-openjdk-1.8.0.111-5.b16.fc25.x86_64

Reproducer:

import javax.net.ssl.*;
public class SSLSocketClient {
  public static void main(String[] args) throws Exception {
    try (SSLSocket socket = (SSLSocket)SSLSocketFactory.getDefault().createSocket("151.101.36.215", 443)) {
      socket.startHandshake();
    }
  }
}

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe00647254f, pid=10298, tid=0x00007fe007126700
#
# JRE version: OpenJDK Runtime Environment (8.0_111-b16) (build 1.8.0_111-b16)
# Java VM: OpenJDK 64-Bit Server VM (25.111-b16 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x14f54f]  __memmove_avx_unaligned_erms+0x4f
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

Crash log attached.

Comment 1 Mikolaj Izdebski 2017-01-20 11:26:37 UTC
Created attachment 1242721 [details]
hs log

Comment 2 jiri vanek 2017-01-20 11:48:00 UTC
java-1.8.0-openjdk-1.8.0.111-5.b16.fc25.x86_64
already have fixed java.security of security.useSystemPropertiesFile=false
So the fault might be aligned with it turned off, but enabled in code

Comment 3 Mikolaj Izdebski 2017-01-20 11:52:37 UTC
Created attachment 1242727 [details]
working java.security file (no segfault)

Comment 4 Mikolaj Izdebski 2017-01-20 11:53:14 UTC
Created attachment 1242728 [details]
non-working java.security file (segfault)

Comment 5 Mikolaj Izdebski 2017-01-20 11:54:03 UTC
Created attachment 1242729 [details]
diff between working and non-working java.security

Comment 6 Gerard Ryan 2017-01-24 12:56:07 UTC
I did a `dnf upgrade` today, and I'm seeing this everywhere now. Is there a fix I can try, or is there a specific package that I can safely downgrade to workaround it?

Looking at the packages that got upgraded, and the conversation here, I'm assuming that it's the new nss* packages that triggered this for me. Does that sound right? Can those be safely downgraded?

Comment 7 buddabrod 2017-01-24 13:06:53 UTC
I hit this, too. Downgrading the nss* packages solved it for the meantime.

Comment 8 Severin Gehwolf 2017-01-24 13:55:59 UTC
(In reply to buddabrod from comment #7)
> I hit this, too. Downgrading the nss* packages solved it for the meantime.

Can you tell us which nss packages work and which do not? Perhaps some /var/log/dnf*.log file has that info?

Comment 9 buddabrod 2017-01-24 14:01:18 UTC
I downgraded
nss nss-softokn nss-softokn-freebl nss-sysinit nss-tools nss-util
each from 3.28.1 to 3.27.0.

I did not test the individual packages, just downgraded all of them.

Comment 10 buddabrod 2017-01-24 14:04:32 UTC
Here is a paste of the relevant snippet (I think)

https://nopaste.me/view/5cfe6b84

Comment 11 Gerard Ryan 2017-01-24 14:33:03 UTC
This appears to be the update that the nss packages came from: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e42b513012

Downgrading everything from that update seems to be the most rational thing to do, and it is a working workaround for me.

Comment 12 Severin Gehwolf 2017-01-24 15:39:25 UTC
Pasting content of paste from comment 10 here directly, since the paste might go away:

Jan 24 13:59:19 INFO Letzte Prüfung auf abgelaufene Metadaten: vor 0:34:02 am Tue Jan 24 13:25:16 2017.
Jan 24 13:59:19 DEBUG --> Abhängigkeitsauflösung wird gestartet
Jan 24 13:59:20 DEBUG ---> Paket nss.i686 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss.i686 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss.x86_64 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss.x86_64 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.i686 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.i686 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.x86_64 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.x86_64 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.i686 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.i686 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.x86_64 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.x86_64 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-sysinit.x86_64 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-sysinit.x86_64 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-tools.x86_64 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-tools.x86_64 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-util.i686 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-util.i686 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-util.x86_64 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-util.x86_64 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG --> Abhängigkeitsauflösung wurde abgeschlossen
Jan 24 13:59:20 DDEBUG timer: depsolve: 851 ms

Comment 13 Kai Engert (:kaie) (inactive account) 2017-01-24 16:04:54 UTC
I can reproduce the problem, but don't understand the cause yet.

It's sufficient to upgrade the
  nss-util + nss-softokn + nss-softokn-freebl
packages.

(The nss.rpm packages doesn't have an influence).

I discovered another bug while working on this.
Please make sure you NEVER downgrade ONLY nss-softokn. 
Always downgrade BOTH of nss-softokn and nss-softokn-freebl.

(When I accidently downgraded just one of them, which dnf allowed me to do without overrides, that broken dnf completely, by no longer being able to use https. We need to file another bug that ensures these packages are always used together, I guess, or find out what's causing that breakage.)

Back to this original bug report.

With the attached differences, and Hubert's detective skills, the following difference in the 
  /usr/lib/jvm/java-*/jre/lib/security/java.security
file triggers (or prevents the problem).

Broken:
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768

Working:
jdk.tls.disabledAlgorithms=SSLv3, DH keySize < 768, EC, ECDHE, ECDH

Comment 14 Kai Engert (:kaie) (inactive account) 2017-01-24 16:08:37 UTC
This line works, too:
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, ECDHE      

Allowing ECDHE triggers the problem.

Comment 15 David Burke 2017-01-24 16:11:02 UTC
This breaks android dev tooling too. Based on buddabrod's comment here is a copy and paste workaround for now that downgrades nss related packages.

sudo dnf install nss-3.27.0-1.1.fc25.x86_64 nss-softokn-3.27.0-1.0.fc25.x86_64 nss-softokn-freebl-3.27.0-1.0.fc25.x86_64 nss-sysinit-3.27.0-1.1.fc25.x86_64 nss-tools-3.27.0-1.1.fc25.x86_64 nss-util-3.27.0-1.0.fc25.x86_64

Comment 16 Severin Gehwolf 2017-01-24 16:14:27 UTC
(In reply to Kai Engert (:kaie) from comment #14)
> This line works, too:
> jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, ECDHE  
> 
> 
> Allowing ECDHE triggers the problem.

That makes some sense as OpenJDK packages in Fedora gain EC support via nss.

Comment 17 Kai Engert (:kaie) (inactive account) 2017-01-24 16:16:04 UTC
It would be difficult to downgrade nss in fedora-updates.

If you can confirm that adding ECDHE to the disabled algorithms in java.security fixes the issue, would it be an option to publish an updated java-openjdk.rpm package, which disables ECDHE, to give us more time to investigate the issue?

Comment 18 Silas Parker 2017-01-24 16:23:30 UTC
This broke Eclipse and Davmail for me, downgrading nss fixed the issue.

Comment 19 jiri vanek 2017-01-24 16:28:11 UTC
 I can disable ECDHE by rpm patch, but isn't better to permanently downgrade nss? Aka to create upgrade with older, working version inside.

Comment 20 Kai Engert (:kaie) (inactive account) 2017-01-24 17:02:39 UTC
We just discussed this in our weekly meeting.

It seems that you are trying to enable EC in Java at the same time, as NSS adds an additional curve to NSS.

That new curve x25519 requires some new special code for handling it correctly. We suspect that the Java code is broken, and doesn't handle this new curve correctly, and that this situation is actually a bug in the OpenJDK package.

Also, downgrading NSS would cause other negative consequences. Firefox 51 (with new security fixes) was released today, and Firefox requires NSS 3.28.1

Given the complex situation, we'd like to ask you to please please please :-) submit the java package update, with EC* disabled, as a temporary fix, while we all try to better understand the situation.

In parallel, I will attempt to change the nss-softokn package, to disable this new curve, and test locally if it works around the issue. If that works, I will submit that as an update, too, just in case there's any other package could potentially be negatively affected by this new curve.

Is this suggestion acceptable, could you disable EC in a openjdk update?

Comment 22 jiri vanek 2017-01-24 17:06:53 UTC
> Given the complex situation, we'd like to ask you to please please please
> :-) submit the java package update, with EC* disabled, as a temporary fix,
> while we all try to better understand the situation.
> 
...
> 
> Is this suggestion acceptable, could you disable EC in a openjdk update?

I lunched the builds before this comment. Is there anything else to disable except already excluded ECDHE ?

Comment 23 Kai Engert (:kaie) (inactive account) 2017-01-24 17:08:39 UTC
I don't know. Disabling ECDHE was sufficient to work around this specific bug. We don't know if anything else is broken. If you want to play it completely safe, you could disable all the new EC code. But it's up to you, we can try to disable ECDHE, only, since no other breakage is known yet.

Comment 24 jiri vanek 2017-01-24 17:14:44 UTC
(In reply to Kai Engert (:kaie) from comment #23)
> I don't know. Disabling ECDHE was sufficient to work around this specific
> bug. We don't know if anything else is broken. If you want to play it
> completely safe, you could disable all the new EC code. But it's up to you,
> we can try to disable ECDHE, only, since no other breakage is known yet.

ok, but only because you say so nicely;)

> Given the complex situation, we'd like to ask you to please please please
> :-) submit the java package update, with EC* disabled, as a temporary fix,

Comment 25 Christian Dersch 2017-01-24 17:36:38 UTC
(In reply to Kai Engert (:kaie) from comment #17)
> It would be difficult to downgrade nss in fedora-updates.
> 

Just a result of "this update has to be pushed on monday" instead of waiting for some days of testing as suggested multiple times…

Comment 26 Cesar Eduardo Barros 2017-01-24 18:14:38 UTC
I investigated what I think is the same issue at bug #1411116, and found one change added by the Curve25519 code which might explain this issue (see my comments at bug #1411116 for the details): the ec_NewKey function (and others) now depends on a new pointSize field in the ecParams structure, and the Java code seems to create the ecParams by hand, thus missing the new field, which then contains garbage.

If my supposition is correct, just disabling the new curve in NSS wouldn't be enough to fix this issue, since the change affects all curves. The solution would be in the Java side, to correctly set the new field (and so it would have to depend on the new NSS which has that field).

Comment 27 Kai Engert (:kaie) (inactive account) 2017-01-24 18:58:13 UTC
Cesar, thanks a lot for your analysis, which sounds like a bug in the java openjdk code.

In a local experiment, my attempt to disable that curve in 3.28 nss-softokn indeed didn't fix the issue.

Comment 28 Andrew John Hughes 2017-01-24 20:42:53 UTC
Jiri, has OpenJDK been rebuilt against the new NSS?

This issue with NSS 3.28 has also been brought up on Gentoo and rebuilding seems to fix it:

https://bugs.gentoo.org/show_bug.cgi?id=605430

The OpenJDK build has to statically link some parts of NSS because that's the only way they are exposed. Hence, when NSS is upgraded, OpenJDK needs to be re-built against it.

At present, the dependency in java-1.8.0-openjdk is >= %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is re-built against a newer NSS, but I really think it should be switched to '=' to prevent cases like this. The NSS upgrade would then be blocked until OpenJDK is updated and it would be more obvious what the bug is.

Comment 29 Cesar Eduardo Barros 2017-01-24 21:38:27 UTC
(In reply to Andrew John Hughes from comment #28)
> Jiri, has OpenJDK been rebuilt against the new NSS?
> 
> This issue with NSS 3.28 has also been brought up on Gentoo and rebuilding
> seems to fix it:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=605430
> 
> The OpenJDK build has to statically link some parts of NSS because that's
> the only way they are exposed. Hence, when NSS is upgraded, OpenJDK needs to
> be re-built against it.
> 
> At present, the dependency in java-1.8.0-openjdk is >=
> %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> re-built against a newer NSS, but I really think it should be switched to
> '=' to prevent cases like this. The NSS upgrade would then be blocked until
> OpenJDK is updated and it would be more obvious what the bug is.

Does it fully fix the problem, or does it just make it not crash while still not working correctly for ECDH?

Apparently, the Java code does a calloc of the ECParams structure. With a java-1.8.0-openjdk compiled against an older NSS, the structure is 4 bytes shorter than it should be for NSS 3.28, and the field pointSize added by NSS 3.28 is an out-of-bounds read; once java-1.8.0-openjdk is recompiled with the headers for NSS 3.28, the structure should be allocated with the correct size, and the pointSize field should be always zero. Neither is the correct value for the pointSize field, and I have no idea how it would behave when it's zero.

Even then, IMO it's still worth it to recompile java-1.8.0-openjdk against the NSS 3.28 headers, just to prevent the out-of-bounds read.

Comment 30 Andrew John Hughes 2017-01-24 22:42:43 UTC
(In reply to Cesar Eduardo Barros from comment #29)
> (In reply to Andrew John Hughes from comment #28)
> > Jiri, has OpenJDK been rebuilt against the new NSS?
> > 
> > This issue with NSS 3.28 has also been brought up on Gentoo and rebuilding
> > seems to fix it:
> > 
> > https://bugs.gentoo.org/show_bug.cgi?id=605430
> > 
> > The OpenJDK build has to statically link some parts of NSS because that's
> > the only way they are exposed. Hence, when NSS is upgraded, OpenJDK needs to
> > be re-built against it.
> > 
> > At present, the dependency in java-1.8.0-openjdk is >=
> > %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> > re-built against a newer NSS, but I really think it should be switched to
> > '=' to prevent cases like this. The NSS upgrade would then be blocked until
> > OpenJDK is updated and it would be more obvious what the bug is.
> 
> Does it fully fix the problem, or does it just make it not crash while still
> not working correctly for ECDH?
> 
> Apparently, the Java code does a calloc of the ECParams structure. With a
> java-1.8.0-openjdk compiled against an older NSS, the structure is 4 bytes
> shorter than it should be for NSS 3.28, and the field pointSize added by NSS
> 3.28 is an out-of-bounds read; once java-1.8.0-openjdk is recompiled with
> the headers for NSS 3.28, the structure should be allocated with the correct
> size, and the pointSize field should be always zero. Neither is the correct
> value for the pointSize field, and I have no idea how it would behave when
> it's zero.
> 
> Even then, IMO it's still worth it to recompile java-1.8.0-openjdk against
> the NSS 3.28 headers, just to prevent the out-of-bounds read.

Probably not from what you're saying, but it certainly won't work with mixing NSS 3.28 with whatever older version OpenJDK was compiled against. There's a security update due to trickle through anyway, so it should happen automatically with that. The build runs an ECDSA check, so it could even catch the failure there. If it does fail, then the right thing to do would be to disable it temporarily.

I'll look at a fix for the code itself when I have time. It's going to need to work with both versions of NSS. It's great to see NSS adopting ed25519. I do wonder if OpenJDK could also make use of it, but it might be difficult if it requires new fields, as that would require alterations to code that is part of the Java specification.

Comment 31 Andrew John Hughes 2017-01-24 22:43:35 UTC
*** Bug 1411116 has been marked as a duplicate of this bug. ***

Comment 32 Juraci Paixão Kröhling 2017-01-25 08:47:29 UTC
This bug affects pretty much all Gradle and Maven builds consuming artifacts from Maven Central via SSL. As others mentioned, reverting the NSS packages to the previous version is a workaround:

> sudo dnf downgrade nss nss-softokn nss-softokn-freebl nss-sysinit nss-tools nss-util

Comment 33 jiri vanek 2017-01-25 09:21:16 UTC
> At present, the dependency in java-1.8.0-openjdk is >=
> %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> re-built against a newer NSS, but I really think it should be switched to
> '=' to prevent cases like this. The NSS upgrade would then be blocked until
> OpenJDK is updated and it would be more obvious what the bug is.

That would be terrible lock without precedents I think.
Also it may kill many automated services dealng with openjdk  rpm.
NSS_BUILDTIME_NUMBER contans release. So that enfroces >=
If NSS_BUILDTIME_NUMBER would be onl version, it  *may* work... Something  like >= NSS_BUILDTIME_NUMBER && <= NSS_BUILDTIME_VERSIO.99999

Will try to find something.

I will post NSS versions used during builds in next comment

Comment 34 jiri vanek 2017-01-25 09:36:25 UTC
latest builds
java-1.8.0-openjdk-1.8.0.111-3.b16.fc25
          nss x86_64  3.27.0-1.1.fc25
java-1.8.0-openjdk-1.8.0.111-4.b16.fc25
          nss x86_64  3.27.0-1.2.fc25
java-1.8.0-openjdk-1.8.0.111-5.b16.fc25
          nss x86_64  3.27.0-1.3.fc25 
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 (current security update)
           nss x86_64  3.28.1-1.3.fc25 
java-1.8.0-openjdk-1.8.0.121-2.b14.fc25 (^ with disabled ECDHE)
          nss x86_64  3.28.1-1.3.fc25 

So if Your susipicions are correct, it is worthy to try java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 first. If it passes, I will revert  https://bugzilla.redhat.com/show_bug.cgi?id=1415137#c21

Comment 35 Severin Gehwolf 2017-01-25 10:10:03 UTC
(In reply to jiri vanek from comment #34)
> latest builds
[...]
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 (current security update)
>            nss x86_64  3.28.1-1.3.fc25 
[...]
> So if Your susipicions are correct, it is worthy to try
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 first. If it passes, I will revert 
> https://bugzilla.redhat.com/show_bug.cgi?id=1415137#c21

$ rpm -qa | grep java-1.8.0-openjdk
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25.x86_64
java-1.8.0-openjdk-devel-1.8.0.121-1.b14.fc25.x86_64
java-1.8.0-openjdk-headless-1.8.0.121-1.b14.fc25.x86_64
$ rpm -q --requires java-1.8.0-openjdk-headless | grep nss
libnss3.so()(64bit)
libnss3.so(NSS_3.2)(64bit)
libnssutil3.so()(64bit)
libnssutil3.so(NSSUTIL_3.12)(64bit)
nss(x86-64) >= 3.28.1
nss-softokn(x86-64) >= 3.28.1

With that I don't get exceptions/coredumps using the reproducer from comment 0. I suggest to revert java-1.8.0-openjdk-1.8.0.121-2.b14.fc25. It's not needed.

Also from the build.log from java-1.8.0-openjdk-1.8.0.121-1.b14.fc25:

+ /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.121-1.b14.fc25.x86_64/openjdk/build/jdk8.build/images/j2sdk-image/bin/javac -d . /builddir/build/SOURCES/TestECDSA.java
++ sed 's|\.java||'
+++ basename /builddir/build/SOURCES/TestECDSA.java
++ echo TestECDSA.java
+ /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.121-1.b14.fc25.x86_64/openjdk/build/jdk8.build/images/j2sdk-image/bin/java TestECDSA
Signature: 3045022055e475550f489a02b651c5ef66b06d1e51134c923353a997d8cb4694a01413be022100fea86df4d0cb4fa5b0b12b90936e079ca1b4b2cba41d101a6dc0c2bd0134dcde
Test passed.

The test is in [1]. So the rebuild fixed the issue as far as I can tell.

How can we avoid this in future? Kai, would it be possible for you to loop us in when new nss versions get introduced? That would allow us to build OpenJDK against it, we'd be part of the nss update and this should no longer be an issue. Thoughts?

[1] http://pkgs.fedoraproject.org/cgit/rpms/java-1.8.0-openjdk.git/tree/TestECDSA.java?h=f25

Comment 36 Cesar Eduardo Barros 2017-01-25 10:22:03 UTC
(In reply to jiri vanek from comment #34)
> latest builds
> java-1.8.0-openjdk-1.8.0.111-3.b16.fc25
>           nss x86_64  3.27.0-1.1.fc25
> java-1.8.0-openjdk-1.8.0.111-4.b16.fc25
>           nss x86_64  3.27.0-1.2.fc25
> java-1.8.0-openjdk-1.8.0.111-5.b16.fc25
>           nss x86_64  3.27.0-1.3.fc25 
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 (current security update)
>            nss x86_64  3.28.1-1.3.fc25 
> java-1.8.0-openjdk-1.8.0.121-2.b14.fc25 (^ with disabled ECDHE)
>           nss x86_64  3.28.1-1.3.fc25 
> 
> So if Your susipicions are correct, it is worthy to try
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 first. If it passes, I will revert 
> https://bugzilla.redhat.com/show_bug.cgi?id=1415137#c21

I installed java-1.8.0-openjdk-1.8.0.121-1.b14.fc24.x86_64 from koji (note: fc24, not fc25). The reproducer in comment #0, Maven, and IntelliJ IDEA, now all work without crashing; Maven even downloaded something using https. I haven't verified that ECDH works, but it's already much better.

Comment 37 Fedora Update System 2017-01-25 10:54:47 UTC
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4076cf8494

Comment 38 Fedora Update System 2017-01-25 10:58:12 UTC
java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-2176221cd2

Comment 39 jiri vanek 2017-01-25 10:58:54 UTC
> With that I don't get exceptions/coredumps using the reproducer from comment
> 0. I suggest to revert java-1.8.0-openjdk-1.8.0.121-2.b14.fc25. It's not
> needed.
> 
> 
> [1]
> http://pkgs.fedoraproject.org/cgit/rpms/java-1.8.0-openjdk.git/tree/
> TestECDSA.java?h=f25


Reverted. Bumped release in rawhide, creating updates with CPU with 121-1 for f24+f25

Comment 40 Roland Grunberg 2017-01-25 17:40:20 UTC
*** Bug 1416121 has been marked as a duplicate of this bug. ***

Comment 41 Fedora Update System 2017-01-25 23:19:00 UTC
java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-2176221cd2

Comment 42 Severin Gehwolf 2017-01-26 08:55:23 UTC
*** Bug 1416574 has been marked as a duplicate of this bug. ***

Comment 43 Severin Gehwolf 2017-01-26 09:26:13 UTC
*** Bug 1416525 has been marked as a duplicate of this bug. ***

Comment 44 jiri vanek 2017-01-26 10:06:17 UTC
(In reply to jiri vanek from comment #33)
> > At present, the dependency in java-1.8.0-openjdk is >=
> > %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> > re-built against a newer NSS, but I really think it should be switched to
> > '=' to prevent cases like this. The NSS upgrade would then be blocked until
> > OpenJDK is updated and it would be more obvious what the bug is.
> 
> That would be terrible lock without precedents I think.
> Also it may kill many automated services dealng with openjdk  rpm.
> NSS_BUILDTIME_NUMBER contans release. So that enfroces >=
> If NSS_BUILDTIME_NUMBER would be onl version, it  *may* work... Something 
> like >= NSS_BUILDTIME_NUMBER && <= NSS_BUILDTIME_VERSIO.99999
> 
> Will try to find something.
> 
> I will post NSS versions used during builds in next comment

In addition, this will bound us to rebuild with every nss update. Although it sounds like a good idea from point of view of this bug, it is quite an effort to all nss maintainer, jdk maintainer, koji, bodhi and even end user.

Thoughts?

Comment 45 Andrew John Hughes 2017-01-26 10:55:00 UTC
(In reply to jiri vanek from comment #44)
> (In reply to jiri vanek from comment #33)
> > > At present, the dependency in java-1.8.0-openjdk is >=
> > > %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> > > re-built against a newer NSS, but I really think it should be switched to
> > > '=' to prevent cases like this. The NSS upgrade would then be blocked until
> > > OpenJDK is updated and it would be more obvious what the bug is.
> > 
> > That would be terrible lock without precedents I think.
> > Also it may kill many automated services dealng with openjdk  rpm.
> > NSS_BUILDTIME_NUMBER contans release. So that enfroces >=
> > If NSS_BUILDTIME_NUMBER would be onl version, it  *may* work... Something 
> > like >= NSS_BUILDTIME_NUMBER && <= NSS_BUILDTIME_VERSIO.99999
> > 
> > Will try to find something.
> > 
> > I will post NSS versions used during builds in next comment
> 
> In addition, this will bound us to rebuild with every nss update. Although
> it sounds like a good idea from point of view of this bug, it is quite an
> effort to all nss maintainer, jdk maintainer, koji, bodhi and even end user.
> 
> Thoughts?

It would, but on the positive side, it avoids bugs like this and makes it clear to users what the problem is immediately; they won't be able to update to the new NSS because of the OpenJDK requirement.

Comment 46 Andrew John Hughes 2017-01-26 10:57:09 UTC
How complex can dependencies be? Can you say something like e.g. >= 3.28.0 but < 3.29.0, so we'd only rebuild on minor number bumps, not micro?

Comment 47 Hubert Kario 2017-01-26 12:17:55 UTC
ABI stability of internal structures in NSS is not guaranteed at all, they can change even because we went from 3.28.0-1 to 3.28.0-2, as a result of a backported patch to fix a security issue

Comment 48 jiri vanek 2017-01-26 12:20:07 UTC
(In reply to Hubert Kario from comment #47)
> ABI stability of internal structures in NSS is not guaranteed at all, they
> can change even because we went from 3.28.0-1 to 3.28.0-2, as a result of a
> backported patch to fix a security issue

Thanx. Tahts pretty killing the idea.

Comment 49 Kai Engert (:kaie) (inactive account) 2017-01-26 13:19:24 UTC
The correct solution is to no longer use private interface of NSS.

NSS has external interfaces, and makes compatibility and stability promises for them. You can find them in the .def files.

Instead of using freebl directly, you should go through the interfaces that nss-softokn exports.

We should research, how you can achieve your technical requirements with the supported exported interfaces, and in case something is missing, add what's necessary.

I would like to suggest to start a discussion on this mailing list, which is the primary public place for discussing NSS:
  https://lists.mozilla.org/listinfo/dev-tech-crypto

Could you please subscribe to the list, and send a message, describing what you need to do, how you currently do it, and ask if there is a way to to it using exported interfaces?


(The package dependency issue should be avoided. If you require an "equal" package version, you will prevent everyone from being able to upgrade to a newer Firefox, until you have been able to adjust the openjdk package. This would be unfortunate. For all the other packages that use the exported interfaces of NSS, only, the usual requirement of a minimum NSS version is sufficient.)

Comment 50 Kai Engert (:kaie) (inactive account) 2017-01-26 14:43:54 UTC
see also:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1334108

Comment 51 Justin Venus 2017-01-26 16:54:45 UTC
*** Bug 1416561 has been marked as a duplicate of this bug. ***

Comment 52 Donald O. 2017-01-26 17:01:06 UTC
(In reply to Mikolaj Izdebski from comment #3)
> Created attachment 1242727 [details]
> working java.security file (no segfault)

it seems to me that it works. Hope it lasts. Thank you very much Mikolaj and all the other great people helping me.

Comment 53 Severin Gehwolf 2017-01-27 11:37:51 UTC
*** Bug 1417092 has been marked as a duplicate of this bug. ***

Comment 54 Julien HENRY 2017-01-27 13:51:35 UTC
(In reply to Fedora Update System from comment #41)
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24
> testing repository.

Why only F24? I'm on F25 and also affected. I would have been more than happy to test the fix.

Comment 55 Severin Gehwolf 2017-01-27 14:02:52 UTC
(In reply to Julien HENRY from comment #54)
> (In reply to Fedora Update System from comment #41)
> > java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24
> > testing repository.
> 
> Why only F24? I'm on F25 and also affected. I would have been more than
> happy to test the fix.

See comment 37. An update has been submitted, but it's currently stuck for reasons which are beyond me. I've filed this ticket to get this resolved:
https://pagure.io/fedora-infrastructure/issue/5726

For the time being, please download builds from koji for F25. Sorry for the inconvenience.

Comment 56 Fedora Update System 2017-01-27 19:18:36 UTC
java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 57 Omair Majid 2017-01-27 21:20:20 UTC
*** Bug 1417295 has been marked as a duplicate of this bug. ***

Comment 58 John Gotts 2017-01-27 22:04:59 UTC
I can confirm that version 1.8.0.121-1.b14.fc25 fixes azureus. Thanks!

Comment 59 Fedora Update System 2017-01-28 04:53:05 UTC
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4076cf8494

Comment 60 cube00 2017-01-28 05:01:17 UTC
*** Bug 1417350 has been marked as a duplicate of this bug. ***

Comment 61 Andrew John Hughes 2017-01-28 20:03:14 UTC
*** Bug 1417317 has been marked as a duplicate of this bug. ***

Comment 62 Fedora Update System 2017-01-29 00:21:21 UTC
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 63 Piergiorgio Sartor 2017-01-29 15:11:43 UTC
*** Bug 1416173 has been marked as a duplicate of this bug. ***

Comment 64 Severin Gehwolf 2017-01-30 13:56:27 UTC
*** Bug 1417544 has been marked as a duplicate of this bug. ***

Comment 65 Kai Engert (:kaie) (inactive account) 2017-02-13 17:47:47 UTC
This is a follow up to my comment 49 and comment 50.

You actually didn't make a mistake, the mistake was made by upstream NSS.

The structure ECParams, which the openjdk code uses, was extended in version NSS 3.28, but it shouldn't have been extended.

We're trying to get this change revert up stream, and release a fixed version of NSS, with the original change revert.

Bob Relyea, who works on the fix for upstream NSS, is asking for testing, and feedback if the intended patch fixes the issue.


I suggest we produce a scratch build of NSS 3.28.x that contains the upstream candidate fix, and test that with the older openjdk builds that were broken.

Would you be able to help with such testing?

Comment 66 Andrew John Hughes 2017-02-14 05:05:36 UTC
(In reply to Kai Engert (:kaie) from comment #65)
> This is a follow up to my comment 49 and comment 50.
> 
> You actually didn't make a mistake, the mistake was made by upstream NSS.
> 

I never thought we did.

> The structure ECParams, which the openjdk code uses, was extended in version
> NSS 3.28, but it shouldn't have been extended.
> 
> We're trying to get this change revert up stream, and release a fixed
> version of NSS, with the original change revert.
> 
> Bob Relyea, who works on the fix for upstream NSS, is asking for testing,
> and feedback if the intended patch fixes the issue.
> 
> 
> I suggest we produce a scratch build of NSS 3.28.x that contains the
> upstream candidate fix, and test that with the older openjdk builds that
> were broken.
> 
> Would you be able to help with such testing?

Just rebuild java-1.8.0-openjdk against the new NSS update. We don't want the old OpenJDK builds, as they don't include the latest security fixes.

Comment 67 Kai Engert (:kaie) (inactive account) 2017-02-14 14:26:51 UTC
(In reply to Andrew John Hughes from comment #66)
> 
> Just rebuild java-1.8.0-openjdk against the new NSS update. We don't want
> the old OpenJDK builds, as they don't include the latest security fixes.


We're trying to ensure (and test), that the most recent NSS will restore binary compatibility with old builds, that's why I'd like to test with the older Java build (which was broken initially when used with NSS 3.28.x)

In the meantime I've done some tests.


I've locally built NSS with the candidate upstream NSS fix.

Then I've installed the java package that was initially reported in this bug, version java-1.8.0-openjdk-1.8.0.111-5.b16.fc25.x86_64
  https://koji.fedoraproject.org/koji/buildinfo?buildID=832422

The test case succeeds.


Then I've upgraded java to the latest released package for f25, version 1.8.0.121-1.b14.fc25

I've confirmed that, as expected, the test case fails when using the experimental fixed NSS version.


Then I create a local build of the most recent checked in java package, which is java-1.8.0-openjdk-1.8.0.121-3.b14

I've confirmed this build passes the test case.


We'll wait for NSS upstream to accept the fix and release it. Then we should rebuild java again, and use a Conflicts: statement, to declare it incompatible with any NSS that's older (older than the upcoming version number).

I'll coordinate with you by email, or in a separate bug, to get that executed. We should then submit a combined Fedora update, which contains both the NSS update and the new java build. After that, we should be back at binary compatibility between NSS versions.

Comment 68 jiri vanek 2017-02-28 08:34:40 UTC
build against fixed nss in rawhide: https://koji.fedoraproject.org/koji/buildinfo?buildID=861016

hopefully fixed build for f25: https://koji.fedoraproject.org/koji/buildinfo?buildID=862733

and freshly launched f24: https://koji.fedoraproject.org/koji/taskinfo?taskID=18105734

Comment 69 Fedora Update System 2017-02-28 17:56:41 UTC
java-1.8.0-openjdk-1.8.0.121-8.b14.fc25 nss-3.28.3-1.0.fc25 nss-softokn-3.28.3-1.1.fc25 nss-util-3.28.3-1.0.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f66b383735

Comment 70 Fedora Update System 2017-02-28 17:57:04 UTC
java-1.8.0-openjdk-1.8.0.121-8.b14.fc25 nss-3.28.3-1.0.fc25 nss-softokn-3.28.3-1.1.fc25 nss-util-3.28.3-1.0.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f66b383735

Comment 71 Fedora Update System 2017-03-01 02:54:08 UTC
java-1.8.0-openjdk-1.8.0.121-8.b14.fc25, nss-3.28.3-1.0.fc25, nss-softokn-3.28.3-1.1.fc25, nss-util-3.28.3-1.0.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f66b383735

Comment 72 Fedora Update System 2017-03-01 09:14:17 UTC
nss-3.28.3-1.0.fc24 nss-softokn-3.28.3-1.1.fc24 nss-util-3.28.3-1.0.fc24 java-1.8.0-openjdk-1.8.0.121-8.b14.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f22b7e0c19

Comment 73 Fedora Update System 2017-03-02 02:52:25 UTC
java-1.8.0-openjdk-1.8.0.121-8.b14.fc24, nss-3.28.3-1.0.fc24, nss-softokn-3.28.3-1.1.fc24, nss-util-3.28.3-1.0.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f22b7e0c19

Comment 74 Fedora Update System 2017-03-03 21:18:34 UTC
java-1.8.0-openjdk-1.8.0.121-8.b14.fc24, nss-3.28.3-1.0.fc24, nss-softokn-3.28.3-1.1.fc24, nss-util-3.28.3-1.0.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 75 Fedora Update System 2017-03-03 21:49:55 UTC
java-1.8.0-openjdk-1.8.0.121-8.b14.fc25, nss-3.28.3-1.0.fc25, nss-softokn-3.28.3-1.1.fc25, nss-util-3.28.3-1.0.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.