Bug 1310452

Summary: OpenJDK unpack200 of gzipped files fails with CRC error
Product: [Fedora] Fedora Reporter: Mark <mark>
Component: java-1.8.0-openjdkAssignee: Omair Majid <omajid>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: ahughes, dbhole, jerboaa, jvanek, msrb, omajid
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-03 20:23:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
remove backport that does crc checks
none
Direct 8u compatible backport of 8074839 omajid: review?

Description Mark 2016-02-21 19:00:46 UTC 
Description of problem:

I'm trying to do a maven/eclipse/tycho build and I am seeing the following error occurring repeatedly on different hosts (all running Fedora 23) and networks:

[INFO] Fetching com.ibm.icu_54.1.1.v201501272100.jar from http://ftp-stud.fht-esslingen.de/pub/Mirrors/eclipse/releases/mars/201506241002/plugins/ (10.2MB of 10.88MB at 1.73MB/s)
[WARNING] Some attempts to read artifact osgi.bundle,com.ibm.icu,54.1.1.v201501272100 failed:
[WARNING]    An error occurred while transferring artifact packed: osgi.bundle,com.ibm.icu,54.1.1.v201501272100 from repository http://download.eclipse.org/releases/mars/201506241002:
[WARNING]       Retry another mirror:
[WARNING]          Problems downloading artifact: osgi.bundle,com.ibm.icu,54.1.1.v201501272100.:
[WARNING]             Unpacking fails because intermediate file is empty: /tmp/work2632612192824623467/p2.optimizers.incoming1297827583558447260.jar

While searching for possible reasons I only found https://issues.jboss.org/browse/JBTIS-553 which may indicate that this is a specific OpenJDK problem on Fedora though I'm not sure. Below is a simple procedure to reproduce the problem (the output from maven is not very helpful).

Version-Release number of selected component (if applicable):

$ rpm -qa|grep openjdk
java-1.8.0-openjdk-devel-1.8.0.71-1.b15.fc23.x86_64
java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64
java-1.8.0-openjdk-headless-1.8.0.71-1.b15.fc23.x86_64

$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)

How reproducible:

Steps to Reproduce:

1. Download a pack200 compressed jar file from Eclipse Mars that triggers the problem

$ wget http://ftp-stud.fht-esslingen.de/pub/Mirrors/eclipse/releases/mars/201510021000/plugins/com.ibm.icu_54.1.1.v201501272100.jar.pack.gz

2. Check gzip compression integrity

$ gunzip -vt com.ibm.icu_54.1.1.v201501272100.jar.pack.gz 
com.ibm.icu_54.1.1.v201501272100.jar.pack.gz:	 OK

3. Unpack file with unpack200 from openjdk

$ unpack200 com.ibm.icu_54.1.1.v201501272100.jar.pack.gz x.jar
CRC error, invalid compressed data.

4. gunzip archive file

$ gunzip com.ibm.icu_54.1.1.v201501272100.jar.pack.gz

5. Unpack file with unpack200 from openjdk

$ unpack200 com.ibm.icu_54.1.1.v201501272100.jar.pack y.jar

(No output, no error)

Actual results:

CRC error in step 3.

Expected results:

No error in step 3 (especially, since step 4 and 5 work).

Additional info:

The same test succeeds in step 3 with Oracle's JDK. There is no CRC error reported. JDK version used for testing this is

$ java -version
java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)

I found possibly related bug reports at

https://issues.jboss.org/browse/JBTIS-553
https://bugs.eclipse.org/bugs/show_bug.cgi?id=486638

In particular, https://bugs.eclipse.org/bugs/show_bug.cgi?id=486638#c4 might be helpful.

I don't know whether the CRC's are broken in the original archive file or the handling of the CRC in openjdk is broken. Since the split of the decompression into two separate steps with different tools (first gunzip, then unpack200) does not show errors I am assuming the problem to be in openjdk.

There are more Eclipse bundles triggering the problem. To find those:

1. Get *pack.gz files from http://download.eclipse.org/releases/mars/201506241002/plugins/
2. for j in *.pack.gz; do x=`echo $j | sed 's/jar.pack.gz/jar/'`; echo "unpack200 ${j} ${x}"; unpack200 ${j} ${x}; done
Comment 1 Andrew John Hughes 2016-02-22 18:20:13 UTC 
The CRC check is from OpenJDK 9 so the first thing to check is whether it fails there too or not.
Comment 2 Omair Majid 2016-02-22 18:23:10 UTC 
(In reply to Andrew John Hughes from comment #1)
> The CRC check is from OpenJDK 9 so the first thing to check is whether it
> fails there too or not.

$ ~/devel/jdk9-jdk9/build/images/jdk/bin/java -version
openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-2016-02-16-123502.omajid.jdk9-jdk9)
OpenJDK 64-Bit Server VM (build 9-internal+0-2016-02-16-123502.omajid.jdk9-jdk9, mixed mode)

$ ~/devel/jdk9-jdk9/build/images/jdk/bin/unpack200 -v com.ibm.icu_54.1.1.v201501272100.jar.pack.gz com.ibm.icu_54.1.1.v201501272100.jar

Unpacking from com.ibm.icu_54.1.1.v201501272100.jar.pack.gz to com.ibm.icu_54.1.1.v201501272100.jar
com.sun.java.util.jar.pack.unpack.log.file=-
unpack.deflate.hint=(not set)
com.sun.java.util.jar.pack.unpack.remove.packfile=false
com.sun.java.util.jar.pack.verbose=1
com.sun.java.util.jar.pack.unpack.modification.time=(not set)
CRC error, invalid compressed data.
Comment 3 Omair Majid 2016-02-22 21:55:17 UTC 
The version I am using:

$ readlink -f $(which unpack200)
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.x86_64/jre/bin/unpack200

The downloaded file, which fails to validate using unpack200:

$ unpack200 ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz ./com.ibm.icu_54.1.1.v201501272100.jar
CRC error, invalid compressed data.

So, lets gunzip it. Then gzip it back. Twice:

$ gunzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz 
$ gzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack 
$ gunzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz 
$ gzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack 

And even the locally generated pack.gz file which just went through gzip/gunzip multiple times fails to validate according to unpack200.
  
$ unpack200 ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz ./com.ibm.icu_54.1.1.v201501272100.jar
CRC error, invalid compressed data.

I think we should remove the backported crc-check patch from Fedora. The crc checking code seems a bit broken.
Comment 4 Andrew John Hughes 2016-02-23 02:51:44 UTC 
Ok. It sounds like an upstream bug also needs to be filed.
Comment 5 Omair Majid 2016-02-23 16:06:31 UTC 
(In reply to Andrew John Hughes from comment #4)
> Ok. It sounds like an upstream bug also needs to be filed.

I left a comment on the original bug that added the crc check to unpack200:
https://bugs.openjdk.java.net/browse/JDK-8000650
Comment 6 Omair Majid 2016-02-23 17:57:37 UTC 
Created attachment 1129875 [details]
remove backport that does crc checks

The attached patch fixes the verification of jars. It removes the backport we added from JDK 9 that does CRC verification of pack.gz files. It selectively fixes a few things to allow the build to continue working in presence of format security flags.

Scratch build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=13099739
Comment 7 Andrew John Hughes 2016-02-23 18:00:37 UTC 
Why are you removing:

8074839, PR2462: Resolve disabled warnings for libunpack and the unpack200 binary

It should be possible to re-generate this patch against 8 without the CRC patch applied, rather than introducing a new separate patch.
Comment 8 Omair Majid 2016-02-23 18:09:20 UTC 
(In reply to Andrew John Hughes from comment #7)
> Why are you removing:
> 
> 8074839, PR2462: Resolve disabled warnings for libunpack and the unpack200
> binary

I want to minimize changes from jdk8u upstream.

> It should be possible to re-generate this patch against 8 without the CRC
> patch applied, rather than introducing a new separate patch.

Indeed. It seemed easier to just fix code that was triggering the warnings rather than rebasing the entire patch.
Comment 9 Andrew John Hughes 2016-02-23 18:18:58 UTC 
(In reply to Omair Majid from comment #8)
> (In reply to Andrew John Hughes from comment #7)
> > Why are you removing:
> > 
> > 8074839, PR2462: Resolve disabled warnings for libunpack and the unpack200
> > binary
> 
> I want to minimize changes from jdk8u upstream.
> 

My plan is to submit that backport for 8u. I'm not going to submit a completely new patch instead.
Comment 10 Omair Majid 2016-02-23 18:19:56 UTC 
(In reply to Andrew John Hughes from comment #9)
> My plan is to submit that backport for 8u. I'm not going to submit a
> completely new patch instead.

Ah. In that case, I will try and come up with a more exact backport.
Comment 11 Andrew John Hughes 2016-02-23 19:02:57 UTC 
Use what you have now for Fedora. We can just remove it when it goes upstream.
Comment 12 Omair Majid 2016-02-26 14:56:49 UTC 
(In reply to Andrew John Hughes from comment #11)
> Use what you have now for Fedora. We can just remove it when it goes
> upstream.

FWIW, Jiri is not happy with my custom patch. He wants a proper backport too. So I will work on that.
Comment 13 Omair Majid 2016-02-29 16:16:38 UTC 
Created attachment 1131608 [details]
Direct 8u compatible backport of 8074839
Comment 14 Andrew John Hughes 2016-02-29 16:38:44 UTC 
Note that a patch for the original issue has been posted to the OpenJDK bug:

http://cr.openjdk.java.net/~ksrini/8150469/webrev.00/jdk.changeset
Comment 15 Omair Majid 2016-02-29 16:45:38 UTC 
(In reply to Andrew John Hughes from comment #14)
> Note that a patch for the original issue has been posted to the OpenJDK bug:
> 
> http://cr.openjdk.java.net/~ksrini/8150469/webrev.00/jdk.changeset

Yes, and I can confirm it fixes this bug.

Do you think it makes sense to backport that? Or - given that we have seen that this feature has bugs that are not identified upstream - we should keep this feature out of our 8 builds?
Comment 16 Andrew John Hughes 2016-02-29 16:48:26 UTC 
I'd go with what we have now i.e. leave it out. I only included it originally because it kept the patches clean and didn't seem that troublesome. Given it has proved untested, I'd prefer to leave it out.

Thanks for confirming the patch does fix it. I can't see it posted anywhere publicly, but we should ack it if possible.
Comment 17 Omair Majid 2016-02-29 16:54:18 UTC 
(In reply to Andrew John Hughes from comment #16)
> I'd go with what we have now i.e. leave it out. I only included it
> originally because it kept the patches clean and didn't seem that
> troublesome. Given it has proved untested, I'd prefer to leave it out.

Okay.

> Thanks for confirming the patch does fix it. I can't see it posted anywhere
> publicly, but we should ack it if possible.

No problem. It was posted on core-libs-dev@: http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-February/039145.html. I replied confirming that it works.
Comment 18 Andrew John Hughes 2016-02-29 17:02:17 UTC 
Thanks. I see it now. It hadn't been posted when I checked earlier.
Comment 19 Fedora Update System 2016-03-01 17:57:49 UTC 
java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a8a484888
Comment 20 Fedora Update System 2016-03-02 12:52:21 UTC 
java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a8a484888
Comment 21 Fedora Update System 2016-03-03 20:23:36 UTC 
java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.