Description of problem: I'm trying to do a maven/eclipse/tycho build and I am seeing the following error occurring repeatedly on different hosts (all running Fedora 23) and networks: [INFO] Fetching com.ibm.icu_54.1.1.v201501272100.jar from http://ftp-stud.fht-esslingen.de/pub/Mirrors/eclipse/releases/mars/201506241002/plugins/ (10.2MB of 10.88MB at 1.73MB/s) [WARNING] Some attempts to read artifact osgi.bundle,com.ibm.icu,54.1.1.v201501272100 failed: [WARNING] An error occurred while transferring artifact packed: osgi.bundle,com.ibm.icu,54.1.1.v201501272100 from repository http://download.eclipse.org/releases/mars/201506241002: [WARNING] Retry another mirror: [WARNING] Problems downloading artifact: osgi.bundle,com.ibm.icu,54.1.1.v201501272100.: [WARNING] Unpacking fails because intermediate file is empty: /tmp/work2632612192824623467/p2.optimizers.incoming1297827583558447260.jar While searching for possible reasons I only found https://issues.jboss.org/browse/JBTIS-553 which may indicate that this is a specific OpenJDK problem on Fedora though I'm not sure. Below is a simple procedure to reproduce the problem (the output from maven is not very helpful). Version-Release number of selected component (if applicable): $ rpm -qa|grep openjdk java-1.8.0-openjdk-devel-1.8.0.71-1.b15.fc23.x86_64 java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64 java-1.8.0-openjdk-headless-1.8.0.71-1.b15.fc23.x86_64 $ java -version openjdk version "1.8.0_71" OpenJDK Runtime Environment (build 1.8.0_71-b15) OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode) How reproducible: Steps to Reproduce: 1. Download a pack200 compressed jar file from Eclipse Mars that triggers the problem $ wget http://ftp-stud.fht-esslingen.de/pub/Mirrors/eclipse/releases/mars/201510021000/plugins/com.ibm.icu_54.1.1.v201501272100.jar.pack.gz 2. Check gzip compression integrity $ gunzip -vt com.ibm.icu_54.1.1.v201501272100.jar.pack.gz com.ibm.icu_54.1.1.v201501272100.jar.pack.gz: OK 3. Unpack file with unpack200 from openjdk $ unpack200 com.ibm.icu_54.1.1.v201501272100.jar.pack.gz x.jar CRC error, invalid compressed data. 4. gunzip archive file $ gunzip com.ibm.icu_54.1.1.v201501272100.jar.pack.gz 5. Unpack file with unpack200 from openjdk $ unpack200 com.ibm.icu_54.1.1.v201501272100.jar.pack y.jar (No output, no error) Actual results: CRC error in step 3. Expected results: No error in step 3 (especially, since step 4 and 5 work). Additional info: The same test succeeds in step 3 with Oracle's JDK. There is no CRC error reported. JDK version used for testing this is $ java -version java version "1.8.0_74" Java(TM) SE Runtime Environment (build 1.8.0_74-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode) I found possibly related bug reports at https://issues.jboss.org/browse/JBTIS-553 https://bugs.eclipse.org/bugs/show_bug.cgi?id=486638 In particular, https://bugs.eclipse.org/bugs/show_bug.cgi?id=486638#c4 might be helpful. I don't know whether the CRC's are broken in the original archive file or the handling of the CRC in openjdk is broken. Since the split of the decompression into two separate steps with different tools (first gunzip, then unpack200) does not show errors I am assuming the problem to be in openjdk. There are more Eclipse bundles triggering the problem. To find those: 1. Get *pack.gz files from http://download.eclipse.org/releases/mars/201506241002/plugins/ 2. for j in *.pack.gz; do x=`echo $j | sed 's/jar.pack.gz/jar/'`; echo "unpack200 ${j} ${x}"; unpack200 ${j} ${x}; done
The CRC check is from OpenJDK 9 so the first thing to check is whether it fails there too or not.
(In reply to Andrew John Hughes from comment #1) > The CRC check is from OpenJDK 9 so the first thing to check is whether it > fails there too or not. $ ~/devel/jdk9-jdk9/build/images/jdk/bin/java -version openjdk version "9-internal" OpenJDK Runtime Environment (build 9-internal+0-2016-02-16-123502.omajid.jdk9-jdk9) OpenJDK 64-Bit Server VM (build 9-internal+0-2016-02-16-123502.omajid.jdk9-jdk9, mixed mode) $ ~/devel/jdk9-jdk9/build/images/jdk/bin/unpack200 -v com.ibm.icu_54.1.1.v201501272100.jar.pack.gz com.ibm.icu_54.1.1.v201501272100.jar Unpacking from com.ibm.icu_54.1.1.v201501272100.jar.pack.gz to com.ibm.icu_54.1.1.v201501272100.jar com.sun.java.util.jar.pack.unpack.log.file=- unpack.deflate.hint=(not set) com.sun.java.util.jar.pack.unpack.remove.packfile=false com.sun.java.util.jar.pack.verbose=1 com.sun.java.util.jar.pack.unpack.modification.time=(not set) CRC error, invalid compressed data.
The version I am using: $ readlink -f $(which unpack200) /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.72-5.b15.fc24.x86_64/jre/bin/unpack200 The downloaded file, which fails to validate using unpack200: $ unpack200 ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz ./com.ibm.icu_54.1.1.v201501272100.jar CRC error, invalid compressed data. So, lets gunzip it. Then gzip it back. Twice: $ gunzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz $ gzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack $ gunzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz $ gzip ./com.ibm.icu_54.1.1.v201501272100.jar.pack And even the locally generated pack.gz file which just went through gzip/gunzip multiple times fails to validate according to unpack200. $ unpack200 ./com.ibm.icu_54.1.1.v201501272100.jar.pack.gz ./com.ibm.icu_54.1.1.v201501272100.jar CRC error, invalid compressed data. I think we should remove the backported crc-check patch from Fedora. The crc checking code seems a bit broken.
Ok. It sounds like an upstream bug also needs to be filed.
(In reply to Andrew John Hughes from comment #4) > Ok. It sounds like an upstream bug also needs to be filed. I left a comment on the original bug that added the crc check to unpack200: https://bugs.openjdk.java.net/browse/JDK-8000650
Created attachment 1129875 [details] remove backport that does crc checks The attached patch fixes the verification of jars. It removes the backport we added from JDK 9 that does CRC verification of pack.gz files. It selectively fixes a few things to allow the build to continue working in presence of format security flags. Scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=13099739
Why are you removing: 8074839, PR2462: Resolve disabled warnings for libunpack and the unpack200 binary It should be possible to re-generate this patch against 8 without the CRC patch applied, rather than introducing a new separate patch.
(In reply to Andrew John Hughes from comment #7) > Why are you removing: > > 8074839, PR2462: Resolve disabled warnings for libunpack and the unpack200 > binary I want to minimize changes from jdk8u upstream. > It should be possible to re-generate this patch against 8 without the CRC > patch applied, rather than introducing a new separate patch. Indeed. It seemed easier to just fix code that was triggering the warnings rather than rebasing the entire patch.
(In reply to Omair Majid from comment #8) > (In reply to Andrew John Hughes from comment #7) > > Why are you removing: > > > > 8074839, PR2462: Resolve disabled warnings for libunpack and the unpack200 > > binary > > I want to minimize changes from jdk8u upstream. > My plan is to submit that backport for 8u. I'm not going to submit a completely new patch instead.
(In reply to Andrew John Hughes from comment #9) > My plan is to submit that backport for 8u. I'm not going to submit a > completely new patch instead. Ah. In that case, I will try and come up with a more exact backport.
Use what you have now for Fedora. We can just remove it when it goes upstream.
(In reply to Andrew John Hughes from comment #11) > Use what you have now for Fedora. We can just remove it when it goes > upstream. FWIW, Jiri is not happy with my custom patch. He wants a proper backport too. So I will work on that.
Created attachment 1131608 [details] Direct 8u compatible backport of 8074839
Note that a patch for the original issue has been posted to the OpenJDK bug: http://cr.openjdk.java.net/~ksrini/8150469/webrev.00/jdk.changeset
(In reply to Andrew John Hughes from comment #14) > Note that a patch for the original issue has been posted to the OpenJDK bug: > > http://cr.openjdk.java.net/~ksrini/8150469/webrev.00/jdk.changeset Yes, and I can confirm it fixes this bug. Do you think it makes sense to backport that? Or - given that we have seen that this feature has bugs that are not identified upstream - we should keep this feature out of our 8 builds?
I'd go with what we have now i.e. leave it out. I only included it originally because it kept the patches clean and didn't seem that troublesome. Given it has proved untested, I'd prefer to leave it out. Thanks for confirming the patch does fix it. I can't see it posted anywhere publicly, but we should ack it if possible.
(In reply to Andrew John Hughes from comment #16) > I'd go with what we have now i.e. leave it out. I only included it > originally because it kept the patches clean and didn't seem that > troublesome. Given it has proved untested, I'd prefer to leave it out. Okay. > Thanks for confirming the patch does fix it. I can't see it posted anywhere > publicly, but we should ack it if possible. No problem. It was posted on core-libs-dev@: http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-February/039145.html. I replied confirming that it works.
Thanks. I see it now. It hadn't been posted when I checked earlier.
java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a8a484888
java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a8a484888
java-1.8.0-openjdk-1.8.0.72-9.b16.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.