Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1652177 - Size of modular runtime image (jlink) is much larger than expected
Summary: Size of modular runtime image (jlink) is much larger than expected
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: java-11-openjdk
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Severin Gehwolf
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-21 15:56 UTC by Gunnar Morling
Modified: 2020-05-26 17:56 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-26 17:56:21 UTC
Type: Bug


Attachments (Terms of Use)
Quick and dirty script to repackage custom JDK images (760 bytes, text/plain)
2018-11-23 16:01 UTC, Severin Gehwolf
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github fabric8io-images s2i issues 181 0 'None' open Smaller JDK base image using Java 11 modularity (jlink) 2021-02-04 10:29:31 UTC
openjdk bug system JDK-8214796 0 None None None 2019-06-11 12:02:28 UTC

Description Gunnar Morling 2018-11-21 15:56:22 UTC
# Description of problem:

Modular runtime images as created via jlink get very huge, due to the size of the /lib/server/libjvm.so file in the produced runtime image.

# Version-Release number of selected component (if applicable):

 java-11-openjdk            x86_64  1:11.0.1.13-4.fc29   updates          197 k
 java-11-openjdk-devel      x86_64  1:11.0.1.13-4.fc29   updates          3.3 M
 java-11-openjdk-jmods      x86_64  1:11.0.1.13-4.fc29   updates          321 M

# How reproducible:

Always

# Steps to Reproduce:

Installing OpenJDK via package and via tar.gz (which shows the expected result):

yum install -y \
    java-11-openjdk \
    java-11-openjdk-devel \
    java-11-openjdk-jmods

mkdir /tmp/jdk \
     && cd /tmp/jdk \
     && curl -O https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz \
     && tar -xvf openjdk-11.0.1_linux-x64_bin.tar.gz

jlink \
    --add-modules java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported \
    --verbose --strip-debug --compress 2 --no-header-files --no-man-pages \
    --output /opt/jre-minimal-actual

/tmp/jdk/jdk-11.0.1/bin/jlink \
    --add-modules java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported \
    --verbose --strip-debug --compress 2 --no-header-files --no-man-pages \
    --output /opt/jre-minimal-expected

Actual results:

du -h /opt/jre-minimal-actual
...
711M	/opt/jre-minimal-actual

Expected results:

du -h /opt/jre-minimal-expected
...
49M	/opt/jre-minimal-expected

Additional info:

Comment 1 Severin Gehwolf 2018-11-21 16:41:31 UTC
Thanks, I'll have a look.

Comment 2 Severin Gehwolf 2018-11-22 13:20:57 UTC
I'm on F28 but this shouldn't matter. So I've built JDK 11 with two configs:

A:  with --with-native-debug-symbols=none
B:  with --with-native-debug-symbols=internal

Config A is 'no-debuginfo-jvm' below and config B is 'all-debuginfo-jvm'.

[no-debuginfo-jvm]$ ./bin/jlink  \
  --add-modules \
java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported   \
  --verbose \
  --strip-debug \
  --compress 2 \
  --no-header-files \
  --no-man-pages \
  --output no-debuginfo-jvm-image

[no-debuginfo-jvm]$ du -sh no-debuginfo-jvm-image/
48M	no-debuginfo-jvm-image/

[all-debuginfo-jvm]$ ./bin/jlink  \
  --add-modules \
java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported   \
  --verbose \
  --strip-debug \
  --compress 2 \
  --no-header-files \
  --no-man-pages \
  --output all-debuginfo-jvm-image

[all-debuginfo-jvm]$ du -sh all-debuginfo-jvm-image/
567M	all-debuginfo-jvm-image/

Doing the same on the packaged JDK 11 (which uses jmods from /usr/lib/jvm/java-11-openjdk/jmods):

$ /usr/lib/jvm/java-11-openjdk/bin/jlink \
  --add-modules \
java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported   \
  --verbose \
  --strip-debug \
  --compress 2 \
  --no-header-files \
  --no-man-pages \
  --output packaged-fedora-jdk-image

$ du -sh packaged-fedora-jdk-image/
641M	packaged-fedora-jdk-image/

So the root cause of this is related to I) native debug symbols II) jmods containing a copy of libjvm.so in java.base.jmod:

[no-debuginfo-jvm]$ unzip -l jmods/java.base.jmod | grep libjvm
warning [jmods/java.base.jmod]:  4 extra bytes at beginning or within zipfile
  (attempting to process anyway)
 21074648  11-21-2018 20:17   lib/server/libjvm.so
[no-debuginfo-jvm]$ ls -lh lib/server/libjvm.so 
-rw-rw-r--. 1 sgehwolf sgehwolf 21M Nov 21 20:17 lib/server/libjvm.so

[all-debuginfo-jvm]$ unzip -l jmods/java.base.jmod | grep libjvm
warning [jmods/java.base.jmod]:  4 extra bytes at beginning or within zipfile
  (attempting to process anyway)
544001864  11-21-2018 19:44   lib/server/libjvm.so
[all-debuginfo-jvm]$ ls -lh lib/server/libjvm.so 
-rw-rw-r--. 1 sgehwolf sgehwolf 519M Nov 21 19:44 lib/server/libjvm.so

Note however for the packaged version:

$ unzip -l /usr/lib/jvm/java-11-openjdk/jmods/java.base.jmod | grep libjvm
warning [/usr/lib/jvm/java-11-openjdk/jmods/java.base.jmod]:  4 extra bytes at beginning or within zipfile
  (attempting to process anyway)
625909840  11-01-2018 11:37   lib/server/libjvm.so
$ rpm -qa | grep java-11-openjdk | grep debuginfo
$ ls -sh /usr/lib/jvm/java-11-openjdk/lib/server/libjvm.so 
21M /usr/lib/jvm/java-11-openjdk/lib/server/libjvm.so

This discrepancy occurs because when we build for the distro /usr/lib/rpm/find-debuginfo.sh is being called during the build like this:

/usr/lib/rpm/find-debuginfo.sh -j6 --strict-build-id -m -i --build-id-seed 11.0.1.13-4.fc29 --unique-debug-suffix -11.0.1.13-4.fc29.x86_64 --unique-debug-src-base java-11-openjdk-11.0.1.13-4.fc29.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/java-11-openjdk-11.0.1.13-4.fc29.x86_64

This then extracts debuginfo from all .so files, including libjvm.so:

$ grep -rn 'extracting debug info from' java-11-openjdk-x86_64-build.log | grep libjvm
490239:extracting debug info from /builddir/build/BUILDROOT/java-11-openjdk-11.0.1.13-4.fc29.x86_64/usr/lib/jvm/java-11-openjdk-11.0.1.13-4.fc29.x86_64/lib/server/libjvm.so
490381:extracting debug info from /builddir/build/BUILDROOT/java-11-openjdk-11.0.1.13-4.fc29.x86_64/usr/lib/jvm/java-11-openjdk-11.0.1.13-4.fc29.x86_64-slowdebug/lib/server/libjvm.so

However, the copy in java.base.jmods remains hidden from find-debuginfo.sh:

$ grep -n 'extracting debug info from' java-11-openjdk-x86_64-build.log | grep jmods
<nothing>

The copy part of java.base.jmod has still full debuginfo symbols even after find-debuginfo.sh stripping. Hence, it's this large. jlink uses the copy from java.base.jmods for creating user images.

Comment 3 Severin Gehwolf 2018-11-22 13:25:20 UTC
I should note that Fedora builds use --with-native-debug-symbols=internal[1] so that we get full debug symbols in the binaries which can then be stripped by the rpm buildsystem. Obviously this fails for the create-custom-images-from-jmods use-case for packaged JDKs.

[1] https://src.fedoraproject.org/rpms/java-11-openjdk/blob/master/f/java-11-openjdk.spec#_1349

Comment 4 Gunnar Morling 2018-11-22 14:01:22 UTC
Just a thought: would it be an option to strip the debug symbols when creating modular runtime images?

jlink has a plug-in API which lets you hook into the process of image creation. There's even a plug-in which removes debug symbols (--strip-debug) but this only applies to Java byte code. So one could think about another plug-in which handles the libjvm.so file.

I've blogged about writing custom jlink plug-ins a while ago:

    http://in.relation.to/2017/12/12/exploring-jlink-plugin-api-in-java-9/

The caveat is that it currently requires some magic to inject custom plug-ins as it's not an exported API. OTOH it should be possible to add such custom plug-in into the OpenJDK build itself (if that's "allowed" to do, no idea).

It probably makes more sense to do the stripping when building the actual jmods package, but I wanted to bring up the idea in any case.

Comment 5 Severin Gehwolf 2018-11-22 14:53:55 UTC
(In reply to Gunnar Morling from comment #4)
> Just a thought: would it be an option to strip the debug symbols when
> creating modular runtime images?

That crossed my mind too when looking at --strip-debug sources. It seems the most viable option long-term as it would be entirely user-driven as to what should happen. On the other hand, would users of custom images want debuginfos? If yes, how should that work in a distro-integration-context? Is the expectation to use debuginfo-install or something else?

Other options may be:
- Strip libjvm.so that gets included into java.base.jmod at build time.
- Somehow get java.base.jmod to *not* include libjvm.so in the first place, but
  use the images libjvm.so instead.

Comment 6 Severin Gehwolf 2018-11-23 16:01:41 UTC
Created attachment 1508295 [details]
Quick and dirty script to repackage custom JDK images

Comment 7 Severin Gehwolf 2018-11-23 16:12:38 UTC
Here is a work-around:

$ /usr/lib/jvm/java-11-openjdk/bin/jlink \
  --add-modules java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported
  --verbose
  --strip-debug
  --compress 2
  --no-header-files
  --no-man-pages
  --output custom-jdk-image
$ du -sh custom-jdk-image
641M	custom-jdk-image
$ ./custom-jdk-image/bin/java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)
$ bash repack-jdk-image.sh /usr/lib/jvm/java-11-openjdk custom-jdk-image
Replacing shared libraries from JDK image 'custom-jdk-image' with shared libraries from '/usr/lib/jvm/java-11-openjdk'... Done.
$ du -sh custom-jdk-image
47M	custom-jdk-image
$ ./custom-jdk-image/bin/java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

repack-jdk-image.sh is from comment 6. This even works with distro debug-info:

$ gdb -ex 'set confirm off' -ex 'handle SIGSEGV pass nostop noprint' -ex 'set breakpoint pending on' -ex 'break JavaCalls::call' --args custom-jdk-image/bin/java -version
GNU gdb (GDB) Fedora 8.1.1-3.fc28
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from custom-jdk-image.after/bin/java...done.
Signal        Stop	Print	Pass to program	Description
SIGSEGV       No	No	Yes		Segmentation fault
Function "JavaCalls::call" not defined.
Breakpoint 1 (JavaCalls::call) pending.
(gdb) run
Starting program: /disk/openjdk/upstream-sources/openjdk-11/custom-jdk-image/bin/java -version
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.27-32.fc28.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /disk/openjdk/upstream-sources/openjdk-11/custom-jdk-image.after/bin/../lib/jli/libjli.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9d/8f5ef5a659227261e7a939614519ed17ff8bb9.debug

[...]

$ sudo dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9d/8f5ef5a659227261e7a939614519ed17ff8bb9.debug
[...]
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                           Arch                                  Version                                            Repository                                        Size
===================================================================================================================================================================================================================
Installing:
 java-11-openjdk-headless-debuginfo                                x86_64                                1:11.0.1.13-4.fc28                                 updates-debuginfo                                 78 M
Installing dependencies:
 java-11-openjdk-debuginfo                                         x86_64                                1:11.0.1.13-4.fc28                                 updates-debuginfo                                742 k
Installing weak dependencies:
 java-11-openjdk-debugsource                                       x86_64                                1:11.0.1.13-4.fc28                                 updates-debuginfo                                7.7 M

[...]


$ gdb -ex 'set confirm off' -ex 'handle SIGSEGV pass nostop noprint' -ex 'set breakpoint pending on' -ex 'break JavaCalls::call' -ex 'run' --args custom-jdk-image/bin/java -version
GNU gdb (GDB) Fedora 8.1.1-3.fc28
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from custom-jdk-image/bin/java...done.
Signal        Stop	Print	Pass to program	Description
SIGSEGV       No	No	Yes		Segmentation fault
Function "JavaCalls::call" not defined.
Breakpoint 1 (JavaCalls::call) pending.
Starting program: /disk/openjdk/upstream-sources/openjdk-11/custom-jdk-image/bin/java -version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7fc3700 (LWP 50946)]
[New Thread 0x7fffdf43f700 (LWP 50947)]
[New Thread 0x7fffdcb68700 (LWP 50948)]
[New Thread 0x7fffdca66700 (LWP 50949)]
[New Thread 0x7fffdc15c700 (LWP 50950)]
[New Thread 0x7fffc5a28700 (LWP 50951)]
[New Thread 0x7fffc50a5700 (LWP 50952)]
[Switching to Thread 0x7ffff7fc3700 (LWP 50946)]

Thread 2 "java" hit Breakpoint 1, JavaCalls::call (result=0x7ffff7fc26f0, method=..., args=0x7ffff7fc2700, __the_thread__=0x7ffff0011800)
    at /usr/src/debug/java-11-openjdk-11.0.1.13-4.fc28.x86_64/openjdk/src/hotspot/share/runtime/javaCalls.cpp:335
335	void JavaCalls::call(JavaValue* result, const methodHandle& method, JavaCallArguments* args, TRAPS) {
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.27-32.fc28.x86_64 libgcc-8.2.1-5.fc28.x86_64 libstdc++-8.2.1-5.fc28.x86_64 sssd-client-1.16.3-2.fc28.x86_64 zlib-1.2.11-8.fc28.x86_64
(gdb)

Comment 8 Severin Gehwolf 2018-11-23 16:15:03 UTC
I'm pondering the idea of casting this script into a jlink plugin.

Comment 9 Severin Gehwolf 2018-11-27 09:36:37 UTC
Here is a PR which implements the work-around as jlink plugin:
https://src.fedoraproject.org/rpms/java-11-openjdk/pull-request/23

Comment 10 Severin Gehwolf 2018-11-27 09:44:32 UTC
With a build[1] including PR23 from comment 9 it looks like this:

$ jlink --list-plugins
[...]
Plugin Name: native-libs-replace
Option: --native-libs-replace
Description: Replace native libraries from the system JDK image.
jlink uses native library copies from jmod files by default. When
using this option, native libraries from the system JDK's modules
will be used instead. This is particularly useful when the system
JDK's libraries have gone through debug symbols post-processing
while library-copies in jmod files have not.
The default system JDK is determined via property java.home. An
alternative image can be configured by setting system property:
jdk.jlink.system.image=/path/to/jdk/image
[...]

Without using the plugin:

$ jlink \
    --add-modules \
java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported \
    --verbose \
    --strip-debug \
    --compress 2 \
    --no-header-files \
    --no-man-pages \
    --output custom-image-default
$ du -sh custom-image-default/
712M	custom-image-default/

Using the plugin (--native-libs-replace switch) we have:

$ jlink \
    --native-libs-replace \
    --add-modules \
java.desktop,java.management,java.naming,java.security.jgss,java.security.sasl,java.sql,jdk.unsupported \
    --verbose \
    --strip-debug \
    --compress 2 \
    --no-header-files \
    --no-man-pages \
    --output custom-image-new
$ du -sh custom-image-new/
46M	custom-image-new
$ ./custom-image-new/bin/java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

$ rpm -qa | grep java-11-openjdk
java-11-openjdk-devel-11.0.1.13-8.fc29.x86_64
java-11-openjdk-headless-11.0.1.13-8.fc29.x86_64
java-11-openjdk-11.0.1.13-8.fc29.x86_64
java-11-openjdk-jmods-11.0.1.13-8.fc29.x86_64

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=31136016

Comment 11 Ben Cotton 2019-10-31 19:08:03 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 12 Ben Cotton 2019-11-27 22:52:16 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 13 Severin Gehwolf 2019-11-28 09:02:55 UTC
This issue has only been fixed in JDK 13 and up. Re-opening for keeping track of a potential backport (if any).

Comment 14 Ben Cotton 2020-04-30 20:40:51 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 15 Ben Cotton 2020-05-26 17:56:21 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.