Bug 1509371 - OpenJDK container detection not systemd compatible
Summary: OpenJDK container detection not systemd compatible
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: java-1.8.0-openjdk
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Severin Gehwolf
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-03 16:04 UTC by Marko Myllynen
Modified: 2020-06-03 06:51 UTC (History)
8 users (show)

Fixed In Version: java-1.8.0-openjdk-1.8.0.181-7.b13.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-26 17:46:23 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
openjdk bug system JDK-8146115 0 None None None 2017-11-30 10:54:17 UTC
openjdk bug system JDK-8190942 0 None None None 2018-04-05 15:42:40 UTC
openjdk bug system JDK-8217338 0 None None None 2019-01-17 19:31:21 UTC

Description Marko Myllynen 2017-11-03 16:04:54 UTC
Description of problem:
With a unit file like:

[Unit]
Description=javatest

[Service]
Type=oneshot
ExecStart=/usr/bin/java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm
-version

MemoryAccounting=true
MemoryLimit=100M

the cgroup memory limit is not detected:

Nov 02 15:25:19 f27 systemd[1]: Reloading.
Nov 02 15:25:23 f27 systemd[1]: Started javatest.
Nov 02 15:25:23 f27 java[6657]: VM settings:
Nov 02 15:25:23 f27 java[6657]:     Max. Heap Size (Estimated): 1.95G
Nov 02 15:25:23 f27 java[6657]:     Using VM: OpenJDK 64-Bit Server VM
Nov 02 15:25:23 f27 java[6657]: openjdk version "9"
Nov 02 15:25:23 f27 java[6657]: OpenJDK Runtime Environment (build 9+181)
Nov 02 15:25:23 f27 java[6657]: OpenJDK 64-Bit Server VM (build 9+181, mixed mode)

It might be helpful in some scenarios if cgroup memory limit detection would work with systemd set cgroup memory limits as well.

Version-Release number of selected component (if applicable):
java-9-openjdk-9.0.0.181-7.fc27.x86_64

Comment 1 Christine Flood 2017-11-03 20:54:07 UTC
This is a duplicate of:

https://bugzilla.redhat.com/show_bug.cgi?id=1463098

The command line option was put in as a temporary fix for docker.  It wasn't aimed to fix the problem for everyone.  The long term universal solution is a harder problem and is expected to come from: 

http://openjdk.java.net/jeps/8182070
Container aware Java

Comment 2 Deepak Bhole 2017-11-03 21:19:40 UTC
Closing as deferred. Please use upstream bug for checking progress:
https://bugs.openjdk.java.net/browse/JDK-8182070

Comment 3 Marko Myllynen 2017-11-06 11:41:25 UTC
I'm reopening this Fedora / OpenJDK 9 BZ on the grounds that https://bugzilla.redhat.com/show_bug.cgi?id=1463098 is a non-public BZ about RHEL 7 / OpenJDK 8 and https://bugs.openjdk.java.net/browse/JDK-8182070 specifically says [1] that it's about Docker only while here we specifically talk about non-Docker. Hope this is ok. Thanks.

1) "It is not a goal of this JEP to support any platform other than Docker container technology running on Linux x64."

Comment 4 Deepak Bhole 2017-11-06 22:29:12 UTC
(In reply to Marko Myllynen from comment #3)
> I'm reopening this Fedora / OpenJDK 9 BZ on the grounds that
> https://bugzilla.redhat.com/show_bug.cgi?id=1463098 is a non-public BZ about
> RHEL 7 / OpenJDK 8 and https://bugs.openjdk.java.net/browse/JDK-8182070
> specifically says [1] that it's about Docker only while here we specifically
> talk about non-Docker. Hope this is ok. Thanks.
> 

Fair enough, and thanks for clarifying the reasons.

Christine, so it seems that the JEP will not deal with CGroups as a whole. Is there another fix in the works for this?

Comment 5 Severin Gehwolf 2017-11-30 10:54:17 UTC
(In reply to Marko Myllynen from comment #3)
> I'm reopening this Fedora / OpenJDK 9 BZ on the grounds that
> https://bugzilla.redhat.com/show_bug.cgi?id=1463098 is a non-public BZ about
> RHEL 7 / OpenJDK 8 and https://bugs.openjdk.java.net/browse/JDK-8182070
> specifically says [1] that it's about Docker only while here we specifically
> talk about non-Docker. Hope this is ok. Thanks.
> 
> 1) "It is not a goal of this JEP to support any platform other than Docker
> container technology running on Linux x64."

I've just tested this and upstream JDK-8146115 fixes this. That is, even though the JEP does not make a guarantee that it will work outside of Docker.

$ sudo systemd-run --slice openjdk10-cg.slice --scope bash
Running scope as unit: run-r5a37873d2405446a87d3c96cf8c9cfd1.scope
[root@localhost openjdk-hs]# ./build/linux-x86_64-normal-server-release/images/jdk/bin/java -Xlog:os+container=trace -version
[0.015s][trace][os,container] OSContainer::init: Initializing Container Support
[0.015s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/openjdk10.slice/openjdk10-cg.slice/memory.limit_in_bytes
[0.016s][trace][os,container] Memory Limit is: 104857600
[0.018s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.018s][trace][os,container] CPU Shares is: 1024
[0.018s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.018s][trace][os,container] CPU Quota is: -1
[0.018s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.018s][trace][os,container] CPU Period is: 100000
[0.018s][trace][os,container] OSContainer::active_processor_count: 8
[0.021s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.021s][trace][os,container] CPU Shares is: 1024
[0.021s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.021s][trace][os,container] CPU Quota is: -1
[0.021s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.021s][trace][os,container] CPU Period is: 100000
[0.021s][trace][os,container] OSContainer::active_processor_count: 8
[0.021s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/openjdk10.slice/openjdk10-cg.slice/memory.limit_in_bytes
[0.021s][trace][os,container] Memory Limit is: 104857600
[0.024s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/openjdk10.slice/openjdk10-cg.slice/memory.limit_in_bytes
[0.024s][trace][os,container] Memory Limit is: 104857600
[0.053s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.053s][trace][os,container] CPU Shares is: 1024
[0.053s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.053s][trace][os,container] CPU Quota is: -1
[0.053s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.053s][trace][os,container] CPU Period is: 100000
[0.053s][trace][os,container] OSContainer::active_processor_count: 8
[0.175s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.175s][trace][os,container] CPU Shares is: 1024
[0.175s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.175s][trace][os,container] CPU Quota is: -1
[0.175s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.175s][trace][os,container] CPU Period is: 100000
[0.175s][trace][os,container] OSContainer::active_processor_count: 8
openjdk version "10-internal"
OpenJDK Runtime Environment (build 10-internal+0-adhoc.sgehwolf.openjdk-hs)
OpenJDK 64-Bit Server VM (build 10-internal+0-adhoc.sgehwolf.openjdk-hs, mixed mode)

Comment 6 Marko Myllynen 2018-02-22 12:13:25 UTC
Do we have any rough ETA when this might be available in Fedora for testing? Thanks.

Comment 7 Andrew John Hughes 2018-03-01 16:37:16 UTC
I started looking at backporting it before the last CPU, but that would be for java-1.8.0-openjdk. java-9-openjdk will be supplanted by JDK 10 some time in the next month which has this fix.

Comment 8 Christine Flood 2018-03-01 16:43:58 UTC
Marko,

This patch is in JDK10.  JDK10 is scheduled for general release on March 20.  We are currently building JDK10 rpms and testing them.  

There are plans to back port this patch to earlier OpenJDK versions, see Andrew Hughes comment above.

Is this sufficient?

Comment 9 Marko Myllynen 2018-03-02 08:21:18 UTC
Thanks, this sounds perfect - I'll change this BZ to be against JDK 8 now, please feel free to change against JDK 10 if more appropriate. Thanks.

Comment 10 Severin Gehwolf 2018-04-05 12:55:43 UTC
JDK 10 is currently on review (see bug 1557371). With JDK >= 10 new JVM options will be introduced. Detection itself should be automatic, though.

-XX:ActiveProcessorCount=xx
-XX:-UseContainerSupport

Traces on the container detection level will be shown via:

-Xlog:os+container=trace

See: https://bugs.openjdk.java.net/browse/JDK-8190942

Comment 11 Severin Gehwolf 2018-07-24 09:03:52 UTC
The latest F27 build includes the backport patch of improved docker container detection for JDK 8:
https://bugs.openjdk.java.net/browse/JDK-8146115

That fix should make container detection automatic (no need for UseCGroupMemoryLimitForHeap anymore). However, you might run into bug 1599387 which may prevent slices to work correctly even though the JDK has the patch.

Comment 12 Fedora Update System 2018-07-24 09:04:42 UTC
java-1.8.0-openjdk-1.8.0.181-7.b13.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d4bfa98f6a

Comment 13 Fedora Update System 2018-07-24 09:04:49 UTC
java-1.8.0-openjdk-1.8.0.181-7.b13.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4d58785bcd

Comment 14 Severin Gehwolf 2018-07-24 11:28:15 UTC
One way to figure out whether your JDK supports the container detection features is this:

$ java -XX:+PrintFlagsFinal -XX:+UnlockDiagnosticVMOptions -version | grep PrintContainerInfo
     bool PrintContainerInfo                        = false                               {diagnostic}

Comment 15 Fedora Update System 2018-07-24 15:11:57 UTC
java-1.8.0-openjdk-1.8.0.181-7.b13.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d4bfa98f6a

Comment 16 Fedora Update System 2018-07-24 17:21:28 UTC
java-1.8.0-openjdk-1.8.0.181-7.b13.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4d58785bcd

Comment 17 Fedora Update System 2018-07-29 03:20:44 UTC
java-1.8.0-openjdk-1.8.0.181-7.b13.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2018-07-31 17:13:19 UTC
java-1.8.0-openjdk-1.8.0.181-7.b13.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Marko Myllynen 2018-08-13 10:38:31 UTC
Tested with java-1.8.0-openjdk-1.8.0.181-7.b13.fc28.x86_64 on Fedora 28, doesn't seem to solve this, with the unit file from the comment 0 I still see:

Aug 13 13:31:18 localhost.localdomain systemd[1]: Reloading.
Aug 13 13:31:24 localhost.localdomain systemd[1]: Starting javatest...
Aug 13 13:31:25 localhost.localdomain java[13388]: VM settings:
Aug 13 13:31:25 localhost.localdomain java[13388]:     Max. Heap Size (Estimated): 1.95G
Aug 13 13:31:25 localhost.localdomain java[13388]:     Using VM: OpenJDK 64-Bit Server VM
...

Let me know if there's a better way to test this. Thanks.

Comment 20 Severin Gehwolf 2018-08-17 19:25:21 UTC
(In reply to Marko Myllynen from comment #19)
> Tested with java-1.8.0-openjdk-1.8.0.181-7.b13.fc28.x86_64 on Fedora 28,
> doesn't seem to solve this, with the unit file from the comment 0 I still
> see:
> 
> Aug 13 13:31:18 localhost.localdomain systemd[1]: Reloading.
> Aug 13 13:31:24 localhost.localdomain systemd[1]: Starting javatest...
> Aug 13 13:31:25 localhost.localdomain java[13388]: VM settings:
> Aug 13 13:31:25 localhost.localdomain java[13388]:     Max. Heap Size
> (Estimated): 1.95G
> Aug 13 13:31:25 localhost.localdomain java[13388]:     Using VM: OpenJDK
> 64-Bit Server VM
> ...
> 
> Let me know if there's a better way to test this. Thanks.

Marko, that's likely bug 1599387. Have you checked that out?

Comment 21 Marko Myllynen 2018-08-21 05:36:42 UTC
(In reply to Severin Gehwolf from comment #20)
> > 
> > Let me know if there's a better way to test this. Thanks.
> 
> Marko, that's likely bug 1599387. Have you checked that out?

Sorry, somehow missed that - I'll wait for that BZ to be solved and retest then. Thanks.

Comment 22 Ben Cotton 2018-11-27 14:46:49 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 23 Ben Cotton 2018-11-30 22:25:18 UTC
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 24 Severin Gehwolf 2019-01-17 14:48:43 UTC
Re-opening as this doesn't seem to be fixed for recent kernel versions. It used to work with older kernels.

I can reproduce on F29.

Comment 25 Severin Gehwolf 2019-01-17 15:04:37 UTC
I've proposed a potential fix for this upstream:
http://mail.openjdk.java.net/pipermail/hotspot-dev/2019-January/036292.html

We'll see what comes of it.

Comment 26 Ben Cotton 2019-10-31 20:09:48 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 27 Ben Cotton 2020-04-30 20:46:12 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 28 Ben Cotton 2020-05-26 17:46:23 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 29 Andrew John Hughes 2020-05-27 02:16:05 UTC
Severin, was this fixed? Apparently https://bugs.openjdk.java.net/browse/JDK-8217338 was in the April update.

Comment 30 Severin Gehwolf 2020-06-03 06:51:58 UTC
(In reply to Andrew John Hughes from comment #29)
> Severin, was this fixed?

Yes.

> Apparently
> https://bugs.openjdk.java.net/browse/JDK-8217338 was in the April update.


$ uname -r
5.6.14-300.fc32.x86_64

This is a system with cgroup v1 (hybrid hierarchy) manually configured. cgroups v2 is now default for F31 and up.

With this systemd service file:

$ cat /etc/systemd/system/test.service 
[Unit]
Description=javatest

[Service]
Type=oneshot
ExecStart=/usr/lib/jvm/java-11-openjdk/bin/java -XX:MaxRAMFraction=1 -XshowSettings:system -version

MemoryAccounting=true
MemoryLimit=100M

$ sudo systemctl daemon-reload && sudo systemctl start test

Jun 03 08:42:58 t580-laptop systemd[1]: Starting javatest...
Jun 03 08:42:58 t580-laptop java[5912]: OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
Jun 03 08:42:58 t580-laptop java[5912]: Operating System Metrics:
Jun 03 08:42:58 t580-laptop java[5912]:     Provider: cgroupv1
Jun 03 08:42:58 t580-laptop java[5912]:     Effective CPU Count: 8
Jun 03 08:42:58 t580-laptop java[5912]:     CPU Period: 100000us
Jun 03 08:42:58 t580-laptop java[5912]:     CPU Quota: -1
Jun 03 08:42:58 t580-laptop java[5912]:     CPU Shares: -1
Jun 03 08:42:58 t580-laptop java[5912]:     List of Processors, 8 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0 1 2 3 4 5 6 7
Jun 03 08:42:58 t580-laptop java[5912]:     List of Effective Processors, 8 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0 1 2 3 4 5 6 7
Jun 03 08:42:58 t580-laptop java[5912]:     List of Memory Nodes, 1 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0
Jun 03 08:42:58 t580-laptop java[5912]:     List of Available Memory Nodes, 1 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0
Jun 03 08:42:58 t580-laptop java[5912]:     CPUSet Memory Pressure Enabled: false
Jun 03 08:42:58 t580-laptop java[5912]:     Memory Limit: 100.00M
Jun 03 08:42:58 t580-laptop java[5912]:     Memory Soft Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     Memory & Swap Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     Kernel Memory Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     TCP Memory Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     Out Of Memory Killer Enabled: true

Note the 'Memory Limit: 100.00M' output. With -XshowSettings:vm over XshowSettings:system, I see this:

Jun 03 08:49:47 t580-laptop java[6135]: VM settings:
Jun 03 08:49:47 t580-laptop java[6135]:     Max. Heap Size (Estimated): 48.38M

which seems an expected value.


Note You need to log in before you can comment on or make changes to this bug.