Bug 1509371

This is a duplicate of:

https://bugzilla.redhat.com/show_bug.cgi?id=1463098

The command line option was put in as a temporary fix for docker.  It wasn't aimed to fix the problem for everyone.  The long term universal solution is a harder problem and is expected to come from: 

http://openjdk.java.net/jeps/8182070
Container aware Java

Closing as deferred. Please use upstream bug for checking progress:
https://bugs.openjdk.java.net/browse/JDK-8182070

I'm reopening this Fedora / OpenJDK 9 BZ on the grounds that https://bugzilla.redhat.com/show_bug.cgi?id=1463098 is a non-public BZ about RHEL 7 / OpenJDK 8 and https://bugs.openjdk.java.net/browse/JDK-8182070 specifically says [1] that it's about Docker only while here we specifically talk about non-Docker. Hope this is ok. Thanks.

1) "It is not a goal of this JEP to support any platform other than Docker container technology running on Linux x64."

(In reply to Marko Myllynen from comment #3)
> I'm reopening this Fedora / OpenJDK 9 BZ on the grounds that
> https://bugzilla.redhat.com/show_bug.cgi?id=1463098 is a non-public BZ about
> RHEL 7 / OpenJDK 8 and https://bugs.openjdk.java.net/browse/JDK-8182070
> specifically says [1] that it's about Docker only while here we specifically
> talk about non-Docker. Hope this is ok. Thanks.
> 

Fair enough, and thanks for clarifying the reasons.

Christine, so it seems that the JEP will not deal with CGroups as a whole. Is there another fix in the works for this?

(In reply to Marko Myllynen from comment #3)
> I'm reopening this Fedora / OpenJDK 9 BZ on the grounds that
> https://bugzilla.redhat.com/show_bug.cgi?id=1463098 is a non-public BZ about
> RHEL 7 / OpenJDK 8 and https://bugs.openjdk.java.net/browse/JDK-8182070
> specifically says [1] that it's about Docker only while here we specifically
> talk about non-Docker. Hope this is ok. Thanks.
> 
> 1) "It is not a goal of this JEP to support any platform other than Docker
> container technology running on Linux x64."

I've just tested this and upstream JDK-8146115 fixes this. That is, even though the JEP does not make a guarantee that it will work outside of Docker.

$ sudo systemd-run --slice openjdk10-cg.slice --scope bash
Running scope as unit: run-r5a37873d2405446a87d3c96cf8c9cfd1.scope
[root@localhost openjdk-hs]# ./build/linux-x86_64-normal-server-release/images/jdk/bin/java -Xlog:os+container=trace -version
[0.015s][trace][os,container] OSContainer::init: Initializing Container Support
[0.015s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/openjdk10.slice/openjdk10-cg.slice/memory.limit_in_bytes
[0.016s][trace][os,container] Memory Limit is: 104857600
[0.018s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.018s][trace][os,container] CPU Shares is: 1024
[0.018s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.018s][trace][os,container] CPU Quota is: -1
[0.018s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.018s][trace][os,container] CPU Period is: 100000
[0.018s][trace][os,container] OSContainer::active_processor_count: 8
[0.021s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.021s][trace][os,container] CPU Shares is: 1024
[0.021s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.021s][trace][os,container] CPU Quota is: -1
[0.021s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.021s][trace][os,container] CPU Period is: 100000
[0.021s][trace][os,container] OSContainer::active_processor_count: 8
[0.021s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/openjdk10.slice/openjdk10-cg.slice/memory.limit_in_bytes
[0.021s][trace][os,container] Memory Limit is: 104857600
[0.024s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/openjdk10.slice/openjdk10-cg.slice/memory.limit_in_bytes
[0.024s][trace][os,container] Memory Limit is: 104857600
[0.053s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.053s][trace][os,container] CPU Shares is: 1024
[0.053s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.053s][trace][os,container] CPU Quota is: -1
[0.053s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.053s][trace][os,container] CPU Period is: 100000
[0.053s][trace][os,container] OSContainer::active_processor_count: 8
[0.175s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.shares
[0.175s][trace][os,container] CPU Shares is: 1024
[0.175s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_quota_us
[0.175s][trace][os,container] CPU Quota is: -1
[0.175s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/openjdk10.slice/cpu.cfs_period_us
[0.175s][trace][os,container] CPU Period is: 100000
[0.175s][trace][os,container] OSContainer::active_processor_count: 8
openjdk version "10-internal"
OpenJDK Runtime Environment (build 10-internal+0-adhoc.sgehwolf.openjdk-hs)
OpenJDK 64-Bit Server VM (build 10-internal+0-adhoc.sgehwolf.openjdk-hs, mixed mode)

Do we have any rough ETA when this might be available in Fedora for testing? Thanks.

I started looking at backporting it before the last CPU, but that would be for java-1.8.0-openjdk. java-9-openjdk will be supplanted by JDK 10 some time in the next month which has this fix.

Marko,

This patch is in JDK10.  JDK10 is scheduled for general release on March 20.  We are currently building JDK10 rpms and testing them.  

There are plans to back port this patch to earlier OpenJDK versions, see Andrew Hughes comment above.

Is this sufficient?

Thanks, this sounds perfect - I'll change this BZ to be against JDK 8 now, please feel free to change against JDK 10 if more appropriate. Thanks.

JDK 10 is currently on review (see bug 1557371). With JDK >= 10 new JVM options will be introduced. Detection itself should be automatic, though.

-XX:ActiveProcessorCount=xx
-XX:-UseContainerSupport

Traces on the container detection level will be shown via:

-Xlog:os+container=trace

See: https://bugs.openjdk.java.net/browse/JDK-8190942

The latest F27 build includes the backport patch of improved docker container detection for JDK 8:
https://bugs.openjdk.java.net/browse/JDK-8146115

That fix should make container detection automatic (no need for UseCGroupMemoryLimitForHeap anymore). However, you might run into bug 1599387 which may prevent slices to work correctly even though the JDK has the patch.

java-1.8.0-openjdk-1.8.0.181-7.b13.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d4bfa98f6a

java-1.8.0-openjdk-1.8.0.181-7.b13.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4d58785bcd

One way to figure out whether your JDK supports the container detection features is this:

$ java -XX:+PrintFlagsFinal -XX:+UnlockDiagnosticVMOptions -version | grep PrintContainerInfo
     bool PrintContainerInfo                        = false                               {diagnostic}

java-1.8.0-openjdk-1.8.0.181-7.b13.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d4bfa98f6a

java-1.8.0-openjdk-1.8.0.181-7.b13.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4d58785bcd

java-1.8.0-openjdk-1.8.0.181-7.b13.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

java-1.8.0-openjdk-1.8.0.181-7.b13.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Tested with java-1.8.0-openjdk-1.8.0.181-7.b13.fc28.x86_64 on Fedora 28, doesn't seem to solve this, with the unit file from the comment 0 I still see:

Aug 13 13:31:18 localhost.localdomain systemd[1]: Reloading.
Aug 13 13:31:24 localhost.localdomain systemd[1]: Starting javatest...
Aug 13 13:31:25 localhost.localdomain java[13388]: VM settings:
Aug 13 13:31:25 localhost.localdomain java[13388]:     Max. Heap Size (Estimated): 1.95G
Aug 13 13:31:25 localhost.localdomain java[13388]:     Using VM: OpenJDK 64-Bit Server VM
...

Let me know if there's a better way to test this. Thanks.

(In reply to Marko Myllynen from comment #19)
> Tested with java-1.8.0-openjdk-1.8.0.181-7.b13.fc28.x86_64 on Fedora 28,
> doesn't seem to solve this, with the unit file from the comment 0 I still
> see:
> 
> Aug 13 13:31:18 localhost.localdomain systemd[1]: Reloading.
> Aug 13 13:31:24 localhost.localdomain systemd[1]: Starting javatest...
> Aug 13 13:31:25 localhost.localdomain java[13388]: VM settings:
> Aug 13 13:31:25 localhost.localdomain java[13388]:     Max. Heap Size
> (Estimated): 1.95G
> Aug 13 13:31:25 localhost.localdomain java[13388]:     Using VM: OpenJDK
> 64-Bit Server VM
> ...
> 
> Let me know if there's a better way to test this. Thanks.

Marko, that's likely bug 1599387. Have you checked that out?

(In reply to Severin Gehwolf from comment #20)
> > 
> > Let me know if there's a better way to test this. Thanks.
> 
> Marko, that's likely bug 1599387. Have you checked that out?

Sorry, somehow missed that - I'll wait for that BZ to be solved and retest then. Thanks.

This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Re-opening as this doesn't seem to be fixed for recent kernel versions. It used to work with older kernels.

I can reproduce on F29.

I've proposed a potential fix for this upstream:
http://mail.openjdk.java.net/pipermail/hotspot-dev/2019-January/036292.html

We'll see what comes of it.

This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Severin, was this fixed? Apparently https://bugs.openjdk.java.net/browse/JDK-8217338 was in the April update.

(In reply to Andrew John Hughes from comment #29)
> Severin, was this fixed?

Yes.

> Apparently
> https://bugs.openjdk.java.net/browse/JDK-8217338 was in the April update.


$ uname -r
5.6.14-300.fc32.x86_64

This is a system with cgroup v1 (hybrid hierarchy) manually configured. cgroups v2 is now default for F31 and up.

With this systemd service file:

$ cat /etc/systemd/system/test.service 
[Unit]
Description=javatest

[Service]
Type=oneshot
ExecStart=/usr/lib/jvm/java-11-openjdk/bin/java -XX:MaxRAMFraction=1 -XshowSettings:system -version

MemoryAccounting=true
MemoryLimit=100M

$ sudo systemctl daemon-reload && sudo systemctl start test

Jun 03 08:42:58 t580-laptop systemd[1]: Starting javatest...
Jun 03 08:42:58 t580-laptop java[5912]: OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
Jun 03 08:42:58 t580-laptop java[5912]: Operating System Metrics:
Jun 03 08:42:58 t580-laptop java[5912]:     Provider: cgroupv1
Jun 03 08:42:58 t580-laptop java[5912]:     Effective CPU Count: 8
Jun 03 08:42:58 t580-laptop java[5912]:     CPU Period: 100000us
Jun 03 08:42:58 t580-laptop java[5912]:     CPU Quota: -1
Jun 03 08:42:58 t580-laptop java[5912]:     CPU Shares: -1
Jun 03 08:42:58 t580-laptop java[5912]:     List of Processors, 8 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0 1 2 3 4 5 6 7
Jun 03 08:42:58 t580-laptop java[5912]:     List of Effective Processors, 8 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0 1 2 3 4 5 6 7
Jun 03 08:42:58 t580-laptop java[5912]:     List of Memory Nodes, 1 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0
Jun 03 08:42:58 t580-laptop java[5912]:     List of Available Memory Nodes, 1 total:
Jun 03 08:42:58 t580-laptop java[5912]:     0
Jun 03 08:42:58 t580-laptop java[5912]:     CPUSet Memory Pressure Enabled: false
Jun 03 08:42:58 t580-laptop java[5912]:     Memory Limit: 100.00M
Jun 03 08:42:58 t580-laptop java[5912]:     Memory Soft Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     Memory & Swap Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     Kernel Memory Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     TCP Memory Limit: Unlimited
Jun 03 08:42:58 t580-laptop java[5912]:     Out Of Memory Killer Enabled: true

Note the 'Memory Limit: 100.00M' output. With -XshowSettings:vm over XshowSettings:system, I see this:

Jun 03 08:49:47 t580-laptop java[6135]: VM settings:
Jun 03 08:49:47 t580-laptop java[6135]:     Max. Heap Size (Estimated): 48.38M

which seems an expected value.