Bug 2011102

Description of problem:

Memory growth over time was observed for the RHEL IPA PKI Java process, from ~10% to ~27% in ~18 hours, for a small IPA deployment.

The JVM heap size is not being set anymore in /etc/pki/pki-tomcat/tomcat.conf

We have:
JAVA_OPTS="-Dcom.redhat.fips=false"

The JVM defaults to 1/4th of the system's RAM, 2GB in this case for a 8GB RAM system, and appeared to be too small:
Java metaspace defaults to 1GB,  JVM process size is 1,5GB+, not suspiciously large.
It's possible the heap is so large that no gc activity is happening, and that's why it gets large.

and may need a tuning with
JAVA_OPTS="-Dcom.redhat.fips=false -Xms2g -Xmx2g"


should the JAVA_OPTS options Xms and Xmx be specified?
the challenge is this depends on the system RAM configuration.

Not sure why this was a problem on this system, and not a more common issue, it this is really a configuration change needed.


Version-Release number of selected component (if applicable):

RHEL-8.4 IPA
Java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64
ipa-server-4.9.2-4.module+el8.4.0+11156+94d209c1.x86_64
pki-ca-10.10.5-3.module+el8.4.0+11039+635979e4.noarch
tomcatjss-7.6.1-1.module+el8.4.0+8778+d07929ff.noarch
jss-4.8.1-2.module+el8.4.0+10451+3e5b5448.x86_64
redhat-release-eula-8.4-0.6.el8.x86_64

8GB RAM, VMWare virt guest
2 replicas
"small" IPA LDAP DBs 


How reproducible:
N/A

Steps to Reproduce:
1. N/A
2.
3.

Actual results:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
dirsrv    511292  0.3  2.6 1429520 212148 ?      Ssl  Aug13  19:12 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-edited -i /run/dirsrv/slapd-edited.pid
pkiuser   511747  0.2 20.3 5225532 1620396 ?     Ssl  Aug13  12:04 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start


Expected results:


Additional info:

$ cut -c1-150 ps.out |head
ps.31080054:USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

ps.31080054:pkiuser   185398  0.2 10.0 4637832 799044 ?      Ssl  Aug30   1:39 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080154:pkiuser   185398  0.1 10.0 4637832 799044 ?      Ssl  Aug30   1:41 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080254:pkiuser   185398  0.1 10.0 4637832 799044 ?      Ssl  Aug30   1:43 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080354:pkiuser   185398  0.1 10.0 4637832 799108 ?      Ssl  Aug30   1:45 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080454:pkiuser   185398  0.1 10.0 4637832 799108 ?      Ssl  Aug30   1:47 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080554:pkiuser   185398  0.1 10.0 4637832 799084 ?      Ssl  Aug30   1:48 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080654:pkiuser   185398  0.1 10.0 4637832 799104 ?      Ssl  Aug30   1:50 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.31080754:pkiuser   185398  0.1 10.0 4637832 799112 ?      Ssl  Aug30   1:52 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
$ cut -c1-150 ps.out |tail
ps.06090454:pkiuser   185398  0.1 26.7 5344128 2130924 ?     Ssl  Aug30  17:13 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06090554:pkiuser   185398  0.1 23.0 5344128 1832540 ?     Ssl  Aug30  17:15 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06090654:pkiuser   185398  0.1 23.0 5344128 1832876 ?     Ssl  Aug30  17:17 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06090754:pkiuser   185398  0.1 23.0 5344128 1833764 ?     Ssl  Aug30  17:19 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06090854:pkiuser   185398  0.1 22.7 5344128 1812972 ?     Ssl  Aug30  17:21 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06090954:pkiuser   185398  0.1 22.8 5344128 1818232 ?     Ssl  Aug30  17:23 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06091054:pkiuser   185398  0.1 22.8 5344128 1818460 ?     Ssl  Aug30  17:24 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06091154:pkiuser   185398  0.1 24.4 5347456 1942544 ?     Ssl  Aug30  18:34 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06091254:pkiuser   185398  0.1 24.4 5347456 1944048 ?     Ssl  Aug30  18:36 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
ps.06091354:pkiuser   185398  0.1 26.9 5679884 2146096 ?     Ssl  Aug30  19:33 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -class
// mem usage doubled.