Description of problem: Current carts are tied to hardcoded gear names. Need a default based on a percentage of the gear resource limits. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Relink /etc/openshift/resource_limits.conf to a non-standard gear name 2. Create JBoss app 3. Actual results: Default memory settings are blank Expected results: Default memory settings should be a percentage of available resources Additional info:
Set resource limits in cart based on % of: ulimit -u for threads /usr/bin/oo-cgroup-read memory.limit_in_bytes for heap Create env for setting %
Commits pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/f904ab692db51c0f891b8a7be20a415778c3d67e Bug 906845 https://github.com/openshift/origin-server/commit/f02c3a12df690c3058051573156d26245b603c5e Bug 906845 - maven heap size
Version: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/1.1.z/2013-02-04.1/ Verify: [root@broker ~]# rhc app create jbosseap jbosseap -pjia -g newsize Using jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) for 'jbosseap' Application Options =================== Cartridge: jbosseap-6.0 Gear Size: newsize Namespace: 13 Scaling: no Creating application 'jbosseap' ... done Waiting for your DNS name to be available ... done Downloading the application Git repository ... fatal: destination path 'jbosseap' already exists and is not an empty directory. The directory you are cloning into already exists. jbosseap @ http://jbosseap-13.test.com/ (uuid: 548cbe58c76d40778bc7e0ba0919b1a2) ================================================================================ Created: 7:34 AM Gear Size: newsize Git URL: ssh://548cbe58c76d40778bc7e0ba0919b1a2.com/~/git/jbosseap.git/ SSH: 548cbe58c76d40778bc7e0ba0919b1a2.com jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) ======================================================== [root@broker ~]# ssh 548cbe58c76d40778bc7e0ba0919b1a2.com The authenticity of host 'jbosseap-13.test.com (10.4.59.203)' can't be established. RSA key fingerprint is 6c:a5:e5:fa:75:db:5a:7f:dc:a2:44:ed:e4:97:af:3c. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'jbosseap-13.test.com' (RSA) to the list of known hosts. ********************************************************************* You are accessing a service that is for use only by authorized users. If you do not have authorization, discontinue use at once. Any use of the services is subject to the applicable terms of the agreement which can be found at: https://openshift.redhat.com/app/legal ********************************************************************* Welcome to OpenShift shell This shell will assist you in managing OpenShift applications. !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Shell access is quite powerful and it is possible for you to accidentally damage your application. Proceed with care! If worse comes to worst, destroy your application with 'rhc app destroy' and recreate it !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Type "help" for more info. [jbosseap-13.test.com ~]\> quota quota: Cannot resolve mountpoint path /boot: Permission denied Disk quotas for user 548cbe58c76d40778bc7e0ba0919b1a2 (uid 500): Filesystem blocks quota limit grace files quota limit grace /dev/mapper/VolGroup-lv_root 5268 0 1048576 253 0 40000 [jbosseap-13.test.com ~]\> exit exit Connection to jbosseap-13.test.com closed. [root@broker ~]# ssh node1 root@node1's password: Last login: Wed Feb 6 07:32:50 2013 from vm-199-59-4-10.ose.phx2.redhat.com [root@node1 ~]# cat /etc/cg cgconfig.conf cgrules.conf cgsnapshot_blacklist.conf [root@node1 ~]# cat /etc/cg cgconfig.conf cgrules.conf cgsnapshot_blacklist.conf [root@node1 ~]# cat /etc/cgrules.conf # /etc/cgrules.conf # #Each line describes a rule for a user in the forms: # #<user> <controllers> <destination> #<user>:<process name> <controllers> <destination> # #Where: # <user> can be: # - an user name # - a group name, with @group syntax # - the wildcard *, for any user or group. # - The %, which is equivalent to "ditto". This is useful for # multiline rules where different cgroups need to be specified # for various hierarchies for a single user. # # <process name> is optional and it can be: # - a process name # - a full command path of a process # # <controller> can be: # - comma separated controller names (no spaces) # - * (for all mounted controllers) # # <destination> can be: # - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1) # # Note: # - It currently has rules based on uids, gids and process name. # # - Don't put overlapping rules. First rule which matches the criteria # will be executed. # # - Multiline rules can be specified for specifying different cgroups # for multiple hierarchies. In the example below, user "peter" has # specified 2 line rule. First line says put peter's task in test1/ # dir for "cpu" controller and second line says put peter's tasks in # test2/ dir for memory controller. Make a note of "%" sign in second line. # This is an indication that it is continuation of previous rule. # # #<user> <controllers> <destination> # #john cpu usergroup/faculty/john/ #john:cp cpu usergroup/faculty/john/cp #@student cpu,memory usergroup/student/ #peter cpu test1/ #% memory test2/ #@root * admingroup/ #* * default/ # End of file 548cbe58c76d40778bc7e0ba0919b1a2 cpu,cpuacct,memory,net_cls,freezer /openshift/548cbe58c76d40778bc7e0ba0919b1a2 [root@node1 ~]# cat /cgroup/all/openshift/ 548cbe58c76d40778bc7e0ba0919b1a2/ cpu.rt_period_us memory.max_usage_in_bytes memory.stat cgroup.event_control cpu.rt_runtime_us memory.memsw.failcnt memory.swappiness cgroup.procs cpu.shares memory.memsw.limit_in_bytes memory.usage_in_bytes cpuacct.stat cpu.stat memory.memsw.max_usage_in_bytes memory.use_hierarchy cpuacct.usage freezer.state memory.memsw.usage_in_bytes net_cls.classid cpuacct.usage_percpu memory.failcnt memory.move_charge_at_immigrate notify_on_release cpu.cfs_period_us memory.force_empty memory.oom_control tasks cpu.cfs_quota_us memory.limit_in_bytes memory.soft_limit_in_bytes [root@node1 ~]# cat /cgroup/all/openshift/548cbe58c76d40778bc7e0ba0919b1a2/ cat: /cgroup/all/openshift/548cbe58c76d40778bc7e0ba0919b1a2/: Is a directory [root@node1 ~]# ll /cgroup/all/openshift/548cbe58c76d40778bc7e0ba0919b1a2/ total 0 --w--w--w-. 1 root root 0 Feb 6 07:34 cgroup.event_control -rw-r--r--. 1 root root 0 Feb 6 07:34 cgroup.procs -r--r--r--. 1 root root 0 Feb 6 07:34 cpuacct.stat -rw-r--r--. 1 root root 0 Feb 6 07:34 cpuacct.usage -r--r--r--. 1 root root 0 Feb 6 07:34 cpuacct.usage_percpu -rw-r--r--. 1 root root 0 Feb 6 07:34 cpu.cfs_period_us -rw-r--r--. 1 root root 0 Feb 6 07:35 cpu.cfs_quota_us -rw-r--r--. 1 root root 0 Feb 6 07:34 cpu.rt_period_us -rw-r--r--. 1 root root 0 Feb 6 07:34 cpu.rt_runtime_us -rw-r--r--. 1 root root 0 Feb 6 07:34 cpu.shares -r--r--r--. 1 root root 0 Feb 6 07:34 cpu.stat -rw-r--r--. 1 root root 0 Feb 6 07:34 freezer.state -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.failcnt --w-------. 1 root root 0 Feb 6 07:34 memory.force_empty -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.limit_in_bytes -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.max_usage_in_bytes -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.memsw.failcnt -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.memsw.limit_in_bytes -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.memsw.max_usage_in_bytes -r--r--r--. 1 root root 0 Feb 6 07:34 memory.memsw.usage_in_bytes -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.move_charge_at_immigrate -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.oom_control -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.soft_limit_in_bytes -r--r--r--. 1 root root 0 Feb 6 07:34 memory.stat -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.swappiness -r--r--r--. 1 root root 0 Feb 6 07:34 memory.usage_in_bytes -rw-r--r--. 1 root root 0 Feb 6 07:34 memory.use_hierarchy -rw-r--r--. 1 root root 0 Feb 6 07:34 net_cls.classid -rw-r--r--. 1 root root 0 Feb 6 07:34 notify_on_release -rw-r--r--. 1 548cbe58c76d40778bc7e0ba0919b1a2 548cbe58c76d40778bc7e0ba0919b1a2 0 Feb 6 07:35 tasks [root@node1 ~]# cat /cgroup/all/openshift/548cbe58c76d40778bc7e0ba0919b1a2/memory.* 0 cat: /cgroup/all/openshift/548cbe58c76d40778bc7e0ba0919b1a2/memory.force_empty: Invalid argument 536870912 286420992 0 641728512 286420992 259608576 0 oom_kill_disable 0 under_oom 0 9223372036854775807 cache 4489216 rss 254881792 mapped_file 32768 pgpgin 81311 pgpgout 59379 swap 0 inactive_anon 0 active_anon 254881792 inactive_file 4407296 active_file 81920 unevictable 0 hierarchical_memory_limit 536870912 hierarchical_memsw_limit 641728512 total_cache 4489216 total_rss 254881792 total_mapped_file 32768 total_pgpgin 81311 total_pgpgout 59379 total_swap 0 total_inactive_anon 0 total_active_anon 254881792 total_inactive_file 4407296 total_active_file 81920 total_unevictable 0 60 259608576 0 [root@node1 ~]#
My fault, I use the wrong way to verify it. Test it again. Find an interesting appearance. I use the default node_profile "small" to create a jbosseap. The java process is "/usr/lib/jvm/jre-1.7.0/bin/java -D[Standalone] -client ..." When node_profile equals "newsize" to create another jbosseap, the java process is "/usr/lib/jvm/jre-1.7.0/bin/java -D[Standalone] -server ..." I think the java process should be as "server" role. I don't know in which situation, it can be as "client" role. [ Can you give me an answer? Many thanks ;-) ] So I check again and again, finally, I find the file /usr/libexec/openshift/cartridges/jbosseap-6.0/info/bin/standalone.conf is wrong. The code is not in. But /usr/libexec/openshift/cartridges/jbosseap-6.0/info/bin/standalone.sh is correct. Could you re-package it ? Thanks.
This week has definitely shown that we have a deficiency in our build process. I saw that the build was submitted last Monday but I'm betting it failed and I missed the notification before moving this to ON_QA. I'll be more careful until we can get this automated.
(In reply to comment #6) > I think the java process should be as "server" role. I don't know in which > situation, it can be as "client" role. [ Can you give me an answer? Many > thanks ;-) ] This is an old JVM argument intended to adjust various JVM settings according to intended usage. But with the 64-bit JVM it actually does nothing. java man page: -client Select the Java HotSpot Client VM. A 64-bit capable jdk currently ignores this option and instead uses the Java Hotspot Server VM. [jbosseap-demo.ose10ref.example.com ~]\> /usr/lib/jvm/jre-1.7.0/bin/java -version java version "1.7.0_09-icedtea" OpenJDK Runtime Environment (rhel-2.3.3.2.el6_3-x86_64) OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
The following packages are available in the latest puddle: openshift-origin-cartridge-jbossews-1.0-1.0.13-1.el6op openshift-origin-cartridge-jbosseap-6.0-1.0.4-1.el6op http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/1.1.z/2013-02-07.3/
Version: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/1.1.z/2013-02-07.3/ Verify: [root@broker ~]# mco facts node_profile Report for fact: node_profile newsize found 1 times Finished processing 1 / 1 hosts in 52.17 ms [root@broker ~]# [root@broker ~]# [root@broker ~]# [root@broker ~]# [root@broker ~]# [root@broker ~]# rm -rf jbosseap*; rhc app create jbosseap1 jbosseap -pjia -g newsize Using jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) for 'jbosseap' Application Options =================== Scaling: no Cartridge: jbosseap-6.0 Namespace: jia Gear Size: newsize Creating application 'jbosseap1' ... done Waiting for your DNS name to be available ... done Downloading the application Git repository ... Initialized empty Git repository in /root/jbosseap1/.git/ Your application code is now in 'jbosseap1' jbosseap1 @ http://jbosseap1-jia.test.com/ (uuid: a67b7db2ab34462ea2a311c2a2ae0415) =================================================================================== Created: 9:58 PM Gear Size: newsize Git URL: ssh://a67b7db2ab34462ea2a311c2a2ae0415.com/~/git/jbosseap1.git/ SSH: a67b7db2ab34462ea2a311c2a2ae0415.com jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) ======================================================== RESULT: Application jbosseap1 was created. [root@broker ~]# [root@broker ~]# [root@broker ~]# ssh a67b7db2ab34462ea2a311c2a2ae0415.com ********************************************************************* You are accessing a service that is for use only by authorized users. If you do not have authorization, discontinue use at once. Any use of the services is subject to the applicable terms of the agreement which can be found at: https://openshift.redhat.com/app/legal ********************************************************************* Welcome to OpenShift shell This shell will assist you in managing OpenShift applications. !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Shell access is quite powerful and it is possible for you to accidentally damage your application. Proceed with care! If worse comes to worst, destroy your application with 'rhc app destroy' and recreate it !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Type "help" for more info. [jbosseap1-jia.test.com ~]\> ps aux | grep java 500 23490 14.1 3.4 2775796 275248 ? Sl 21:58 0:18 /usr/lib/jvm/jre-1.7.0/bin/java -D[Standalone] -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xmx256m -XX:MaxPermSize=102m -XX:+AggressiveOpts -Dorg.apache.tomcat.util.LOW_MEMORY=true -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Djboss.node.name=jbosseap1-jia.test.com -Djgroups.bind_addr=127.0.250.1 -Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on -Dorg.jboss.boot.log.file=/var/lib/openshift/a67b7db2ab34462ea2a311c2a2ae0415/jbosseap-6.0/jbosseap-6.0/standalone/log/boot.log -Dlogging.configuration=file:/var/lib/openshift/a67b7db2ab34462ea2a311c2a2ae0415/jbosseap-6.0/jbosseap-6.0/standalone/configuration/logging.properties -jar /var/lib/openshift/a67b7db2ab34462ea2a311c2a2ae0415/jbosseap-6.0/jbosseap-6.0/jboss-modules.jar -mp /var/lib/openshift/a67b7db2ab34462ea2a311c2a2ae0415//jbosseap-6.0/jbosseap-6.0/standalone/configuration/modules:/var/lib/openshift/a67b7db2ab34462ea2a311c2a2ae0415//jbosseap-6.0/jbosseap-6.0/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/var/lib/openshift/a67b7db2ab34462ea2a311c2a2ae0415/jbosseap-6.0/jbosseap-6.0 500 24780 0.0 0.0 103244 804 pts/0 S+ 22:01 0:00 grep java [jbosseap1-jia.test.com ~]\> exit exit Connection to jbosseap1-jia.test.com closed. [root@broker ~]# ssh node1 root@node1's password: Last login: Thu Feb 7 21:54:18 2013 from vm-199-59-4-10.ose.phx2.redhat.com [root@node1 ~]# vim /etc/openshift/resource_limits.conf [root@node1 ~]# exit logout Connection to node1 closed. [root@broker ~]# mco facts node_profile Report for fact: node_profile small found 1 times Finished processing 1 / 1 hosts in 61.43 ms [root@broker ~]# rm -rf jbosseap*; rhc app create jbosseap2 jbosseap -pjia Using jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) for 'jbosseap' Application Options =================== Cartridge: jbosseap-6.0 Gear Size: default Namespace: jia Scaling: no Creating application 'jbosseap2' ... done Waiting for your DNS name to be available ... done Downloading the application Git repository ... Initialized empty Git repository in /root/jbosseap2/.git/ Your application code is now in 'jbosseap2' jbosseap2 @ http://jbosseap2-jia.test.com/ (uuid: 69589f6944004c1683123072f256a51d) =================================================================================== Created: 10:02 PM Gear Size: small Git URL: ssh://69589f6944004c1683123072f256a51d.com/~/git/jbosseap2.git/ SSH: 69589f6944004c1683123072f256a51d.com jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) ======================================================== RESULT: Application jbosseap2 was created. [root@broker ~]# ssh 69589f6944004c1683123072f256a51d.com ********************************************************************* You are accessing a service that is for use only by authorized users. If you do not have authorization, discontinue use at once. Any use of the services is subject to the applicable terms of the agreement which can be found at: https://openshift.redhat.com/app/legal ********************************************************************* Welcome to OpenShift shell This shell will assist you in managing OpenShift applications. !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Shell access is quite powerful and it is possible for you to accidentally damage your application. Proceed with care! If worse comes to worst, destroy your application with 'rhc app destroy' and recreate it !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Type "help" for more info. [jbosseap2-jia.test.com ~]\> ps aux | grep java 508 25734 49.7 3.3 2802376 272272 ? Sl 22:02 0:16 /usr/lib/jvm/jre-1.7.0/bin/java -D[Standalone] -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xmx256m -XX:MaxPermSize=102m -XX:+AggressiveOpts -Dorg.apache.tomcat.util.LOW_MEMORY=true -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Djboss.node.name=jbosseap2-jia.test.com -Djgroups.bind_addr=127.0.254.1 -Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on -Dorg.jboss.boot.log.file=/var/lib/openshift/69589f6944004c1683123072f256a51d/jbosseap-6.0/jbosseap-6.0/standalone/log/boot.log -Dlogging.configuration=file:/var/lib/openshift/69589f6944004c1683123072f256a51d/jbosseap-6.0/jbosseap-6.0/standalone/configuration/logging.properties -jar /var/lib/openshift/69589f6944004c1683123072f256a51d/jbosseap-6.0/jbosseap-6.0/jboss-modules.jar -mp /var/lib/openshift/69589f6944004c1683123072f256a51d//jbosseap-6.0/jbosseap-6.0/standalone/configuration/modules:/var/lib/openshift/69589f6944004c1683123072f256a51d//jbosseap-6.0/jbosseap-6.0/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/var/lib/openshift/69589f6944004c1683123072f256a51d/jbosseap-6.0/jbosseap-6.0 508 26231 0.0 0.0 103244 796 pts/0 S+ 22:03 0:00 grep java
I am re-opening this because the calculation for maven heap size is broken on master: $ cd jtest; echo 'foo' >> README; git commit -am 'test'; git push; cd - [master f565768] test 1 file changed, 1 insertion(+) Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 282 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: restart_on_add=false remote: Sending SIGTERM to jboss:18670 ... remote: kill -9 18670 remote: Done remote: restart_on_add=false remote: Running .openshift/action_hooks/pre_build remote: /usr/bin/env: oo-ruby: Permission denied remote: expr: syntax error remote: (standard_in) 1: syntax error remote: Sourcing pre_build_jbossews-1.0 remote: Found pom.xml... attempting to build with 'mvn -e clean package -Popenshift -DskipTests' remote: Invalid maximum heap size: -Xmxm remote: Error: Could not create the Java Virtual Machine. remote: Error: A fatal exception has occurred. Program will exit. remote: Invalid maximum heap size: -Xmxm remote: Error: Could not create the Java Virtual Machine. remote: Error: A fatal exception has occurred. Program will exit. This happens for as,eap, and ews.
I re-try it again. No such error. I think it only happens on Online. OSE doesn't use oo-ruby. You can see the step I test, if anything I wrong, please tell me , thanks. http://pastebin.test.redhat.com/129123
Hi- I ran into this in a devenv - Bill Decoste tells me that he's waiting on an SELinux policy change that will make this error go away.
This is blocked on this: https://bugzilla.redhat.com/show_bug.cgi?id=912215
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0582.html