Bug 1196783 - OPENSHIFT_GEAR_MEMORY_MB is not updated when resource limits change
Summary: OPENSHIFT_GEAR_MEMORY_MB is not updated when resource limits change
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 2.2.0
Hardware: All
OS: Linux
low
low
Target Milestone: ---
: ---
Assignee: Rory Thrasher
QA Contact: libra bugs
URL:
Whiteboard:
: 1202466 (view as bug list)
Depends On: 1354609
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-26 17:57 UTC by Josep 'Pep' Turro Mauri
Modified: 2019-07-11 08:42 UTC (History)
8 users (show)

Fixed In Version: rubygem-openshift-origin-node-1.38.6.3-1.el6op.noarch.rpm
Doc Type: Bug Fix
Doc Text:
Cause: When updating memory_limit_in_bytes in /etc/openshift/resource_limits.conf, the new memory limit does not update the OPENSHIFT_GEAR_MEMORY_MB env var for current gears. Consequence: Even when the memory limit was updated, current gears would have the same memory limit. Fix: Comments have been added to the resource_limits.conf file to provide instructions to anyone updating the memory_limit_in_bytes variable. The workaround is to run `for i in /var/lib/openshift/*/.env/OPENSHIFT_GEAR_MEMORY_MB; do echo 512 > "$i"; done` with 512 being replaced by the new memory limit in MEGABYTES (note that the two variables have different units). Result: Users changing the file should now have instructions on how to properly apply the change to current gears.
Clone Of:
Environment:
Last Closed: 2016-08-24 19:42:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1381713 0 None None None Never
Red Hat Product Errata RHSA-2016:1773 0 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 2.2.10 security, bug fix, and enhancement update 2016-08-24 23:41:18 UTC

Description Josep 'Pep' Turro Mauri 2015-02-26 17:57:12 UTC
Description of problem:

If you update resource_limits.conf and change memory_limit_in_bytes

Version-Release number of selected component (if applicable):
current OSE versions

How reproducible:
Always

Steps to Reproduce:
1. Create some gears on a node
2. Change memory_limit_in_bytes in /etc/openshift/resource_limits.conf
3. Follow the documented steps (e.g. in the initial comments of the file) to apply the changes

Actual results:

Existing gears don't get their OPENSHIFT_GEAR_MEMORY_MB env var updated.

Expected results:

It should be possible to update this env var. Right now the only known method is something like:

  # find /var/lib/openshift -name OPENSHIFT_GEAR_MEMORY_MB | sed -i -e 's/oldMB/newMB/'

Additional info:

The variable is used in some places, e.g. Java based cartridges (EAP and EWS) calculate the JVM's heap size based on JVM_HEAP_RATIO * OPENSHIFT_GEAR_MEMORY_MB. Failing to update the env var after updating the memory limits means this calculation is not accurate.

Comment 1 Jhon Honce 2015-03-16 17:02:52 UTC
*** Bug 1202466 has been marked as a duplicate of this bug. ***

Comment 6 Rory Thrasher 2016-07-11 17:22:17 UTC
So in order to make this workaround more visible, we're going to add the it to both the instructions at the beginning of resource_limits.conf, as well as to the Administrative guide <https://access.redhat.com/documentation/en-US/OpenShift_Enterprise/2/html-single/Administration_Guide/index.html#Adding_New_Gear_Profiles>.

We're also recommending using the following command instead of the sed command currently recommended in the workaround, where 512 should be replaced with the new value in megabytes.

for i in /var/lib/openshift/*/.env/OPENSHIFT_GEAR_MEMORY_MB; do echo 512 > "$i"; done


I'll be opening a PR shortly with the change the resource_limits.conf file.  A docs bug for the admin guide has been opened here: https://bugzilla.redhat.com/show_bug.cgi?id=1354609

Comment 9 openshift-github-bot 2016-07-20 00:33:00 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/505b7922fe1d5f8a69e2537e535bdb035d2e3933
resource_limits.conf: adds new update instruction

Bug 1196783
https://bugzilla.redhat.com/show_bug.cgi?id=1196783

When the memory_limit_in_bytes variable in resource_limits.conf is updated, the
OPENSHIFT_GEAR_MEMORY_MB env variable does not get updated for existing gears.
Now there is an additional note for users to run a workaround to update that
variable for existing gears.

Comment 14 Johnny Liu 2016-08-09 05:29:13 UTC
Re-test this bug with rubygem-openshift-origin-node-1.38.6.3-1.el6op.noarch in 2.2/2016-08-08.1 puddle, seem like the PR is not merged into the RPM package.

Comment 16 Johnny Liu 2016-08-10 02:05:41 UTC
Installer script - openshift.sh has the following logic:

  node_profile="${CONF_NODE_PROFILE:-small}"
  node && node_profile_name="${CONF_NODE_PROFILE_NAME:-$node_profile}"
  node && node_host_type="${CONF_NODE_HOST_TYPE:-m3.xlarge}"

configure_node()
{
  local resrc='/etc/openshift/resource_limits.conf'
  if [[ -f "${resrc}.${node_profile}.${node_host_type}" ]]
  then cp "${resrc}.${node_profile}.${node_host_type}" "$resrc"
  elif [[ -f "${resrc}.${node_profile}" ]]
  then cp "${resrc}.${node_profile}" "$resrc"
  fi
  ...
}

Though this fix is present in resource_limits.conf, not in resource_limits.conf.large.m3.xlarge, resource_limits.conf.medium.m3.xlarge, resource_limits.conf.small.m3.xlarge, resource_limits.conf.xpaas.m3.xlarge, when running installation using openshift.sh, /etc/openshift/resource_limits.conf will be overwritten, then the fix will be lost, we also need add the fix for the other resource_limits template conf file to avoid such issue.

Comment 17 openshift-github-bot 2016-08-10 18:32:07 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/1c18dea308cf2e58d81736288de352f1eea1e5be
Add new update instruction for different gear types conf

When the memory_limit_in_bytes variable in resource_limits.conf is updated, the
OPENSHIFT_GEAR_MEMORY_MB env variable does not get updated for existing gears.
Now there is an additional note for users to run a workaround to update that
variable for existing gears.

Bug 1196783
Link https://bugzilla.redhat.com/show_bug.cgi?id=1196783

Signed-off-by: Vu Dinh <vdinh>

Comment 19 Johnny Liu 2016-08-12 07:14:13 UTC
Verified this bug with rubygem-openshift-origin-node-1.38.6.4-1.el6op.noarch in 2.2/2016-08-11.1, and PASS.

Now the workaround comments are added into /etc/openshift/resource_limits.conf.

Comment 21 errata-xmlrpc 2016-08-24 19:42:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1773.html


Note You need to log in before you can comment on or make changes to this bug.