Bug 1162950 - OPENSHIFT_PHP_APC_SHM_SIZE is 32M for all gear size in INT, OPENSHIFT_GEAR_MEMORY_MB is correct.
Summary: OPENSHIFT_PHP_APC_SHM_SIZE is 32M for all gear size in INT, OPENSHIFT_GEAR_ME...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Image
Version: 2.x
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.x
Assignee: Vojtech Vitek
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-12 03:55 UTC by Lei Zhang
Modified: 2015-05-15 00:46 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-18 16:53:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Lei Zhang 2014-11-12 03:55:04 UTC
Description of problem:
Found the default OPENSHIFT_PHP_APC_SHM_SIZE is 32M for all gear size php-5.3 app.

Version-Release number of selected component (if applicable):
INT(devenv_5294)

How reproducible:
always

Steps to Reproduce:
1.Create one small,medium and large gear size app
#rhc app create smallapp php-5.3 -g smalll
#rhc app create smallapp php-5.3 -g medium
#rhc app create smallapp php-5.3 -g large
2.Check the OPENSHIFT_PHP_APC_SHM_SIZE of the app
#php -i |grep apc
3.

Actual results:
Found OPENSHIFT_PHP_APC_SHM_SIZE is 32M for all gear size

Expected results:
OPENSHIFT_PHP_APC_SHM_SIZE should be 64M for small gear, 128M for medium gear and 256M for large gear size app


Additional info:
Checked and found OPENSHIFT_GEAR_MEMORY_MB in small/medium/large gear size app
small OPENSHIFT_GEAR_MEMORY_MB=512
medium OPENSHIFT_GEAR_MEMORY_MB=1024
large OPENSHIFT_GEAR_MEMORY_MB=2048

Comment 1 Ben Parees 2014-11-12 19:19:01 UTC
code hasn't changed this sprint and appears to work in online.

I also experimented with changing the memory setting on a devenv and creating the cart and saw apc.shm_size respond accordingly.

QE can you confirm this is an issue and/or check stage?

Comment 3 Vojtech Vitek 2014-11-18 17:26:03 UTC
> for i in small medium large; do
>   rhc app create php$i php-5.3 -g $i
>   rhc ssh php$i 'php -i | grep apc.shm_size'
>   rhc ssh php$i 'echo $OPENSHIFT_GEAR_MEMORY_MB'
> done
>
> Application Options
> -------------------
> Domain:     vvitek
> Cartridges: php-5.3
> Gear Size:  small
> Scaling:    no
> 
> apc.shm_size => 64M => 64M
> 512
>
> Application Options
> -------------------
> Domain:     vvitek
> Cartridges: php-5.3
> Gear Size:  medium
> Scaling:    no
> 
> apc.shm_size => 64M => 64M
> 512
>
> Application Options
> -------------------
> Domain:     vvitek
> Cartridges: php-5.3
> Gear Size:  large
> Scaling:    no
> 
> apc.shm_size => 64M => 64M
> 
> 512

I can confirm this on latest devenv. The problem is caused by $OPENSHIFT_GEAR_MEMORY_MB env var, which is still set to 512.

Comment 4 Vojtech Vitek 2014-11-18 19:24:34 UTC
Lei Zhang, can you confirm this cannot be reproduced in PROD? Otherwise, we'd need migration, once fixed. Thanks!

Comment 5 Vojtech Vitek 2014-11-18 19:56:15 UTC
Just tested on PROD. Can't reproduce there.

> $ rhc app create php$i php-5.3 -g $i
>
> $rhc ssh php$i 'php -i | grep apc.shm_size'
> apc.shm_size => 256M => 256M
>
> $ rhc ssh php$i 'echo $OPENSHIFT_GEAR_MEMORY_MB'
> 2048

Comment 6 Jhon Honce 2014-11-18 20:42:30 UTC
Works as expected on devenv_5305.

From resource_limits.conf:

memory_limit_in_bytes=536870912       # 512MB
memory_memsw_limit_in_bytes=641728512 # 512M + 100M (100M swap)

create application
#echo $OPENSFHIT_GEAR_MEMORY_DB
512

Edit resource_limits.conf:
memory_limit_in_bytes=1073741824       # 1024MB
memory_memsw_limit_in_bytes=1178599424 # 1024M + 100M (100M swap)

# restart mcollective

create application
#echo $OPENSFHIT_GEAR_MEMORY_DB
1024

If other environments are returning wrong values, verify the values in resource_limits.conf with Ops.  Ops is responsible for these files after converting them to be ERB processed via puppet.

Comment 7 Ben Parees 2014-11-18 20:44:52 UTC
i think the reported issue here was if you create a large gear on a devenv w/o changing the resource file (which, strangely, works, i'm not sure sure).  in that case the ENV variable is still 512.

Comment 8 Ben Parees 2014-11-18 20:45:17 UTC
^^ "i'm not sure why"

Comment 9 Jhon Honce 2014-11-18 20:59:20 UTC
node_profile in resource_limits.conf is reported from the Node to the Broker. It is up to the Broker to allocate gears to the correct Node with the given node_profile.

Comment 10 Ben Parees 2014-11-18 21:45:30 UTC
So a couple clarifications:

1) devenvs allow creation of small, medium, and large gears, but will always set the memory to 512megs.

2) the original bug report was against INT which does not have the behavior described in #1.  It should have different memory sizes for small/medium/large, and the original report shows that in the env variable value.  The original issue really had nothing to do w/ the memory env variable, it had to do w/ the php setting not being set correctly.  The bug title is wrong (i'm going to change it now)

3) stage/prod don't have this bug.

4) devenvs appear to work correctly in that if you change the memory size in the resource file, the env variable changes accordingly.  (Jhon's comment #6 and my comment #1)

So all in all it seems we can't recreate this, though i don't think anyone except QE has actually tried in INT.

Comment 11 Ben Parees 2014-11-18 22:29:31 UTC
to save QE some time....i'm still seeing this in INT, it's pretty bizarre:


[medapp-bparees.int.rhcloud.com 546bc59becb8d47bb7000023]\> php -i |grep apc.shm
apc.shm_segments => 1 => 1
apc.shm_size => 32M => 32M

[medapp-bparees.int.rhcloud.com 546bc59becb8d47bb7000023]\> env | grep MB
OPENSHIFT_GEAR_MEMORY_MB=1024

and the code in question is:

apc.shm_size=<%= (ENV['OPENSHIFT_PHP_APC_SHM_SIZE'] =~ /^[0-9]+[MG]$/) ? ENV['OPENSHIFT_PHP_APC_SHM_SIZE'] : ((ENV['OPENSHIFT_GEAR_MEMORY_MB'].to_i/8).to_i.to_s+"M") %>


and OPENSHIFT_PHP_APC_SHM_SIZE is not set.

Comment 12 Ben Parees 2014-11-18 22:30:27 UTC
as best i can tell, apc.ini.erb isn't being processed at all, and we're just picking up a default value from php.

Comment 14 Vojtech Vitek 2014-11-24 14:58:16 UTC
I couldn't reproduce this on STG last Friday. INT was unavailable, as the restart was in progress. I'm waiting on new INT to become available right now.

Comment 17 Liang Xia 2014-11-25 12:12:59 UTC
Sorry for the confuse. Let me make it clear here.

The issue ONLY happens on INT.
STG does NOT have this issue.
PROD should be OK (we checked the small gear size since our account only have that capacity, it shows the correct value, apc.shm_size => 64M => 64M, OPENSHIFT_GEAR_MEMORY_MB=512)

INT:
  values of apc.shm_size are *incorrect*, always 32M
    apc.shm_size => 32M => 32M  (small)
    apc.shm_size => 32M => 32M  (medium)
    apc.shm_size => 32M => 32M  (large)
  values of OPENSHIFT_GEAR_MEMORY_MB are correct
    OPENSHIFT_GEAR_MEMORY_MB=512  (small)
    OPENSHIFT_GEAR_MEMORY_MB=1024 (medium)
    OPENSHIFT_GEAR_MEMORY_MB=2048 (large)

STG:
  values of apc.shm_size are correct
    apc.shm_size =>  64M =>  64M  (small)
    apc.shm_size => 128M => 128M  (medium)
    apc.shm_size => 256M => 256M  (large)
  values of OPENSHIFT_GEAR_MEMORY_MB are correct
    OPENSHIFT_GEAR_MEMORY_MB=512  (small)
    OPENSHIFT_GEAR_MEMORY_MB=1024 (medium)
    OPENSHIFT_GEAR_MEMORY_MB=2048 (large)

Comment 20 Vojtech Vitek 2014-11-25 23:04:38 UTC
Seems like there's APCu RPM installed instead of APC RPM on INT.

The APCu RPM effectively obsoletes the APC RPM.. which is super weird, as it comes different (and thus not OpenShift-customized) .ini file.
> php-pecl-apcu provides "php-pecl-apc = 4.0.7" 

I'm going to downgrade to the "older & stable" APC. I guess we don't want to upgrade all php-5.3 gears to APCu at all, unless we have really strong arguments for this step. If doing so, I suspect there might be really dangerous side effects / backward incompatibilities.

Comment 21 Vojtech Vitek 2014-11-25 23:25:36 UTC
Seems like the APCu conflicts with APC already. Will add subsequent PR to ensure this even from our cartridge RPMs.

> Processing Conflict: php-pecl-apcu-4.0.7-1.el6.x86_64 conflicts php-pecl-apc < 4
> Error: php-pecl-apcu conflicts with php-pecl-apc-3.1.9-2.el6.x86_64

Manual steps to fix the INT, if it doesn't get fixed automatically tomorrow during the reset:
https://github.com/openshift/puppet-openshift-online/issues/975

Comment 23 Vojtech Vitek 2014-11-26 04:26:01 UTC
Once there is new INT (reset) or https://github.com/openshift/puppet-openshift-online/issues/975 is addressed, the issue should be fixed.

Comment 24 Vojtech Vitek 2014-12-01 22:18:47 UTC
https://github.com/openshift/origin-server/pull/5994 (follow-up for PHP 5.4 as well)

Comment 25 Lei Zhang 2014-12-02 05:22:10 UTC
Verified on INT(devenv_5330)

[rayzhang@ray Work]$ rhc app ssh myappsmall "php -i |grep [Aa]pc.shm_size"
apc.shm_size => 64M => 64M
[rayzhang@ray Work]$ rhc app ssh myappmedium "php -i |grep [Aa]pc.shm_size"
apc.shm_size => 128M => 128M
[rayzhang@ray Work]$ rhc app ssh myapplarge "php -i |grep [Aa]pc.shm_size"
apc.shm_size => 256M => 256M


Note You need to log in before you can comment on or make changes to this bug.