Bug 1001400 - Unable to allocate memory for pool. Possible APC bug.
Unable to allocate memory for pool. Possible APC bug.
Product: OpenShift Online
Classification: Red Hat
Component: Image (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Vojtech Vitek
libra bugs
: SupportQuestion, UpcomingRelease
Depends On:
  Show dependency treegraph
Reported: 2013-08-26 20:57 EDT by Sparky
Modified: 2015-05-14 20:34 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-02-17 14:33:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
apc.php - statistics (44.96 KB, text/php)
2013-09-18 12:48 EDT, Vojtech Vitek
no flags Details

  None (edit)
Description Sparky 2013-08-26 20:57:44 EDT
Description of problem:Warning: Unable to allocate memory for pool.

Version-Release number of selected component (if applicable): Drupal 7.23 on Openshift

How reproducible: See http://sal-mandospace.rhcloud.com. In addition, see https://www.openshift.com/forums/openshift/unable-to-allocate-memory-for-pool as numerous Drupal users are experiencing these errors.

Steps to Reproduce:
1. Set up a Drupal app
2. Populate with a number of modules to increase memory usage upwards of 135MB
3. Begin using the site normally (editing and saving content, for example)

Actual results: 

Expected results:

Additional info: Memory limits are set for 512MB Memory, yet problems began occurring with using as little as 135MB Memory

For example, at one point I received the following: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 24938 bytes) in /var/lib/openshift/52152b2f5973caa59f0006bb/app-root/data/downloads/drupal-7.23/includes/database/database.inc on line 2168 Call Stack: 0.0023 840704


Having no issues with previous hosts using the same code base, and being a little suspect of needing higher resources, I dug into the issue and found:


This points to the fact that this is an APC bug which is documented on PHP.net here https://bugs.php.net/bug.php?id=58982

It appears to be a bug with APC on both PHP 5.3 and 5.4 and quite a bit of time has gone into squashing this bug to no avail. The only solution was to completely remove APC.
Comment 1 Michal Fojtik 2013-09-06 08:57:37 EDT

As far as I investigated this I think the problem is with php.ini file where we set the memory limit too low:

See $OPENSHIFT_PHP_DIR/configuration/etc/php.ini:

; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
memory_limit = 128M

I'm not 100% sure if users can modify the 'php.ini' file and set different limits. Vojta?
Comment 2 Vojtech Vitek 2013-09-06 09:41:04 EDT
Seems like the APC in your case reaches it's memory limit:

I will investigate both apc.shm_size and apc.ttl settings and report back if we can prevent this. Increasing APC shared memory size doesn't look like the only thing to change.

Assuming you're using Drupal 7 with APC module enabled, as a workaround, I suggest you changing your CACHE backends in your settings.php config file appropriately to reduce the APC memory usage:

$conf['cache_default_class'] = 'DrupalDatabaseCache';
$conf['cache_class_cache'] = 'DrupalAPCCache';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';
Comment 5 Vojtech Vitek 2013-09-18 12:40:41 EDT
Reproducer - index.php:
$data = array();
for ($i = 0; $i < 100; $i++) {
for ($i = 0; ; $i++) {
  if (apc_store("foo$i", $data))
    echo "$i+\n";
    echo "$i-\n";
  $tmp = apc_fetch("foo$i");
Comment 6 Vojtech Vitek 2013-09-18 12:48:33 EDT
Created attachment 799485 [details]
apc.php - statistics
Comment 7 Vojtech Vitek 2013-09-18 12:58:21 EDT
https://github.com/openshift/origin-server/pull/3661 - a related PR to enable custom APC settings.
Comment 8 Vojtech Vitek 2013-09-18 13:33:24 EDT
Workaround (git):
cd <app-git>
echo "php_flag apc.cache_by_default Off" > php/.htaccess
git add php/.htaccess
git commit -am 'Turn off APC cache by default'
git push

Quick workaround on gear (next git push will overwrite this):
rhc ssh <app> 'echo "php_flag apc.cache_by_default Off" > $OPENSHIFT_HOMEDIR/app-root/repo/php/.htaccess'
Comment 9 patchranger 2013-09-28 10:58:38 EDT
#8 didn't work for me.
I guess in order to disable we should use another variable apc.enabled, that is:

php_flag apc.enabled off

Nevertheless I went to find a working solution, which doesn't disable APC.

Here is the couple of settings I've added to my .htaccess:

# Give more memory.
php_value memory_limit 512M
# APC settings.
php_flag apc.enabled on
php_flag apc.cache_by_default on
php_value apc.shm_segments 1
#32M per OpenShift install.
php_value apc.shm_size 128M
#This MUST be 0, OpenShift can have errors otherwise!
php_flag apc.include_once_override off
#Leave at 2M or lower. Drupal seems not having any file sizes close to 2M.
php_value apc.max_file_size 2M

As you could guess, the key is apc.include_once_override, which is described as EXPERIMENTAL at http://www.php.net/manual/en/apc.configuration.php#ini.apc.include-once-override - so we definitely should avoid using it at production.

These settings completely get rid of annoying warnings - yay!
Please review the solution - and correct the default php.ini settings in order to fix the bug for others.
@assignee Please let us know when it is going to be released.

**Related links**
- https://drupal.org/node/807680 - apc.include_once_override caused another problem in another module.
- https://bugs.php.net/bug.php?id=58469 - PHP bug discussion, concluding that apc.include_once_override should be disabled.
Comment 10 patchranger 2013-09-28 11:05:50 EDT
Forgot to mention that my solution is based on (almost stolen from) this link: http://gregrickaby.com/the-perfect-apc-configuration . Thanks Gregory for nice post, describing how to configure APC at AWS, which is very much related to our use case.
Comment 12 patchranger 2013-10-17 03:25:06 EDT
I was wrong. According to https://bugs.php.net/bug.php?id=57064 and http://www.php.net/manual/ru/ini.list.php it becomes clear that the only APC setting which could be customized via .htaccess is "apc.cache_by_default".
Setting it to Off did the trick for me - though it means disabling, because "apc.filters" is nil.
My vote is for increasing "apc.shm_size" globally. I am going to dig the opportunity mentioned by @Vojtech in #7.
Comment 13 Michal Fojtik 2013-10-22 09:40:53 EDT
Moving this bug back to Vojtech, as the APC was not part of the cartridge tuning. APC is specific PHP extension that might/might be not enabled or used in application. Also there are known workarounds currently.
Comment 14 ekes 2014-02-03 12:00:16 EST
Confirm that you can't set the above in .htaccess http://www.php.net/manual/en/apc.configuration.php so it is just a case of switching it off in .htaccess 
  # APC settings (which can be used depends on version)
  php_flag apc.enabled off
  php_flag apc.cache_by_default off
Does seem that the cache size just wants to be increased. But it also doesn't seem that you can change the system values - due to permissions. Guess it can just be changed in the cartridge? Or may to be a variable read from somewhere? 

And can confirm from experience that this is an issue with APC as it is now configured and Drupal as soon as you've got a decent number of modules enabled.
Comment 15 Vojtech Vitek 2014-02-13 15:58:33 EST
Trello cards for APC:
1. Disable/Edit APC settings manually: https://trello.com/c/1VcpoZTf
2. Disable APC automatically via DEVELOPMENT MODE: https://trello.com/c/lbjDOA6S
Comment 16 openshift-github-bot 2014-03-29 19:42:47 EDT
Commit pushed to master at https://github.com/openshift/origin-server

Copy PHP extensions INI files to the cartridge code for maintainability

- Copy RHEL PHP 5.3 RPM /etc/php.d/*.ini files
- Copy RHEL PHP 5.4 SCL /opt/rh/php54/root/etc/php.d/*.ini files
- Lock all the INI files by default (not editable in userspace)

- We want to be able to enable/disable PHP extensions in gears
- We want to be able to edit PHP extensions settings in gears
- We want consistency for all the INI files across all nodes.
  Currently, the INI files are marked %config(noreplace) in RPMs
  and may theoretically differ from node to node.

- Set PHP_INI_SCAN_DIR env var to scan customized INI files
  from ${OPENSHIFT_PHP_DIR}etc/php.d/ directory
  instead of the default system's scan dir (eg. /etc/php.d/)

https://trello.com/c/lbjDOA6S - origin_cartridge_20
https://trello.com/c/1VcpoZTf - origin_cartridge_140
https://trello.com/c/iDrhIUof - origin_cartridge_136
https://bugzilla.redhat.com/show_bug.cgi?id=1001400 - bug 1001400
Comment 17 Vojtech Vitek 2014-03-30 08:27:07 EDT
This will get addressed by https://trello.com/c/1VcpoZTf functionality during this sprint:

> - APC is enabled by default
> - Users can enable/disable APC by running
>   `rhc env-set OPENSHIFT_PHP_APC_ENABLED=false --app <app>`
>   and `rhc app-restart <app>` commands
> APC shared memory size:
> - Formula: GEAR_MEMORY/8
>   - 64M on Small gears by default
>   - 128M on Medium gears by default
>   - 256M on Large gears by default
> - Users can change the APC shared memory size by running
>   `rhc env-set OPENSHIFT_PHP_APC_SHM_SIZE=128M --app <app>`
>   and `rhc app-restart <app>` commands
> New environment variables:

Note You need to log in before you can comment on or make changes to this bug.