Description of problem:Warning: Unable to allocate memory for pool. Version-Release number of selected component (if applicable): Drupal 7.23 on Openshift How reproducible: See http://sal-mandospace.rhcloud.com. In addition, see https://www.openshift.com/forums/openshift/unable-to-allocate-memory-for-pool as numerous Drupal users are experiencing these errors. Steps to Reproduce: 1. Set up a Drupal app 2. Populate with a number of modules to increase memory usage upwards of 135MB 3. Begin using the site normally (editing and saving content, for example) Actual results: Expected results: Additional info: Memory limits are set for 512MB Memory, yet problems began occurring with using as little as 135MB Memory For example, at one point I received the following: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 24938 bytes) in /var/lib/openshift/52152b2f5973caa59f0006bb/app-root/data/downloads/drupal-7.23/includes/database/database.inc on line 2168 Call Stack: 0.0023 840704 POTENTIAL SOLUTION: Having no issues with previous hosts using the same code base, and being a little suspect of needing higher resources, I dug into the issue and found: https://drupal.org/node/1154448 This points to the fact that this is an APC bug which is documented on PHP.net here https://bugs.php.net/bug.php?id=58982 It appears to be a bug with APC on both PHP 5.3 and 5.4 and quite a bit of time has gone into squashing this bug to no avail. The only solution was to completely remove APC.
Hi, As far as I investigated this I think the problem is with php.ini file where we set the memory limit too low: See $OPENSHIFT_PHP_DIR/configuration/etc/php.ini: ; Maximum amount of memory a script may consume (128MB) ; http://www.php.net/manual/en/ini.core.php#ini.memory-limit memory_limit = 128M I'm not 100% sure if users can modify the 'php.ini' file and set different limits. Vojta?
Seems like the APC in your case reaches it's memory limit: apc.shm_size=64M I will investigate both apc.shm_size and apc.ttl settings and report back if we can prevent this. Increasing APC shared memory size doesn't look like the only thing to change. Assuming you're using Drupal 7 with APC module enabled, as a workaround, I suggest you changing your CACHE backends in your settings.php config file appropriately to reduce the APC memory usage: $conf['cache_default_class'] = 'DrupalDatabaseCache'; $conf['cache_class_cache'] = 'DrupalAPCCache'; $conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';
Reproducer - index.php: <?php $data = array(); for ($i = 0; $i < 100; $i++) { $data=array_merge($data,$GLOBALS); } for ($i = 0; ; $i++) { if (apc_store("foo$i", $data)) echo "$i+\n"; else echo "$i-\n"; $tmp = apc_fetch("foo$i"); }
Created attachment 799485 [details] apc.php - statistics
https://github.com/openshift/origin-server/pull/3661 - a related PR to enable custom APC settings.
Workaround (git): cd <app-git> echo "php_flag apc.cache_by_default Off" > php/.htaccess git add php/.htaccess git commit -am 'Turn off APC cache by default' git push Quick workaround on gear (next git push will overwrite this): rhc ssh <app> 'echo "php_flag apc.cache_by_default Off" > $OPENSHIFT_HOMEDIR/app-root/repo/php/.htaccess'
#8 didn't work for me. I guess in order to disable we should use another variable apc.enabled, that is: php_flag apc.enabled off Nevertheless I went to find a working solution, which doesn't disable APC. **Solution** Here is the couple of settings I've added to my .htaccess: # Give more memory. php_value memory_limit 512M # APC settings. php_flag apc.enabled on php_flag apc.cache_by_default on php_value apc.shm_segments 1 #32M per OpenShift install. php_value apc.shm_size 128M #This MUST be 0, OpenShift can have errors otherwise! php_flag apc.include_once_override off #Leave at 2M or lower. Drupal seems not having any file sizes close to 2M. php_value apc.max_file_size 2M As you could guess, the key is apc.include_once_override, which is described as EXPERIMENTAL at http://www.php.net/manual/en/apc.configuration.php#ini.apc.include-once-override - so we definitely should avoid using it at production. These settings completely get rid of annoying warnings - yay! Please review the solution - and correct the default php.ini settings in order to fix the bug for others. @assignee Please let us know when it is going to be released. **Related links** - https://drupal.org/node/807680 - apc.include_once_override caused another problem in another module. - https://bugs.php.net/bug.php?id=58469 - PHP bug discussion, concluding that apc.include_once_override should be disabled.
Forgot to mention that my solution is based on (almost stolen from) this link: http://gregrickaby.com/the-perfect-apc-configuration . Thanks Gregory for nice post, describing how to configure APC at AWS, which is very much related to our use case.
I was wrong. According to https://bugs.php.net/bug.php?id=57064 and http://www.php.net/manual/ru/ini.list.php it becomes clear that the only APC setting which could be customized via .htaccess is "apc.cache_by_default". Setting it to Off did the trick for me - though it means disabling, because "apc.filters" is nil. My vote is for increasing "apc.shm_size" globally. I am going to dig the opportunity mentioned by @Vojtech in #7.
Moving this bug back to Vojtech, as the APC was not part of the cartridge tuning. APC is specific PHP extension that might/might be not enabled or used in application. Also there are known workarounds currently.
Confirm that you can't set the above in .htaccess http://www.php.net/manual/en/apc.configuration.php so it is just a case of switching it off in .htaccess <code> # APC settings (which can be used depends on version) php_flag apc.enabled off php_flag apc.cache_by_default off </code> Does seem that the cache size just wants to be increased. But it also doesn't seem that you can change the system values - due to permissions. Guess it can just be changed in the cartridge? Or may to be a variable read from somewhere? And can confirm from experience that this is an issue with APC as it is now configured and Drupal as soon as you've got a decent number of modules enabled.
Trello cards for APC: 1. Disable/Edit APC settings manually: https://trello.com/c/1VcpoZTf 2. Disable APC automatically via DEVELOPMENT MODE: https://trello.com/c/lbjDOA6S
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/b149d7e27c485ab91ba71ed509b6b28535e14c43 Copy PHP extensions INI files to the cartridge code for maintainability - Copy RHEL PHP 5.3 RPM /etc/php.d/*.ini files - Copy RHEL PHP 5.4 SCL /opt/rh/php54/root/etc/php.d/*.ini files - Lock all the INI files by default (not editable in userspace) Motivation: - We want to be able to enable/disable PHP extensions in gears - We want to be able to edit PHP extensions settings in gears - We want consistency for all the INI files across all nodes. Currently, the INI files are marked %config(noreplace) in RPMs and may theoretically differ from node to node. PHP_INI_SCAN_DIR (PHP/PECL extensions): - Set PHP_INI_SCAN_DIR env var to scan customized INI files from ${OPENSHIFT_PHP_DIR}etc/php.d/ directory instead of the default system's scan dir (eg. /etc/php.d/) https://trello.com/c/lbjDOA6S - origin_cartridge_20 https://trello.com/c/1VcpoZTf - origin_cartridge_140 https://trello.com/c/iDrhIUof - origin_cartridge_136 https://bugzilla.redhat.com/show_bug.cgi?id=1001400 - bug 1001400
This will get addressed by https://trello.com/c/1VcpoZTf functionality during this sprint: > - APC is enabled by default > - Users can enable/disable APC by running > `rhc env-set OPENSHIFT_PHP_APC_ENABLED=false --app <app>` > and `rhc app-restart <app>` commands > > APC shared memory size: > - Formula: GEAR_MEMORY/8 > - 64M on Small gears by default > - 128M on Medium gears by default > - 256M on Large gears by default > - Users can change the APC shared memory size by running > `rhc env-set OPENSHIFT_PHP_APC_SHM_SIZE=128M --app <app>` > and `rhc app-restart <app>` commands > > New environment variables: > - $OPENSHIFT_PHP_APC_ENABLED=true/false > - $OPENSHIFT_PHP_APC_SHM_SIZE={64,128,256}M