Bug 1082963

I suspect these lines are the problem:

cartridges/openshift-origin-cartridge-php/usr/5.4/etc/php.d/locked-mongo.ini:extension=mongo.so

cartridges/openshift-origin-cartridge-php/usr/5.3/etc/php.d/locked-mongo.ini:extension=mongo.so

We should probably remove them until we ship the SCL mongo cartridge.

Just noting that this might be fixable using oo-erb, similar to what's going on here:
https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-php/usr/5.4/etc/php.d/apc.ini.erb
https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-php/bin/control#L27

Also: The problem is that there are a number of php extension-related .ini files that are now being shipped with the php cartridge (as opposed to the relevant php extension package). These .ini files make the assumption that the related extension is present and tries to load it, leading to the error. On a recent 2.1 install, just running "php" from within a php-5.3 gear gives this output: 

[cloud-user@broker phpapp53]$ rhc ssh phpapp53
Connecting to 533c62ac6892dfe8c5000015.com ...

    *********************************************************************

    You are accessing a service that is for use only by authorized users.
    If you do not have authorization, discontinue use at once.
    Any use of the services is subject to the applicable terms of the
    agreement which can be found at:
    https://www.openshift.com/legal

    *********************************************************************

    Welcome to OpenShift shell

    This shell will assist you in managing OpenShift applications.

    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!
    Shell access is quite powerful and it is possible for you to
    accidentally damage your application.  Proceed with care!
    If worse comes to worst, destroy your application with "rhc app delete"
    and recreate it
    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!

    Type "help" for more info.


[phpapp53-jltest.example.com 533c62ac6892dfe8c5000015]\> php
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/apc.so' - /usr/lib64/php/modules/apc.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/bcmath.so' - /usr/lib64/php/modules/bcmath.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/dom.so' - /usr/lib64/php/modules/dom.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/gd.so' - /usr/lib64/php/modules/gd.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/imagick.so' - /usr/lib64/php/modules/imagick.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/imap.so' - /usr/lib64/php/modules/imap.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/intl.so' - /usr/lib64/php/modules/intl.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/mbstring.so' - /usr/lib64/php/modules/mbstring.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/mcrypt.so' - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/memcached.so' - /usr/lib64/php/modules/memcached.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/mongo.so' - /usr/lib64/php/modules/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/posix.so' - /usr/lib64/php/modules/posix.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/soap.so' - /usr/lib64/php/modules/soap.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/sysvmsg.so' - /usr/lib64/php/modules/sysvmsg.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/sysvsem.so' - /usr/lib64/php/modules/sysvsem.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/sysvshm.so' - /usr/lib64/php/modules/sysvshm.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/tidy.so' - /usr/lib64/php/modules/tidy.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/wddx.so' - /usr/lib64/php/modules/wddx.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/xmlreader.so' - /usr/lib64/php/modules/xmlreader.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/xmlwriter.so' - /usr/lib64/php/modules/xmlwriter.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/xsl.so' - /usr/lib64/php/modules/xsl.so: cannot open shared object file: No such file or directory in Unknown on line 0

(In reply to John W. Lamb from comment #4)
> Also: The problem is that there are a number of php extension-related .ini
> files that are now being shipped with the php cartridge (as opposed to the
> relevant php extension package). These .ini files make the assumption that
> the related extension is present and tries to load it, leading to the error.
> On a recent 2.1 install, just running "php" from within a php-5.3 gear gives
> this output: 
If you deploy this gear on node that optional metadata package is already installed, most of these missing dynamic libraries could be loaded except mongo.so, I thought your gear is a recommended node.

A php module can be detected inside a cart using php's "dl()" function, like so:
  php -n -r "dl('xsl.so') or exit(1);"
The -n option prevents php from parsing the config file, and -r executes its string argument as php code. This command will return exit code 1 if the module can't be found, 0 otherwise.

I discussed this with vvitek and mfojtik, and we found 3 possible solutions:

1) Scan for available modules at every app start/restart
2) Checking available modules in the php cartridge RPM %install or %post, which would remove the extra .ini files from the installed cartridge itself
3) Scan for available modules at cartridge install time, i.e. when /var/lib/openshift/APP_DIR/php/configuration/etc/php.d/ is populated

Further discussion is needed

I would definitely prefer #1.  Are there concerns about startup latency?

With #3 would the user be able to manually fix the ini files if a new dependency they need was added after their application was created?

(In reply to Brenton Leanhardt from comment #7)
> I would definitely prefer #1.  Are there concerns about startup latency?

I think that's the main issue.

> With #3 would the user be able to manually fix the ini files if a new
> dependency they need was added after their application was created?

Re: #3, I suggested using a marker file (like force_clean_build in the ruby cart) to allow the user to force a re-scan of available modules, but Michal objected.



4th option: Since we provide a known set of avaliable PHP modules, the user could select which php modules are enabled by setting env vars. Then no .ini file/PHP module scan is needed. If the module isn't available on the node, it just won't work.

Naive implementation of method #1 here: https://github.com/dobbymoodge/origin-server/commit/d831ab5b70330dbf350e4df68068543a8c0c3e77

This adds about 3 to 4 seconds to gear restart for an otherwise unmodified php app. Presumably this is too slow. The implementation may be able to be cleaned up and optimized a little bit, but it's unlikely that this approach will be sufficiently quick.

Just thinking a bit more about this.  Another approach would be to inject additional logging into OSE.  If we don't see the optional-php metapackage install we print something like: "Info: For compatibility with OpenShift Online runtime dependencies contact your OpenShift system administrator and ask for openshift-origin-cartridge-depedencies-optional-php to be installed".

I don't really like this option but if the other alternatives are too slow it is indeed an option.  Anything that we simply don't ship in OSE (say php-mcrypt) we would remove from any referenced ini file.

(In reply to Brenton Leanhardt from comment #10)
> Just thinking a bit more about this.  Another approach would be to inject
> additional logging into OSE.  If we don't see the optional-php metapackage
> install we print something like: "Info: For compatibility with OpenShift
> Online runtime dependencies contact your OpenShift system administrator and
> ask for openshift-origin-cartridge-depedencies-optional-php to be installed".
> 
> I don't really like this option but if the other alternatives are too slow
> it is indeed an option.  Anything that we simply don't ship in OSE (say
> php-mcrypt) we would remove from any referenced ini file.

This doesn't address the issue that the ini files packaged with the php cart would still reference the missing modules.

Correct, I was merely stating that if no performant solution was found we could simply print a warning.  Alternatively we could have the php cartridge require the optional dependency metapackage for now (as much as I would hate that) and then remove any ini files for dependencies we don't ship in OSE.  I would really prefer a dynamic soluation though if one can be found.

(In reply to Brenton Leanhardt from comment #12)
> Correct, I was merely stating that if no performant solution was found we
> could simply print a warning.  Alternatively we could have the php cartridge
> require the optional dependency metapackage for now (as much as I would hate
> that) and then remove any ini files for dependencies we don't ship in OSE. 
> I would really prefer a dynamic soluation though if one can be found.

https://github.com/dobbymoodge/origin-server/commit/1cad75a15af4c6529413537643f84f510eec688e

This solution seems to be no slower than the current php cart implementation:

Unmodified PHP cart:
  [root@ip-10-187-28-121 ~]# time for ii in $(seq 1 30) ; do rhc app-restart pt1 ; done
  real    1m6.751s
  user    0m13.293s
  sys     0m8.722s

envscan PHP cart:
  [root@ip-10-187-28-121 ~]# time for ii in $(seq 1 30) ; do rhc app-restart envscan ; done
  real    1m6.685s
  user    0m13.345s
  sys     0m8.612s

It's a little cumbersome, as it defaults to all modules being disabled. It depends on the ENABLED_PHP_MODULES environment variable being set, and it expects a space-or-comma separated list of modules in that variable:

  rhc env-set -a appname \
    -e ENABLED_PHP_MODULES="bcmath,curl,dom,fileinfo,imagick,intl,json"

If this approach is approved, we can figure out a nicer way to implement default enabled carts (maybe add a DISABLED_PHP_MODULES and only modify .ini files mentioned in either one or the other *_PHP_MODULES vars?), etc.

An updated version of the env-var based solution is here: https://github.com/dobbymoodge/origin-server/tree/test_php_env_scan

This scans for available modules at install and enables them by writing to cart env vars like "OPENSHIFT_PHP_DEFAULT_{MODULENAME}". These defaults can be overridden via user env vars like "OPENSHIFT_PHP_{MODULENAME}".

We may (probably?) want to update this to take advantage of the fact that user env vars can override cart env vars of the same name.

Some numbers - the changes to php/bin/control add on average 0.2s to app restart, though some of that could be noise. Will look into further optimization tomorrow.

[root@ip-10-187-28-121 ~]# for jj in pt1 envscan2 ; do echo "--- $jj" ; time for ii in $(seq 1 60) ; do rhc app-restart $jj > /dev/null ; done ; done                                                                                                                            
--- pt1

real    2m6.348s
user    0m26.379s
sys     0m16.514s
--- envscan2

real    2m20.827s
user    0m26.583s
sys     0m16.526s

Following a discussion with vvitek, we know that only the cart's "php.d" directory is scanned for .inis, so any PHP modules installed that don't have corresponding .inis packaged with the PHP cartridge won't be available for OpenShift users. Copying the system-wide .inis installed by these modules into the cart will ameliorate the issue.

The install-time module scan and the cart/user env var-triggered module-enabling routines need to check the appropriate (SCL or non-SCL) system-wide "php.d" directories for any module .inis that don't appear in the cartridge directory either as .ini files or as .erb templates, and create the cart env var (for php/bin/install) or copy the .ini into the php/configuration/etc/php.d directory (for php/bin/control start) as needed.

Would the system .ini cp be fast enough to happen on every restart?  I think if a new dependency is added it's reasonable for an application to require a restart.

(In reply to Brenton Leanhardt from comment #16)
> Would the system .ini cp be fast enough to happen on every restart?  I think
> if a new dependency is added it's reasonable for an application to require a
> restart.

No, that's why the .ini scan is in php/bin/install, and the env-var check is in php/bin/control.

I've updated the test branch (https://github.com/dobbymoodge/origin-server/tree/test_php_env_scan) here:
https://github.com/dobbymoodge/origin-server/commit/58e50582c4b151ecc1eb333d47a492b9f7010693

This change includes a (not very pretty yet) scan for system-wide php.d/*.ini files as well as the php cart-packaged ini files. This also replaces the OPENSHIFT_PHP_DEFAULT_{MODULE} env var scheme with a shared cart/user env var OPENSHIFT_PHP_{MODULE}_ENABLE scheme.

This PR ought to fix the issue, pending review: https://github.com/openshift/origin-server/pull/5307

I am experiencing this issue on new OSE 2.1 nodes with openshift-origin-cartridge-dependencies-optional-php installed. 

Is a temporary fix available to disable loading this module while we wait for a patch?

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/02b8866be70322f4a0c0ac255479c0871e0fa4fb
php cart: dynamic, controllable php.d seeding

Bug 1082963
https://bugzilla.redhat.com/show_bug.cgi?id=1082963

Scan the cart-provided and system `php.d` directories to determine what
php modules are available, and enable them by appending the `.ini`
contents into the file `openshift.ini`, located in deployed cartridge's
`php.d` directory.

Allow users to enable or disable php modules by setting
`OPENSHIFT_PHP_{MODULENAME}_ENABLE=true/false`. Modules are **disabled
by default**, so modules installed after an app is deployed will have to
be enabled by setting the appropriate environment variable via `rhc env-set`.

Additional changes/squashed commits

* Obtain PHP version from ident variable `OPENSHIFT_PHP_IDENT`

* `php_module_scan` function `scan_for_php_modules` depended on
`OPENSHIFT_PHP_VERSION` being set, but this isn't necessarily going to
be the case.

* Amended `cartridge-lifecycle-php.feature` tests to include checks for
enable/disable PHP module via user env var

* Added needed step to `cartridge-php_steps.rb`

Waiting on related bugfix for https://bugzilla.redhat.com/show_bug.cgi?id=1108017 before cherry-picking into enterprise-2.1

Fix submitted as https://github.com/openshift/enterprise-server/pull/305

Fix includes supporting change to enterprise devenv tooling: https://github.com/openshift/enterprise/pull/89

Download the package ' openshift-origin-cartridge-php-1.23.3.5-1.el6op', update the env and check the issue.

1.create the php app on recommended node
#rhc app create sphp54 php-5.4
2.login the app ,run the command
>php -S  $OPENSHIFT_PHP_IP:8000
[Sun Jun 29 19:06:20 2014] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/rh/php54/root/usr/lib64/php/modules/apc.so' - /opt/rh/php54/root/usr/lib64/php/modules/apc.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Sun Jun 29 19:06:20 2014] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/rh/php54/root/usr/lib64/php/modules/bcmath.so' - /opt/rh/php54/root/usr/lib64/php/modules/bcmath.so: cannot open shared object file: No such file or directory in Unknown on line 0
<--snip-->

Check the package files
#rpm -ql openshift-origin-cartridge-php-1.23.3.5-1.el6op|grep php_module_scan

Can not found the php_module_scan, so move the issue to ASSIGNED

After upgrading the package did you run the following steps?:

1) install the new cartridge with oo-admin-cartridge
2) oo-admin-ctl-cartridge -c import-node --activate --obsolete
3) oo-admin-ctl-cartridge -c migrate
4) oo-admin-upgrade archive
5) oo-admin-upgrade upgrade-node --version 2.1.3

http://etherpad.corp.redhat.com/puddle-2-1-2014-06-30

Sorry about any confusion the missing puddle caused earlier; this updated puddle should do the trick.

Very sorry, I forgot to do these operations.

Check on puddle [2.1.z/2014-06.30.3]
Scenarios 1:
1.create an app on optional node
#rhc app create mphp54 php-5.4 -g medium 
2.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
The output contains soap, apc modules and so on

Scenarios 2:
1.create an app on recommended node
#rhc app create sphp54 php-5.4 
2.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
The output doesn't contain soap, apc modules.

Scenarios 3:
1.Install the apc and soap on recommended node
2.create an app on recommended node
#rhc app create sphp541 php-5.4 
3.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
The output contains soap, apc modules.
4. restart the 'sphp54' app 
3.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
The output contains soap, apc modules.

Scenarios 4:
1.remove the apc on the recommended node
2.create an app on recommended node
#rhc app create sphp542 php-5.4 
3.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
the output contain soap, not apc
4.set the env for the sphp541 
#rhc env set -e OPENSHIFT_PHP_SOAP_ENABLED=false -a sphp541 
5.restart the sphp541
6.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
The output does not contains soap, apc modules.
7.set the env for the sphp541 
#rhc env set -e OPENSHIFT_PHP_APC_ENABLED=true -a sphp54
8.restart the sphp54
9.login the app and check the openshift.ini
>cat php/configuration/etc/php.d/openshift.ini 
The output contains soap, not apc 
10.run the command on the app
> php -S  $OPENSHIFT_PHP_IP:8000                
[Mon Jun 30 19:23:21 2014] Failed to listen on 127.8.62.1:8000 (reason: Address already in use)
No warning message is given out.

Check the php-5.3 cartridge according to the above scenarios

All results meet the expect, so move the issue to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0878.html