Bug 800304 - NSS module reload doesn't succeed in apache prefork process
Summary: NSS module reload doesn't succeed in apache prefork process
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: nss
Version: 6.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Elio Maldonado Batiz
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks: 782183 835616 960054 994246 840699 1067443
TreeView+ depends on / blocked
 
Reported: 2012-03-06 09:26 UTC by Steven Capper
Modified: 2018-11-27 20:07 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-05-20 17:26:20 UTC
Target Upstream Version:


Attachments (Terms of Use)
A php file that causes the problem to occur immediately (1.08 KB, text/plain)
2012-03-06 09:26 UTC, Steven Capper
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1006280 None None None Never
Red Hat Bugzilla 1317691 None None None Never

Internal Links: 1006280 1317691

Description Steven Capper 2012-03-06 09:26:03 UTC
Created attachment 567877 [details]
A php file that causes the problem to occur immediately

Description of problem:
If a site on a virtual host on a webserver runs a component that relies on nss (such as php_ldap using the ldaps://); then another site uses another component (such as php_curl using https://), then the second component fails with a module load error. 

Version-Release number of selected component (if applicable):
libcurl-7.19.7-26.el6_1.2.x86_64
nss-3.13.1-7.el6_2.x86_64
php-ldap-5.3.3-3.el6_2.6.x86_64
php-5.3.3-3.el6_2.6.x86_64
httpd-2.2.15-15.el6_2.1.x86_64

How reproducible:
Easily (please see attached php script that performs an ldaps:// query and a https:// download)

Steps to Reproduce:
1. Set up php-ldap, httpd (defaults i.e. prefork MPM) with the script below.
2. Flesh out the details (i.e. an ldap server address, search filter and base address).
3. Access the script.
  
Actual results:
One of the tests will pass. (the first one executed per apache worker process).

Expected results:
Both tests to pass. (and both tests to pass when the order of execution of them is switched).

Additional info:
The module that fails to load is "libnsspem.so".

I thought the problem was located somewhere in secmod_ModuleInit near the "if (CKR_CRYPTOKI_ALREADY_INITIALIZED == crv) {" code block.
I experimented with setting 
"static PRBool enforceAlreadyInitializedError = PR_FALSE;"
And recompiling allowed the attached php test script to work. Unfortunately when I then switched the ordering of execution of "search_ldap" and "connect_https" in the test script; it then promptly failed.

Comment 1 Steven Capper 2012-03-06 09:29:01 UTC
Forgot the ldap versions sorry:
php-ldap-5.3.3-3.el6_2.6.x86_64
openldap-2.4.23-20.el6.x86_64

Comment 3 RHEL Program Management 2012-05-03 05:09:54 UTC
Since RHEL 6.3 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 7 Bob Relyea 2012-08-21 23:59:04 UTC
So this appears to be NSS "functioning as designed". The PKCS #11 spec says that it's not permissible to open a module, fork, then continue without reinitializing the module. NSS handles this by supplying a function that applications may call if they decide to fork() after NSS has already been initialized: SECMOD_RestartModules(PRBool force);

If force is true, all modules are restarted, if it is false, only those modules which appear to need reloading will be reloaded. This means calling SECMOD_RestartModules(PR_FALSE) is always safe, even if you haven't forked.

Note: SECMOD_RestartModules() will force all existing handles to be invalid, so you can't pass key handles from the parent to the child this way.

There is an environment variable that you can set which turns off the fork_check behavior in softoken. This doesn't affect other PKCS #11 modules, however (like libpem, or vendor tokens). https://developer.mozilla.org/en-US/docs/NSS_reference/NSS_environment_variables#NSS_STRICT_NO_FORK

You can try:
   export NSS_STRICK_NO_FORK=DISABLED

And see if that affects the results.

Thing to look for:

Libraries like libldap, and possibly libcurl, may try to detect the fork() scenario themselves and try to call SECMOD_RestartModules(). It would be good to see if SECMOD_RestartModules is being called and if it is somehow failing, or if libcurl and libldap are stomping on each other trying to call SECMOD_RestartModules, and why the stomping is occuring. If SECMOD_RestartModules is not being called, how could add that call to apache in a clean way (question, is mod_nss being used, for instance?).

bob

Comment 8 Elio Maldonado Batiz 2012-10-22 18:13:22 UTC
Code inspection reveals that SECMOD_RestartModules is being called by openldap but not by libcurl.

Comment 9 Kamil Dudka 2012-10-22 19:02:37 UTC
(lib)curl itself never forks.  If an application using libcurl forks, libcurl does not care.  On which event is libcurl supposed to call SECMOD_RestartModules?

Comment 11 Kai Engert (:kaie) (inactive account) 2013-04-27 21:20:07 UTC
This bug is waiting for someone to answer Kamil's question from comment 9.

It appears it's unclear which module should be responsible to fix this issue.

Let me attempt to draw conclusions:
- Bob says, if fork, then call SECMOD_RestartModules
- Elio says, openldap calls SECMOD_RestartModules,
  so one could argue the openldap isn't the culprit
- Kamil says, libcurl never forks,
  so libcurl isn't the culprit either.

So, who is actually performing the fork? Is it php? Is it apache?

Comment 12 Kamil Dudka 2013-04-28 11:28:25 UTC
This OpenLDAP commit seems to introduce the call to SECMOD_RestartModules():

http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blobdiff;f=libraries/libldap/tls_m.c;hb=d27f458b;hpb=ba70ec8b

Should the above be also applied to curl?

SECMOD_RestartModules() is called before NSS is initialized.  Is it a safe/recommended order of the initialization steps?

How can we ensure that libcurl initialization will not invalidate OpenLDAP handles or vice versa?

Comment 13 Rich Megginson 2013-04-28 23:14:02 UTC
(In reply to comment #11)
> This bug is waiting for someone to answer Kamil's question from comment 9.
> 
> It appears it's unclear which module should be responsible to fix this issue.
> 
> Let me attempt to draw conclusions:
> - Bob says, if fork, then call SECMOD_RestartModules
> - Elio says, openldap calls SECMOD_RestartModules,
>   so one could argue the openldap isn't the culprit
> - Kamil says, libcurl never forks,
>   so libcurl isn't the culprit either.
> 
> So, who is actually performing the fork? Is it php? Is it apache?

Apache forks

Comment 14 Kai Engert (:kaie) (inactive account) 2013-04-28 23:46:38 UTC
> > So, who is actually performing the fork? Is it php? Is it apache?
> 
> Apache forks

ok.

I assume apache simply makes library calls into applications. If that's true, I guess, apache doesn't send anything like a "fork notification event" after a fork.

If there isn't, it would be a library's job to detect, that a fork has happened, and do whatever is necessary to ensure correct operation.

I further guess that comparing old and new values of "getpid" are the only way to detect that self has been forked. Correct? Or is there anything else that changes after a fork?

If above is true, the next question is: Which library should be responsible to contain the code to remember old process id, detect new pid, and re-initialize libraries?

It would be good if we were able to minimize the amount of places where such detection needs to happen. Trying to detect that in each NSS library call seems too much to ask for.

Could this detection be done inside each API that get called from apache? In other words.

I assume that apache doesn't call NSS directly - rather, that apache calls ldap APIs and curl APIs.

Let's talk about libcurl. Ideally, apache could notify each apache-module that it has loaded. Does it do that? If not, then the work has to be done completely inside libcurl.

libcurl could remember the current process ID, and whenever a libcurl API gets called, the first action is does, it compares the remembered pid, and if it's different, it calls the NSS function to restart modules.

Is above approach necessary?

Can anyone think of a simpler approach?

Comment 15 Kamil Dudka 2013-04-29 08:19:52 UTC
(In reply to comment #14)
> Can anyone think of a simpler approach?

What about fixing NSS so that no such workaround is necessary?

If NSS intends to be a well-designed library, it should survive a fork fully transparently as libcurl does.

Comment 16 Bob Relyea 2013-05-16 18:21:07 UTC
> If NSS intends to be a well-designed library,
> it should survive a fork fully transparently as libcurl does.

To be clear: This is a issue with the PKCS #11 standard. NSS needs to handle modules that are PKCS #11 compliant. NSS itself does not generate issues specifically from forking after init.

In order to deal with this situation NSS provides 2 strategies: 1) A way to recover after a fork. Once you fork, you must arrange to call SECMOD_RestartModules() in the child. Old objects will no longer be valid, but new objects will work. You don't have to shutdown and restart NSS itself. or 2) set an environment variable which tells softoken not do do the fork() check. This means you won't work with other tokens!

If there are additional things we can supply to help make this easier, I'm open to suggestions, but just saying "make it work the way I think it should" is not going to fly.

bob

Comment 17 Kamil Dudka 2013-05-16 20:10:09 UTC
(In reply to comment #16)
> To be clear: This is a issue with the PKCS #11 standard. NSS needs to handle
> modules that are PKCS #11 compliant. NSS itself does not generate issues
> specifically from forking after init.

The problem is that we (ab)use a PKCS #11 module to load certificates from files, which was not necessary with OpenSSL, and users expect it to work with NSS, too.  Users do not care which standard we use to reimplement a working stuff.

> In order to deal with this situation NSS provides 2 strategies: 1) A way to
> recover after a fork. Once you fork, you must arrange to call
> SECMOD_RestartModules() in the child. Old objects will no longer be valid,
> but new objects will work. You don't have to shutdown and restart NSS
> itself. or 2) set an environment variable which tells softoken not do do the
> fork() check. This means you won't work with other tokens!

As far as I understand:

- Calling SECMOD_RestartModules() from libcurl invalidates OpenLDAP's objects.

- Calling SECMOD_RestartModules() from OpenLDAP invalidates libcurl's objects.

- Neither OpenLDAP nor libcurl are aware of each other in the running process.

- The only library aware of both higher-level libraries is NSS, so it is a hot candidate to put the fix for this bug to.  Is not it?

Comment 19 Kamil Dudka 2013-05-21 11:13:56 UTC
Bob, could you please answer my questions from comment #12?  Thanks in advance!

Comment 20 Bob Relyea 2013-05-21 22:04:36 UTC

> Bob, could you please answer my questions from comment #12?  Thanks in advance!

Sure, sorry... here's the answers.

> Should the above be also applied to curl?

If you return from your library with NSS initialized, and you think the application my actually fork() in the interim, then you need to do restart modules().

Ideally the application should actually do this.

> SECMOD_RestartModules() is called before NSS is initialized.  Is it a 
> safe/recommended order of the initialization steps?


There are two interpretations I have of your question My first interpretation was. "Since SECMOD_RestartModules() is called before NSS is initialized.  Is it a safe/recommended order of the initialization steps?"

The second is "Is it safe to call SECMOD_RestartModules() even if I haven't initialized NSS?" I'm going to answer this one first because I think this is really what you are asking.

In general, NSS calls should not be made when NSS has not been initialized, but SECMOD_RestartModules() does check to see that the module lock has been created (which implies NSS has been initialized), so you are likely to get an error rather than a crash if you call SECMOD_RestartModules() without NSS being initialized.

If you really meant:  "Since SECMOD_RestartModules() is called before NSS is initialized.  Is it a safe/recommended order of the initialization steps?"

Then no, SECMOD_RestartModules() only needs to be called if NSS has already been initialized. If NSS has not be initialized, there is no need to call this over a fork().

So, for instance, Say you do:


NSS_Initialize()
.
.
.
.
.

fork()

{ at this point you would need to do a SECMOD_RestartModules() }

.
.


but if you do:

.
.
.

fork()
.
.
NSS_Initialize()

- then you don't need to do a restart modules.


or if you do:

NSS_Initialize()
.
.
.
.
NSS_Shutdown()

fork();

NSS_Initialize()
.
.
.

agin, you don't need to do a restartmodule.


> How can we ensure that libcurl initialization will not invalidate OpenLDAP 
> handles or vice versa?

Ah, yes, we did think of that issue. If you call SECMOD_RestartModules(PR_FALSE); it will only restart those modules that will fail because of a fork(); That means if OpenLDAP has already called SECMOD_RestartModules(), your call will be a noop (other than looping through all the modules trying to do a C_GetSlotList() while holding the nss internal module log, so it's not a time noop). If you call it first, OpenLDAPs will be a noop. NSS will lock so if you both try it as the same time on different threads, only one will get through first, and the second will be a noop.

bob

Hope that helps.

Comment 21 Bob Relyea 2013-05-21 22:05:36 UTC
I should also point out that only the child of a fork() needs to call SECMOD_RestartModules(), you don't need to do it in the parent.

bob

Comment 22 Kamil Dudka 2013-05-24 15:03:35 UTC
I am unable to repeat the failure as described in comment #0 using the above mentioned versions of packages.  Do I need any special configuration of httpd or something to trigger the bug?

Comment 26 Kai Engert (:kaie) (inactive account) 2014-02-13 21:19:41 UTC
Who is most likely to produce a test environment? Set a "needinfo" flag for them.

Comment 27 Suzanne Forsberg 2014-05-20 17:26:20 UTC
We cannot reproduce this issue. Closing as WONTFIX


Note You need to log in before you can comment on or make changes to this bug.