Forgot the ldap versions sorry:
php-ldap-5.3.3-3.el6_2.6.x86_64
openldap-2.4.23-20.el6.x86_64

Since RHEL 6.3 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

So this appears to be NSS "functioning as designed". The PKCS #11 spec says that it's not permissible to open a module, fork, then continue without reinitializing the module. NSS handles this by supplying a function that applications may call if they decide to fork() after NSS has already been initialized: SECMOD_RestartModules(PRBool force);

If force is true, all modules are restarted, if it is false, only those modules which appear to need reloading will be reloaded. This means calling SECMOD_RestartModules(PR_FALSE) is always safe, even if you haven't forked.

Note: SECMOD_RestartModules() will force all existing handles to be invalid, so you can't pass key handles from the parent to the child this way.

There is an environment variable that you can set which turns off the fork_check behavior in softoken. This doesn't affect other PKCS #11 modules, however (like libpem, or vendor tokens). https://developer.mozilla.org/en-US/docs/NSS_reference/NSS_environment_variables#NSS_STRICT_NO_FORK

You can try:
   export NSS_STRICK_NO_FORK=DISABLED

And see if that affects the results.

Thing to look for:

Libraries like libldap, and possibly libcurl, may try to detect the fork() scenario themselves and try to call SECMOD_RestartModules(). It would be good to see if SECMOD_RestartModules is being called and if it is somehow failing, or if libcurl and libldap are stomping on each other trying to call SECMOD_RestartModules, and why the stomping is occuring. If SECMOD_RestartModules is not being called, how could add that call to apache in a clean way (question, is mod_nss being used, for instance?).

bob

Code inspection reveals that SECMOD_RestartModules is being called by openldap but not by libcurl.

(lib)curl itself never forks.  If an application using libcurl forks, libcurl does not care.  On which event is libcurl supposed to call SECMOD_RestartModules?

This bug is waiting for someone to answer Kamil's question from comment 9.

It appears it's unclear which module should be responsible to fix this issue.

Let me attempt to draw conclusions:
- Bob says, if fork, then call SECMOD_RestartModules
- Elio says, openldap calls SECMOD_RestartModules,
  so one could argue the openldap isn't the culprit
- Kamil says, libcurl never forks,
  so libcurl isn't the culprit either.

So, who is actually performing the fork? Is it php? Is it apache?

This OpenLDAP commit seems to introduce the call to SECMOD_RestartModules():

http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blobdiff;f=libraries/libldap/tls_m.c;hb=d27f458b;hpb=ba70ec8b

Should the above be also applied to curl?

SECMOD_RestartModules() is called before NSS is initialized.  Is it a safe/recommended order of the initialization steps?

How can we ensure that libcurl initialization will not invalidate OpenLDAP handles or vice versa?

(In reply to comment #11)
> This bug is waiting for someone to answer Kamil's question from comment 9.
> 
> It appears it's unclear which module should be responsible to fix this issue.
> 
> Let me attempt to draw conclusions:
> - Bob says, if fork, then call SECMOD_RestartModules
> - Elio says, openldap calls SECMOD_RestartModules,
>   so one could argue the openldap isn't the culprit
> - Kamil says, libcurl never forks,
>   so libcurl isn't the culprit either.
> 
> So, who is actually performing the fork? Is it php? Is it apache?

Apache forks

> > So, who is actually performing the fork? Is it php? Is it apache?
> 
> Apache forks

ok.

I assume apache simply makes library calls into applications. If that's true, I guess, apache doesn't send anything like a "fork notification event" after a fork.

If there isn't, it would be a library's job to detect, that a fork has happened, and do whatever is necessary to ensure correct operation.

I further guess that comparing old and new values of "getpid" are the only way to detect that self has been forked. Correct? Or is there anything else that changes after a fork?

If above is true, the next question is: Which library should be responsible to contain the code to remember old process id, detect new pid, and re-initialize libraries?

It would be good if we were able to minimize the amount of places where such detection needs to happen. Trying to detect that in each NSS library call seems too much to ask for.

Could this detection be done inside each API that get called from apache? In other words.

I assume that apache doesn't call NSS directly - rather, that apache calls ldap APIs and curl APIs.

Let's talk about libcurl. Ideally, apache could notify each apache-module that it has loaded. Does it do that? If not, then the work has to be done completely inside libcurl.

libcurl could remember the current process ID, and whenever a libcurl API gets called, the first action is does, it compares the remembered pid, and if it's different, it calls the NSS function to restart modules.

Is above approach necessary?

Can anyone think of a simpler approach?

(In reply to comment #14)
> Can anyone think of a simpler approach?

What about fixing NSS so that no such workaround is necessary?

If NSS intends to be a well-designed library, it should survive a fork fully transparently as libcurl does.

> If NSS intends to be a well-designed library,
> it should survive a fork fully transparently as libcurl does.

To be clear: This is a issue with the PKCS #11 standard. NSS needs to handle modules that are PKCS #11 compliant. NSS itself does not generate issues specifically from forking after init.

In order to deal with this situation NSS provides 2 strategies: 1) A way to recover after a fork. Once you fork, you must arrange to call SECMOD_RestartModules() in the child. Old objects will no longer be valid, but new objects will work. You don't have to shutdown and restart NSS itself. or 2) set an environment variable which tells softoken not do do the fork() check. This means you won't work with other tokens!

If there are additional things we can supply to help make this easier, I'm open to suggestions, but just saying "make it work the way I think it should" is not going to fly.

bob

(In reply to comment #16)
> To be clear: This is a issue with the PKCS #11 standard. NSS needs to handle
> modules that are PKCS #11 compliant. NSS itself does not generate issues
> specifically from forking after init.

The problem is that we (ab)use a PKCS #11 module to load certificates from files, which was not necessary with OpenSSL, and users expect it to work with NSS, too.  Users do not care which standard we use to reimplement a working stuff.

> In order to deal with this situation NSS provides 2 strategies: 1) A way to
> recover after a fork. Once you fork, you must arrange to call
> SECMOD_RestartModules() in the child. Old objects will no longer be valid,
> but new objects will work. You don't have to shutdown and restart NSS
> itself. or 2) set an environment variable which tells softoken not do do the
> fork() check. This means you won't work with other tokens!

As far as I understand:

- Calling SECMOD_RestartModules() from libcurl invalidates OpenLDAP's objects.

- Calling SECMOD_RestartModules() from OpenLDAP invalidates libcurl's objects.

- Neither OpenLDAP nor libcurl are aware of each other in the running process.

- The only library aware of both higher-level libraries is NSS, so it is a hot candidate to put the fix for this bug to.  Is not it?

Bob, could you please answer my questions from comment #12?  Thanks in advance!

> Bob, could you please answer my questions from comment #12?  Thanks in advance!

Sure, sorry... here's the answers.

> Should the above be also applied to curl?

If you return from your library with NSS initialized, and you think the application my actually fork() in the interim, then you need to do restart modules().

Ideally the application should actually do this.

> SECMOD_RestartModules() is called before NSS is initialized.  Is it a 
> safe/recommended order of the initialization steps?


There are two interpretations I have of your question My first interpretation was. "Since SECMOD_RestartModules() is called before NSS is initialized.  Is it a safe/recommended order of the initialization steps?"

The second is "Is it safe to call SECMOD_RestartModules() even if I haven't initialized NSS?" I'm going to answer this one first because I think this is really what you are asking.

In general, NSS calls should not be made when NSS has not been initialized, but SECMOD_RestartModules() does check to see that the module lock has been created (which implies NSS has been initialized), so you are likely to get an error rather than a crash if you call SECMOD_RestartModules() without NSS being initialized.

If you really meant:  "Since SECMOD_RestartModules() is called before NSS is initialized.  Is it a safe/recommended order of the initialization steps?"

Then no, SECMOD_RestartModules() only needs to be called if NSS has already been initialized. If NSS has not be initialized, there is no need to call this over a fork().

So, for instance, Say you do:


NSS_Initialize()
.
.
.
.
.

fork()

{ at this point you would need to do a SECMOD_RestartModules() }

.
.


but if you do:

.
.
.

fork()
.
.
NSS_Initialize()

- then you don't need to do a restart modules.


or if you do:

NSS_Initialize()
.
.
.
.
NSS_Shutdown()

fork();

NSS_Initialize()
.
.
.

agin, you don't need to do a restartmodule.


> How can we ensure that libcurl initialization will not invalidate OpenLDAP 
> handles or vice versa?

Ah, yes, we did think of that issue. If you call SECMOD_RestartModules(PR_FALSE); it will only restart those modules that will fail because of a fork(); That means if OpenLDAP has already called SECMOD_RestartModules(), your call will be a noop (other than looping through all the modules trying to do a C_GetSlotList() while holding the nss internal module log, so it's not a time noop). If you call it first, OpenLDAPs will be a noop. NSS will lock so if you both try it as the same time on different threads, only one will get through first, and the second will be a noop.

bob

Hope that helps.

I should also point out that only the child of a fork() needs to call SECMOD_RestartModules(), you don't need to do it in the parent.

bob

I am unable to repeat the failure as described in comment #0 using the above mentioned versions of packages.  Do I need any special configuration of httpd or something to trigger the bug?

Who is most likely to produce a test environment? Set a "needinfo" flag for them.

We cannot reproduce this issue. Closing as WONTFIX

see https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c32
and https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c34