// Please consider updating nss in rawhide and f34 with the fix for the bug below

+++ This bug was initially created as a clone of Bug #1881999 +++

This bug was initially created as a copy of Bug #1868432
Creating and deleting a series of subCAs in rapid succession.

(...) 
--- Additional comment from Bob Relyea on 2021-06-28 17:11:55 UTC ---

I now have a reproducer that does not require JSS. On my machine, with the latest version of NSS, I needed to use 30 threads instead of 10 to cause the issue.

I've attached the test program to the bug. To compile:

cc -o create_key_thread -g -I/usr/include/nss3 -I/usr/include/nspr4  -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lpthread -ldl  create_key_thread.c

to run:

./create_key_thread -l 30 -t 30

It exists with 2 for test environment failures (bad arguments, issues initializing NSS, issues with thread creation).
It exists with 1 for actual test case failures (couldn't create the key pairs).
It exists with 0 for success.

CKR_GENERAL_ERROR is returned when a FIPS required test fails, in this case the pairwise consistency checks. Current theories on why that may fail: 
   1) there is some non-treadsafe failure in the actual encrypt/decrypt functions (unlikely or we would blow up all the time in multithreaded SSL operations.
   2) Some non-thread safe issue in the RSA freebl code.
   3) We somehow are getting the same session back when creating our RWsessions, or we get two sessions mapping to the same session structure.

Things that make the issue go away:

1) Unsurprisingly setting export NSS_FORCE_TOKEN_LOCK=1 causes the problem to go away. NSS_FORCE_TOKEN_LOCK is a debug feature with causes NSS to treat all tokens as not thread safe, and all PKCS #11 calls are locked. The environment variable is for NSS testing to make sure thread locking is consistent for non-thread safe tokens, and not meant for production, but it's a good test to see if a race is in the token or in NSS proper.
2) adding -e to the command line. This causes us to create session keys rather than token keys. NSS uses a single slot session, and locks the entire generate operation, so again it's not surprising.
3) using -t 10 will not create enough contention to cause failures, even if you set loop count up to 100.