I've already talked to some people about this via IRC and email. Will reproduce the main parts of the communication with Robert (thanks!) here:

After a fork in the app (anaconda, the fork happens through python's multiprocessing fwiw) calls to nss are failing. first i saw HASH_Create is failing, so asked around and nalin tipped me it could be that we don't call SECMOD_RestartModules() I didn't think we needed that since RPM recreates the entire nss context after a fork but I'm calling RestartModules now instead and the call fails. I put the debugger on it and was able to track the failure down to PK11_InitToken there this line returns NULL:

crv = PK11_GETTAB(slot)->C_GetTokenInfo(slot->slotID,&tokenInfo);

and the debugger won't let me step into that (it somehow has troubles mapping the call address to the implementation). It seems that the error returned is CKR_SLOT_ID_INVALID.

This doesn't happen if we plain fork() a trivial program using librpm (i.e. some crypto has to take place first). After installing nss-softkn-debuginfo I still wasn't able to step into the calls through PK11_GETTAB.

printing *slot from the debugger yields:

$1 = {functionList = 0x7fe8c1e6e7a0, module = 0x2868b40, needTest = 0, isPerm = 1, isHW = 0, isInternal = 1,
  disabled = 0, reason = PK11_DIS_NONE, readOnly = 1, needLogin = 0, hasRandom = 1, defRWSession = 0,
  isThreadSafe = 1, flags = 1539, session = 1, sessionLock = 0x27de4d0, slotID = 1, defaultFlags = 2684616509,
  refCount = 17, freeListLock = 0x27ede40, freeSymKeysWithSessionHead = 0x2e2b260, freeSymKeysHead = 0x2e02880,
  keyCount = 7, maxKeyCount = 800, askpw = 0, timeout = 30, authTransact = 0, authTime = 0, minPassword = 0,
  maxPassword = 0, series = 2, flagSeries = 13107, flagState = 858993459, wrapKey = 0, wrapMechanism = 306, refKeys = {
    10}, mechanismList = 0x28370e0, mechanismCount = 148, cert_array = 0x0, array_size = 858993459, cert_count = 0,
  serial = '0' <repeats 16 times>, slot_name = "NSS Internal Cryptographic Services\000", '\063' <repeats 29 times>,
  token_name = "NSS Generic Crypto Services\000\063\063\063\063\063", hasRootCerts = 0, hasRootTrust = 858993459,
  hasRSAInfo = 1, RSAInfoFlags = 424704, protectedAuthPath = 0, isActiveCard = 0, lastLoginCheck = 0, lastState = 0,
  nssToken = 0x2836720,
  mechanismBits = "\a\a\006\003\003\003\001\000\000\000\000\000\000\001\000\000\a\a\005", '\000' <repeats 13 times>, "\a\a\006\002\002\002\000\000\000\000\000\000\000\000\000\000\002\002\002\002\002\002\002\000\000\000\000\000\000\000\000\000\003\003\003\002\002\002\001\000\000\000\000\000\000\000\000\000ldd``d\004\004\000\000\000\000\000\000\000\000\f\004\f\b\b\b\000\000\000\000\000\000\000\000\000\000\f\f\f\b\000\b\b\b\000\000\000\000\000\000\000\000\b\b", '\000' <repeats 14 times>, "\b\b\b\b\b\b\b\000\000\000\000\000\000\000\000\000\b\b\000\000\000\000\b\b\b\b\b\b\000\000\000\000\b", '\000' <repeats 15 times>, "\b", '\000' <repeats 62 times>} 

If I ignore the failed call to SECMOD_RestartModules() the crypto won't work, HASH_Create(HASH_AlgSHA256) typically fails.

I am opening this bug to keep track. I've already spent well over the time I had to debug this so the Anaconda DNF Payload implementation will proceed by disabling signatures verification/digests in rpm. If this poses a problem in the future either this bug needs to be resolved by the NSS maintainers (if possible---the fork-checking magic in the NSS code is inviting bugs like this) or the DNF Payload will have to do the full dance of spawning a new process, feeding it all the necessary data and letting it do the transaction. This is of course much less elegant and harms the performance of the payload.