Description of problem: The performance team noticed an issue on some OpenShift Online nodes where the negative dentry cache was consuming a large amount of memory. We tracked down the cause on one node to the sdb_measureAccess function in the nss library's soft token code ( https://github.com/servo/nss/blob/master/lib/softoken/sdb.c ) The purpose of the function is to determine whether a cache would be faster that reading data from disk, but in our case, the test is probably hurting the gear's performance significantly, as well as the node overall. I don't know if this nss module is commonly used in OpenShift Online (this particular app we found is using it to authenticate to facebook), but it's easier for us to just define the variable everywhere than to continuously watch for apps causing this problem. It does not really matter to me whether we set the variable to yes or no, but my gut says that no, is reasonable, since the code comments indicate that the authors expect to use cache when the alternative is a slow network filesystem.
Sounds like you're proposing that NSS_SDB_USE_CACHE=no be set on all gears (not something each cartridge would set), so that sounds like a node level thing...Jhon correct me if I'm wrong.
Is it acceptable populate /etc/openshift/env/NSS_SDB_USE_CACHE with the value 'no' using puppet? This take effect on the gear's next start or unidle.
Ok, the suggestion did what I wanted, but the problem is still occurring. I'll have to investigate this further tomorrow.
@Andy Any update here?
It's been more than two years since we dealt with this issue, and I have not seen the problem again. This seems to have been a characteristic of one particular app, so we'll defer it for now.