Bug 1136572

Which process exactly allocates the memory?

It could hardly be a long-term memory leak of curl as you create a fresh process of curl in each iteration.  Even if there was a memory leak in curl, the memory would be released by operating system in each iteration.  Nevertheless, valgrind reports no memory leaks in the above curl commands.

I need help with providing you more info.
I cannot identify which exact process allocates the memory - I don't know how. If you could help me with that, I'll gladly investigate it more. I tried to see it with 'top', but couldn't get any process with high usage to show.

Right now I did some testing (in virtual environment): ran Fedora 20 "Minimal install" and Fedora Rawhide "Minimal install" (both fully updated) with script running for three hours.

This is Fedora 20's 'free':
Before:
             total       used       free     shared    buffers     cached
Mem:       2050364     165164    1885200        264       7016      62116
-/+ buffers/cache:      96032    1954332
Swap:      1218556          0    1218556

After:
             total       used       free     shared    buffers     cached
Mem:       2050364    1973404      76960        176       3400      45244
-/+ buffers/cache:    1924760     125604
Swap:      1218556        380    1218176


and Fedora Rawhide's 'free':
Before:
             total       used       free     shared    buffers     cached
Mem:       2050012     176744    1873268        540       8472      70184
-/+ buffers/cache:      98088    1951924
Swap:      1218556          0    1218556

After:
             total       used       free     shared    buffers     cached
Mem:       2050012    1975848      74164        196       9152      52852
-/+ buffers/cache:    1913844     136168
Swap:      1218556        344    1218212


Now, I learned that the line 'buffers/cache' actually shows the memory usage, and I believe when swap is starting to be used the system is out of memory.
So, it seems to me that there is a memory leak somewhere, and newer curl is not solution.

If I can do anything more, please let me know.

I would suggest to install/run htop - an interactive text-mode process viewer.  You can sort the processes by memory consumption to see what the culprit is.

It can also be the terminal window what continuously allocates memory for the output produced by curl.  Which terminal do you actually use?

Could you please try to append -o/dev/null to those curl commands to check whether it makes any difference?

Thanks for the instructions!

I don't have X installed, I am using text mode. I suppose that means I am not using any terminals. Either way, adding '-o/dev/null' didn't make any difference.

Installing htop didn't help. It shows that memory is used, but when I sort it by either VIRT or MEM% it shows nothing (screenshot attached).

Created attachment 934526 [details]
htop MEM%

Created attachment 934527 [details]
htop VIRT

Thanks for the additional info!  I was finally able to reproduce the memory consumption locally.  Indeed, the allocated memory did not appear in the htop screen, which implies it was likely allocated at the kernel level.

Could you please execute the following command as root to check whether it releases the memory excessively allocated by kernel?

echo 3 > /proc/sys/vm/drop_caches

Could you please execute the following command prior to running your script and check whether it reduces the memory consumption?

export NSS_SDB_USE_CACHE=no

You are on to something...

- When memory is used up, it seems that I can free it with dropping caches!

- And I added the var in my script and there is no more memory leak!

So, your assumptions were correct, what now? Is this the way it's supposed to work, or is this indeed a bug?

It is either a kernel bug, or improper kernel use by nss-softokn.  Bug 1044666 introduces a workaround for this.  I am switching the bug to kernel for now.  A network-independent reproducer follows:

openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
openssl s_server -cert mycert.pem -www &
while true; do curl -so/dev/null https://localhost:4433; done

Adding -k to curl (to prevent a handshake failure) makes it allocate faster.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.17.2-200.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21.

If you experience different issues, please open a new bug report for those.

still reproducible with kernel-3.17.2-300.fc21.x86_64

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 21 kernel bugs.

Fedora 21 has now been rebased to 3.18.3-201.fc21.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.

*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in over 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.