Description of problem: curl has a reproducible memory leak Version-Release number of selected component (if applicable): curl-7.32.0-12.fc20.x86_64 How reproducible: Always Steps to Reproduce: 1. Create script with curl running in loops 2. Execute it 3. Have it running for a while Actual results: Memory usage rises Expected results: Memory usage remains Additional info: It seems that curl has fixed a lot of memory leaks in last few versions, which may indicate that we only need to update fedora package to 7.37 This is the script I used (it swallowed about 8GB in three hours) : #!/bin/bash while true; do curl --silent "https://api.twitch.tv/kraken/channels/thebigfatmoe/follows?direction=desc&limit=100" curl --silent -H 'Accept: application/vnd.twitchtv.v2+json' -X GET https://api.twitch.tv/kraken/streams/thebigfatmoe read -t 10 -n 1 if [ $? = 0 ] ; then break ; fi done echo Done!
Which process exactly allocates the memory? It could hardly be a long-term memory leak of curl as you create a fresh process of curl in each iteration. Even if there was a memory leak in curl, the memory would be released by operating system in each iteration. Nevertheless, valgrind reports no memory leaks in the above curl commands.
I need help with providing you more info. I cannot identify which exact process allocates the memory - I don't know how. If you could help me with that, I'll gladly investigate it more. I tried to see it with 'top', but couldn't get any process with high usage to show. Right now I did some testing (in virtual environment): ran Fedora 20 "Minimal install" and Fedora Rawhide "Minimal install" (both fully updated) with script running for three hours. This is Fedora 20's 'free': Before: total used free shared buffers cached Mem: 2050364 165164 1885200 264 7016 62116 -/+ buffers/cache: 96032 1954332 Swap: 1218556 0 1218556 After: total used free shared buffers cached Mem: 2050364 1973404 76960 176 3400 45244 -/+ buffers/cache: 1924760 125604 Swap: 1218556 380 1218176 and Fedora Rawhide's 'free': Before: total used free shared buffers cached Mem: 2050012 176744 1873268 540 8472 70184 -/+ buffers/cache: 98088 1951924 Swap: 1218556 0 1218556 After: total used free shared buffers cached Mem: 2050012 1975848 74164 196 9152 52852 -/+ buffers/cache: 1913844 136168 Swap: 1218556 344 1218212 Now, I learned that the line 'buffers/cache' actually shows the memory usage, and I believe when swap is starting to be used the system is out of memory. So, it seems to me that there is a memory leak somewhere, and newer curl is not solution. If I can do anything more, please let me know.
I would suggest to install/run htop - an interactive text-mode process viewer. You can sort the processes by memory consumption to see what the culprit is. It can also be the terminal window what continuously allocates memory for the output produced by curl. Which terminal do you actually use? Could you please try to append -o/dev/null to those curl commands to check whether it makes any difference?
Thanks for the instructions! I don't have X installed, I am using text mode. I suppose that means I am not using any terminals. Either way, adding '-o/dev/null' didn't make any difference. Installing htop didn't help. It shows that memory is used, but when I sort it by either VIRT or MEM% it shows nothing (screenshot attached).
Created attachment 934526 [details] htop MEM%
Created attachment 934527 [details] htop VIRT
Thanks for the additional info! I was finally able to reproduce the memory consumption locally. Indeed, the allocated memory did not appear in the htop screen, which implies it was likely allocated at the kernel level. Could you please execute the following command as root to check whether it releases the memory excessively allocated by kernel? echo 3 > /proc/sys/vm/drop_caches Could you please execute the following command prior to running your script and check whether it reduces the memory consumption? export NSS_SDB_USE_CACHE=no
You are on to something... - When memory is used up, it seems that I can free it with dropping caches! - And I added the var in my script and there is no more memory leak! So, your assumptions were correct, what now? Is this the way it's supposed to work, or is this indeed a bug?
It is either a kernel bug, or improper kernel use by nss-softokn. Bug 1044666 introduces a workaround for this. I am switching the bug to kernel for now. A network-independent reproducer follows: openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem openssl s_server -cert mycert.pem -www & while true; do curl -so/dev/null https://localhost:4433; done
Adding -k to curl (to prevent a handshake failure) makes it allocate faster.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs. Fedora 20 has now been rebased to 3.17.2-200.fc20. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21. If you experience different issues, please open a new bug report for those.
still reproducible with kernel-3.17.2-300.fc21.x86_64
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 21 kernel bugs. Fedora 21 has now been rebased to 3.18.3-201.fc21. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in over 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.