Description of problem: using the API results in memory leak Version-Release number of selected component (if applicable): python2-dnf.noarch 1.1.10-6.fc25 How reproducible: ``` import dnf def get_base(): base = dnf.Base() base.conf.assumeyes = True base.read_all_repos() base.fill_sack(load_system_repo='auto') base.close() while True: get_base() ``` Actual results: Out of memory Expected results: Additional info:
I tried to reproduce the problem and got following: Traceback (most recent call last): File "<stdin>", line 2, in <module> File "<stdin>", line 5, in get_base File "/usr/lib/python2.7/site-packages/dnf/base.py", line 314, in fill_sack self._add_repo_to_sack(r) File "/usr/lib/python2.7/site-packages/dnf/base.py", line 128, in _add_repo_to_sack load_updateinfo=True) IOError: can not read file /var/tmp/dnf-mhatina-q9t71bzx/mhatina-modularity-demo-c3d7dc7fad9d9682/repodata/repomd.xml: Too many open files The memory was oscillating on the same level, it wasn't rising.
that means that either librepo doesn't close files properly....
I can still reproduce on fedora 26. I created a docker image to reproduce it on a clean system: docker run ludwigf/dnf-1461423 It will linearly increase memory usage for me: starting endless loop 85 mb memory, delta: 85 mb 142 mb memory, delta: 57 mb 191 mb memory, delta: 49 mb 252 mb memory, delta: 61 mb 298 mb memory, delta: 46 mb 353 mb memory, delta: 55 mb 407 mb memory, delta: 54 mb 461 mb memory, delta: 54 mb 512 mb memory, delta: 51 mb 562 mb memory, delta: 49 mb 618 mb memory, delta: 56 mb 671 mb memory, delta: 54 mb 723 mb memory, delta: 52 mb 778 mb memory, delta: 55 mb 827 mb memory, delta: 49 mb 882 mb memory, delta: 55 mb [...]
The code used is still the same, just added some output to display memory usage after each base.close() Link to full code used: https://github.com/FlorianLudwig/dnf-1461423/blob/master/bug.py
I have found first memory leak in base.read_all_repos(). Here is the patch (https://github.com/rpm-software-management/librepo/pull/118). But I believe that this is not the last problem.
I create two additional patches that solve the issue: https://github.com/rpm-software-management/dnf/pull/943 https://github.com/rpm-software-management/libdnf/pull/337 Hope that it will solve the problem completly.
dnf-2.7.4-1.fc26 libdnf-0.11.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
dnf-2.7.4-1.fc27 libdnf-0.11.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
dnf-2.7.4-1.fc27, libdnf-0.11.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Tested it on fc26 updates-testing. Looks good!
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
dnf-2.7.4-1.fc27, libdnf-0.11.1-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.