Bug 1461423 - Memory leak in python-dnf
Memory leak in python-dnf
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
26
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jaroslav Mracek
Fedora Extras Quality Assurance
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-14 08:27 EDT by Florian Ludwig
Modified: 2017-11-10 21:54 EST (History)
5 users (show)

See Also:
Fixed In Version: dnf-2.7.4-1.fc26 dnf-2.7.4-1.fc27
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-25 19:11:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Florian Ludwig 2017-06-14 08:27:40 EDT
Description of problem:
using the API results in memory leak

Version-Release number of selected component (if applicable):
python2-dnf.noarch                     1.1.10-6.fc25

How reproducible:
```
import dnf


def get_base():
    base = dnf.Base()
    base.conf.assumeyes = True
    base.read_all_repos()
    base.fill_sack(load_system_repo='auto')
    base.close()

while True:
    get_base()
```

Actual results:
Out of memory

Expected results:


Additional info:
Comment 1 Martin Hatina 2017-06-21 07:43:37 EDT
I tried to reproduce the problem and got following:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 5, in get_base
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 314, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 128, in _add_repo_to_sack
    load_updateinfo=True)
IOError: can not read file /var/tmp/dnf-mhatina-q9t71bzx/mhatina-modularity-demo-c3d7dc7fad9d9682/repodata/repomd.xml: Too many open files

The memory was oscillating on the same level, it wasn't rising.
Comment 2 Igor Gnatenko 2017-06-21 07:46:48 EDT
that means that either librepo doesn't close files properly....
Comment 3 Florian Ludwig 2017-09-08 05:00:47 EDT
I can still reproduce on fedora 26.


I created a docker image to reproduce it on a clean system:

docker run ludwigf/dnf-1461423

It will linearly increase memory usage for me:

starting endless loop
85 mb memory, delta: 85 mb
142 mb memory, delta: 57 mb
191 mb memory, delta: 49 mb
252 mb memory, delta: 61 mb
298 mb memory, delta: 46 mb
353 mb memory, delta: 55 mb
407 mb memory, delta: 54 mb
461 mb memory, delta: 54 mb
512 mb memory, delta: 51 mb
562 mb memory, delta: 49 mb
618 mb memory, delta: 56 mb
671 mb memory, delta: 54 mb
723 mb memory, delta: 52 mb
778 mb memory, delta: 55 mb
827 mb memory, delta: 49 mb
882 mb memory, delta: 55 mb
[...]
Comment 4 Florian Ludwig 2017-09-08 05:02:49 EDT
The code used is still the same, just added some output to display memory usage after each base.close()

Link to full code used:

https://github.com/FlorianLudwig/dnf-1461423/blob/master/bug.py
Comment 5 Jaroslav Mracek 2017-10-05 04:44:47 EDT
I have found first memory leak in base.read_all_repos(). Here is the patch (https://github.com/rpm-software-management/librepo/pull/118). But I believe that this is not the last problem.
Comment 6 Jaroslav Mracek 2017-10-05 09:22:58 EDT
I create two additional patches that solve the issue:

https://github.com/rpm-software-management/dnf/pull/943

https://github.com/rpm-software-management/libdnf/pull/337

Hope that it will solve the problem completly.
Comment 7 Fedora Update System 2017-10-16 05:05:38 EDT
dnf-2.7.4-1.fc26 libdnf-0.11.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
Comment 8 Fedora Update System 2017-10-16 05:06:44 EDT
dnf-2.7.4-1.fc27 libdnf-0.11.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Comment 9 Fedora Update System 2017-10-16 14:22:53 EDT
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
Comment 10 Fedora Update System 2017-10-16 22:24:27 EDT
dnf-2.7.4-1.fc27, libdnf-0.11.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Comment 11 Florian Ludwig 2017-10-17 19:09:08 EDT
Tested it on fc26 updates-testing.

Looks good!
Comment 12 Fedora Update System 2017-10-25 19:11:22 EDT
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 13 Fedora Update System 2017-11-10 21:54:39 EST
dnf-2.7.4-1.fc27, libdnf-0.11.1-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.