Bug 1461423 - Memory leak in python-dnf
Memory leak in python-dnf
Status: ON_QA
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
26
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jaroslav Mracek
Fedora Extras Quality Assurance
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-14 08:27 EDT by Florian Ludwig
Modified: 2017-10-17 19:09 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Florian Ludwig 2017-06-14 08:27:40 EDT
Description of problem:
using the API results in memory leak

Version-Release number of selected component (if applicable):
python2-dnf.noarch                     1.1.10-6.fc25

How reproducible:
```
import dnf


def get_base():
    base = dnf.Base()
    base.conf.assumeyes = True
    base.read_all_repos()
    base.fill_sack(load_system_repo='auto')
    base.close()

while True:
    get_base()
```

Actual results:
Out of memory

Expected results:


Additional info:
Comment 1 Martin Hatina 2017-06-21 07:43:37 EDT
I tried to reproduce the problem and got following:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 5, in get_base
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 314, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 128, in _add_repo_to_sack
    load_updateinfo=True)
IOError: can not read file /var/tmp/dnf-mhatina-q9t71bzx/mhatina-modularity-demo-c3d7dc7fad9d9682/repodata/repomd.xml: Too many open files

The memory was oscillating on the same level, it wasn't rising.
Comment 2 Igor Gnatenko 2017-06-21 07:46:48 EDT
that means that either librepo doesn't close files properly....
Comment 3 Florian Ludwig 2017-09-08 05:00:47 EDT
I can still reproduce on fedora 26.


I created a docker image to reproduce it on a clean system:

docker run ludwigf/dnf-1461423

It will linearly increase memory usage for me:

starting endless loop
85 mb memory, delta: 85 mb
142 mb memory, delta: 57 mb
191 mb memory, delta: 49 mb
252 mb memory, delta: 61 mb
298 mb memory, delta: 46 mb
353 mb memory, delta: 55 mb
407 mb memory, delta: 54 mb
461 mb memory, delta: 54 mb
512 mb memory, delta: 51 mb
562 mb memory, delta: 49 mb
618 mb memory, delta: 56 mb
671 mb memory, delta: 54 mb
723 mb memory, delta: 52 mb
778 mb memory, delta: 55 mb
827 mb memory, delta: 49 mb
882 mb memory, delta: 55 mb
[...]
Comment 4 Florian Ludwig 2017-09-08 05:02:49 EDT
The code used is still the same, just added some output to display memory usage after each base.close()

Link to full code used:

https://github.com/FlorianLudwig/dnf-1461423/blob/master/bug.py
Comment 5 Jaroslav Mracek 2017-10-05 04:44:47 EDT
I have found first memory leak in base.read_all_repos(). Here is the patch (https://github.com/rpm-software-management/librepo/pull/118). But I believe that this is not the last problem.
Comment 6 Jaroslav Mracek 2017-10-05 09:22:58 EDT
I create two additional patches that solve the issue:

https://github.com/rpm-software-management/dnf/pull/943

https://github.com/rpm-software-management/libdnf/pull/337

Hope that it will solve the problem completly.
Comment 7 Fedora Update System 2017-10-16 05:05:38 EDT
dnf-2.7.4-1.fc26 libdnf-0.11.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
Comment 8 Fedora Update System 2017-10-16 05:06:44 EDT
dnf-2.7.4-1.fc27 libdnf-0.11.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Comment 9 Fedora Update System 2017-10-16 14:22:53 EDT
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
Comment 10 Fedora Update System 2017-10-16 22:24:27 EDT
dnf-2.7.4-1.fc27, libdnf-0.11.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Comment 11 Florian Ludwig 2017-10-17 19:09:08 EDT
Tested it on fc26 updates-testing.

Looks good!

Note You need to log in before you can comment on or make changes to this bug.