Bug 1012290 - dnf hits "Too many open files" when downloading RPMs
Summary: dnf hits "Too many open files" when downloading RPMs
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: librepo
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Tomas Mlcoch
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-26 08:39 UTC by Michal Schmidt
Modified: 2013-10-07 06:37 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-10-01 08:57:47 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Michal Schmidt 2013-09-26 08:39:47 UTC
Description of problem:
I tried upgrading from F19 to F20 Alpha using dnf. First I updated the "fedora-release" package to the fc20 version, then I did "dnf update rpm dnf". Then:

 # dnf distro-sync --best
 ... [ transaction was calculated and confirmed by me ] ...
 ... [ RPMs were being downloaded ] ...
 (993/2021): OpenEXR-libs-1.7.1-6.fc20.x86_64.rpm           | 217 kB  00:00:00
 (994/2021): PackageKit-0.8.11-2.fc20.x86_64.rpm            | 620 kB  00:00:00
 (995/2021): PackageKit-command-not-found-0.8.11-2.fc20.x86_64.rpm
                                                            |  26 kB  00:00:00
 (996/2021): PackageKit-device-rebind-0.8.11-2.fc20.x86_64.rpm
                                                            |  17 kB  00:00:00
 [Errno unknown] dup(1023) failed: Too many open files: 'Input/Output error'

Here dnf exited.
There may be an fd leak somewhere in dnf or in a library it uses.


Version-Release number of selected component (if applicable):
dnf-0.4.1-1.git55e6369.fc20.noarch
librepo-1.1.0-1.fc20.x86_64
hawkey-0.4.1-1.git6f35513.fc20.x86_64
rpm-4.11.1-7.fc20.x86_64


How reproducible:
I saw the same problem on two different systems where I attempted the same upgrade.

Steps to Reproduce:
1. "dnf distro-sync" where it results in many RPMs to download.
My guess is any dnf operation that would involve a download of ~1000 RPMs would
hit the same issue. Should be more easily reproducible by setting "ulimit -n" intentionally to a lower value. I didn't try that.

Actual results:
[Errno unknown] dup(1023) failed: Too many open files: 'Input/Output error'

Expected results:
Should be able to download any number of RPMs successfully.

Additional info:
Re-running the same dnf command is a workaround. It will simply recognize already downloaded RPMs and continue downloading where it hit the error before.

Comment 1 Zdeněk Pavlas 2013-09-26 08:50:48 UTC
Confirmed, librepo does not close files after download (both metadata and rpms). Reassigning.

Comment 3 Tomas Mlcoch 2013-10-01 08:57:47 UTC
Build for F20 is ready: http://koji.fedoraproject.org/koji/taskinfo?taskID=6009388, it will be pushed to updates soon.

Comment 4 Michal Schmidt 2013-10-07 06:37:22 UTC
I am now seeing a "Too many open files" error again, but in slightly different circumstances: bug 1015957.


Note You need to log in before you can comment on or make changes to this bug.