Bug 1214538 - dnf hangs during a package download
Summary: dnf hangs during a package download
Keywords:
Status: CLOSED DUPLICATE of bug 1212320
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 22
Hardware: i686
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-04-23 02:52 UTC by Ralf Corsepius
Modified: 2016-04-15 08:39 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-04-27 12:49:52 UTC


Attachments (Terms of Use)
'dnf update' output with LIBREPO_DEBUG=1 (7.16 KB, application/zip)
2016-04-15 08:39 UTC, Semyon
no flags Details

Description Ralf Corsepius 2015-04-23 02:52:26 UTC
Description of problem:

# dnf update
...
Transaction Summary
=====================================================================================================================================
Install    2 Packages
Upgrade  117 Packages

Total download size: 194 M
Is this ok [y/N]: y
Downloading Packages:
...
(60/119): libreoffice-writer-4.4.2.2-1.fc22.i686.rpm                                                 2.2 MB/s | 4.2 MB     00:01    
(61-63/119): libreoffice-core-4.4.2.2-1.fc22.i6 38% [=================-                            ] 1.8 MB/s |  74 MB     01:08 ETA

At this point, dnf does not progress any further. 

The command line alternates between libreoffice-base, libreoffice-calc and libreoffice-core without any progress for many (>6) hours.


When aborting and re-running dnf, this happened
...
[MIRROR] gdm-3.16.0.1-2.fc22.i686.rpm: Status code: 404 for http://ftp.halifax.rwth-aachen.de/fedora/linux/development/22/i386/os/Packages/g/gdm-3.16.0.1-2.fc22.i686.rpm
....
(24/119): git-2.3.5-1.fc22.i686.rpm                                                                  2.3 MB/s | 5.0 MB     00:02    
[MIRROR] gdm-3.16.0.1-2.fc22.i686.rpm: Curl error (78): Remote file not found for ftp://ftp.halifax.rwth-aachen.de/fedora/linux/development/22/i386/os/Packages/g/gdm-3.16.0.1-2.fc22.i686.rpm [RETR response: 550]
(25/119): gmp-6.0.0-9.fc22.i686.rpm                                                                  1.2 MB/s | 423 kB     00:00    
...
[Lots of futher [MIRROR] errors show up referring to gdm-3.16.0.1-2.fc22.i686.rpm referring to mirrors all around Europe]
...
(26-28/119): libtasn1-4.4-1.fc22.i686.rpm       20% [=========                                     ] 1.9 MB/s |  40 MB     01:23 ETA
[dnf hangs and alternates between gdm, libtasn1, git]


Version-Release number of selected component (if applicable):
dnf-0.6.5-1.fc22.noarch

How reproducible:
No idea.

Further info:
- IMO, this issue qualifies dnf as non-production ready. Either the decision to force dnf on users in F22 should be reverted or this bug be made a release blocker.

Comment 1 Adam Williamson (Fedora) 2015-04-23 05:33:34 UTC
Those URLs do have the errors cited, dnf is not lying to you. Which indicates there's some sort of mirror issue, not a problem in dnf, probably.

Comment 2 Ralf Corsepius 2015-04-23 05:46:02 UTC
(In reply to Adam Williamson (Fedora) from comment #1)
> Those URLs do have the errors cited, dnf is not lying to you. Which
> indicates there's some sort of mirror issue, not a problem in dnf, probably.

There definitely are mirror issue currently going on, but this is not an excuse for dnf behaving such kind of lousily poor.

Please excuse my tone, but we are in a "close to product" stage, where such kind of behavior of a tool with the importance of dnf are inacceptable and must be fixed RSN.

Comment 3 Adam Williamson (Fedora) 2015-04-23 05:47:22 UTC
What behaviour are you expecting, when none of the available mirrors can provide the package it needs? Did you check whether yum's behaviour is in fact any better (or different)?

Comment 4 Ralf Corsepius 2015-04-23 06:08:17 UTC
(In reply to Adam Williamson (Fedora) from comment #3)
> What behaviour are you expecting, when none of the available mirrors can
> provide the package it needs? Did you check whether yum's behaviour is in
> fact any better (or different)?

yum iterated over all mirrors and gave up when it didn't find a workable mirror. The way yum iterated was quite "improvable" (it retried to access all update candidate packages on all mirrors), but we have been put off to "with dnf this will improve".

Now, dnf just hangs for in a busy waiting loop.

To me this is a massive regression, which longs for drastic consequences up to the management level in Fedora.

Comment 5 Ralf Corsepius 2015-04-23 06:37:05 UTC
Putting aside the current mirror issues, I would not be surprised if the actual cause of the hangs is some mirrors which are disallowing parallel connections.

To investiate this is would be necessary to limit the number of parallel connections and to know the origins dnf ties to download the packages from.

Unfortunately, I can't find any option in dnf/dnf.conf to set the number of parallel connections, nor does "-d ..." provide the origins of downloads.

Comment 6 Radek Holy 2015-04-23 08:20:14 UTC
I believe this is something that should be handled by the underlying librepo. Do you agree, Tomas?

Ralf, if it happens to you again, could you please re-run the command with LIBREPO_DEBUG environment variable set to "1"?

BTW, bug 1206878 sounds very similar. But IIUIC, the always download hangs on 0% there.

Tomas, we call librepo like this: "librepo.download_packages(targets, failfast=True)". The targets are instances of "librepo.PackageTarget" with some "handle", "dest", "relativeurl", "checksum_type", "checksum", "expectedsize", "baseurl", "resume==True", "cbdata", "progresscb", "endcb" and "mirrorfailurecb". The "handle" has some "gpgcheck", "interruptible=True", "repotype==LR_YUMREPO", "useragent", "yumdlist", "varsub", "destdir", url, "progresscb", "fastestmirrorcb", "maxspeed", "proxy", "lowspeedlimit" and "connecttimeout". If you need more info, let us know.

Comment 7 Ralf Corsepius 2015-04-27 06:22:01 UTC
(In reply to Radek Holy from comment #6)

> Ralf, if it happens to you again, could you please re-run the command with
> LIBREPO_DEBUG environment variable set to "1"?

I could reproduce this issue deterministically until I issued
"dnf clean metadata; dnf update" this morning, when preparing to have a clean environment for experiments with LIBREPO_DEBUG.

Afterwards it vanished. Further attempts to reproduce it have not been successful, so far.

Could it be, dnf is having problems with corrupted metadata (or caches) or that this issue was "magically" healed by some other update? 

[When this issue had popped up, I was facing several issues with this F22-system. Amongst them had been "bad mirrors", bad connectivity to "fedoramirrors.fedoraproject.org" and other severe issues - I would not want to exclude a kernel-bug]

Comment 8 Radek Holy 2015-04-27 10:38:14 UTC
Hm, OK, that's how others work around similar issues. Anyway, I think that librepo should have some timeout here. On the other hand, no one probably knows where exactly. Maybe librepo already has some but DNF sets it incorrectly. So, let's switch it back to DNF and let's hope that somebody else hits this too. (I need to remind them not to run "dnf clean" in that case.)

Comment 9 Honza Silhan 2015-04-27 12:49:52 UTC
Anyone who is able to reproduce it, attach stdout of dnf after executing `LIBREPO_DEBUG=1 dnf ...` and try to experiment by increasing `minrate` and decreasing `timeout` option in /etc/dnf/dnf.conf, please. Thanks

Comment 10 Michal Luscon 2015-06-12 12:18:34 UTC

*** This bug has been marked as a duplicate of bug 1212320 ***

Comment 11 Semyon 2016-04-15 08:39:30 UTC
Created attachment 1147529 [details]
'dnf update' output with LIBREPO_DEBUG=1

(In reply to Jan Silhan from comment #9)
> Anyone who is able to reproduce it, attach stdout of dnf after executing
> `LIBREPO_DEBUG=1 dnf ...` and try to experiment by increasing `minrate` and
> decreasing `timeout` option in /etc/dnf/dnf.conf, please. Thanks

I have experienced this bug on Fedora 23.

I have attached two 'LIBREPO_DEBUG=1 dnf update' stdouts - dnf.log with default dnf settings (interrupted as there was no progress) and dnf-2.log with

timeout=1
minrate=10000

added to /etc/dnf/dnf.conf.

This problem was temporary - some hours later I've updated successfully, but it would be better if dnf would print out some error message if there are any problems downloading rpm packages.


Note You need to log in before you can comment on or make changes to this bug.