Bug 1247076 - should allow to configure http vs ftp preference: ftp mirrors lot slower than http mirror
should allow to configure http vs ftp preference: ftp mirrors lot slower than...
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
22
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: rpm-software-management
Fedora Extras Quality Assurance
:
: 1245753 1258191 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-27 05:43 EDT by Török Edwin
Modified: 2017-11-23 02:01 EST (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-19 13:12:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
FTP Connection to 200.93.227.165 working (858.43 KB, application/octet-stream)
2015-09-01 09:14 EDT, Giovanni Tirloni
no flags Details
FTP Connection to 200.236.31.8 getting stuck (4.53 KB, application/octet-stream)
2015-09-01 09:14 EDT, Giovanni Tirloni
no flags Details
Add ftp_use_epsv option to dnf (1.78 KB, text/plain)
2015-09-11 18:25 EDT, Giovanni Tirloni
no flags Details
Exposes libcurl EPSV enable/disable option (4.11 KB, text/plain)
2015-09-11 18:26 EDT, Giovanni Tirloni
no flags Details

  None (edit)
Description Török Edwin 2015-07-27 05:43:00 EDT
Description of problem:
dnf check-update and dnf upgrade is very slow (50 - 150 KB/s), and sometimes it is normal speed (4.5 - 5.3 MB/s), but its more often slow than fast.

I would need a way to tell DNF to always prefer HTTP mirrors over FTP mirrors (when the hostname is the same), or perhaps you could implement that logic on mirrors.fedoraproject.org. Besides the lower latency of HTTP vs FTP, the FTP mirror here is order of magnitude slower than HTTP on same server.

Using fastestmirror=1 (which is NOT the default despite what the docs say at https://fedoraproject.org/wiki/Yum_to_DNF_Cheatsheet) works this around,
but there was a bug https://bugzilla.redhat.com/show_bug.cgi?id=1051554 asking to turn it off, so turning it back on again would just shift the problem to other users.

Version-Release number of selected component (if applicable):
1.0.1

How reproducible:
Dnf seems to randomly pick between http and ftp URL of same mirror. When it picks ftp it is always MUCH slower.

Steps to Reproduce:
1. dnf clean all
2. dnf check-update
3. look at download speed

Actual results:
very slow, between 50-200 KB/s

Expected results:
normal speed, around 4.5 - 5.3 MB/s


Additional info:
By default dnf sometimes picks this ftp mirror which is very slow:
ftp://fedora.mirrors.telekom.ro/fedora/linux/updates/22/x86_64/

If I enable fastestmirror=1 in /etc/dnf/dnf.conf then it always picks the http mirror: 
http://fedora.mirrors.telekom.ro/pub/fedora/linux/updates/22/x86_64

FWIW both connect to the same IPv4 address, I don't know why FTP is that much slower, but since I don't see any advantage of using FTP over HTTP, I'd just always want to use HTTP(S) by default where available.

Here is the curl -O output showing the speed difference:
curl -O http://fedora.mirrors.telekom.ro/pub/fedora/linux/updates/22/x86_64/repodata/bf4506b4c2b5040bc6da647a7a1e72b9beffaeab697165ab48fb3f4e5b13262a-filelists.xml.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8758k  100 8758k    0     0  6076k      0  0:00:01  0:00:01 --:--:-- 6073k

curl -O ftp://fedora.mirrors.telekom.ro/fedora/linux/updates/22/x86_64/repodata/bf4506b4c2b5040bc6da647a7a1e72b9beffaeab697165ab48fb3f4e5b13262a-filelists.xml.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8758k  100 8758k    0     0  49911      0  0:02:59  0:02:59 --:--:-- 46419
Comment 1 Honza Silhan 2015-08-10 10:50:36 EDT
*** Bug 1245753 has been marked as a duplicate of this bug. ***
Comment 2 Honza Silhan 2015-08-10 10:53:28 EDT
We can add config option for protocol preference and sort mirrorlists accordingly.
Comment 3 EMR_Fedora 2015-08-26 14:52:18 EDT
What about changing the timeout's:

[MIRROR] libpurple-devel-2.10.11-12.fc22.i686.rpm: Curl error (28): Timeout was reached for ftp://mirror.steadfast.net/fedora/updates/22/i386/l/libpurple-devel-2.10.11-12.fc22.i686.rpm [Connection time-out]
[MIRROR] vala-0.28.1-1.fc22.i686.rpm: Curl error (28): Timeout was reached for ftp://mirror.steadfast.net/fedora/updates/22/i386/v/vala-0.28.1-1.fc22.i686.rpm [Connection time-out]
Comment 4 Peter Robinson 2015-08-26 20:14:05 EDT
(In reply to EMR_Fedora from comment #3)
> What about changing the timeout's:

Completely irrelevant for this bug, if ftp doesn't work, or isn't allowed thought a firewall all you're then doing then doing by changing the timeout is prolonging the pain and making the problem worse.
Comment 5 Honza Silhan 2015-09-01 08:34:17 EDT
*** Bug 1258191 has been marked as a duplicate of this bug. ***
Comment 6 Giovanni Tirloni 2015-09-01 09:12:51 EDT
I'm afraid FTP/slower might be a red herring in this case.

I've no problem getting fast downloads from FTP server 200.93.227.165. If dnf connects to FTP server 200.236.31.8, then it just gets stuck, consistently.
Comment 7 Giovanni Tirloni 2015-09-01 09:14:01 EDT
Created attachment 1069006 [details]
FTP Connection to 200.93.227.165 working
Comment 8 Giovanni Tirloni 2015-09-01 09:14:33 EDT
Created attachment 1069007 [details]
FTP Connection to 200.236.31.8 getting stuck
Comment 9 EMR_Fedora 2015-09-03 19:06:06 EDT
(In reply to Giovanni Tirloni from comment #6)
> I'm afraid FTP/slower might be a red herring in this case.
> 
> I've no problem getting fast downloads from FTP server 200.93.227.165. If
> dnf connects to FTP server 200.236.31.8, then it just gets stuck,
> consistently.

But the point is that yum never had this problem. I changed my timeout to 5 seconds and it seems to have improved performance.
Comment 10 Saeed Abbasi 2015-09-10 17:09:24 EDT
Is there any specific reason in Fedora 22, while I run "dnf check-update" or "dnf update" after "dnf clean all", it first request the repomd.xml by HTTP and then It request the same data by FTP as well? (Both receive the data successfully according to network traffic) Why it request it two time?

HTTP:
Request: GET /fedora/linux/releases/22/Everything/x86_64/os/repodata/repomd.xml HTTP/1.1\r\n

Response:
HTTP/1.1 200 OK\r\n
eXtensible Markup Language
<repomd
xmlns="http://linux.duke.edu/metadata/repo"...


FTP:
Request:RETR repomd.xml\r\n

Response:FTP Data (<?xml version="1.0" encoding="UTF-8"?>\n<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">\n  <revision>1441821652</revision>\n  <tags>\n    <content>binary-x86_64</content>\n  </t...
Comment 11 Saeed Abbasi 2015-09-11 10:11:49 EDT
I would like to know how it design, when it use FTP and When HTTP, in default?
Comment 12 EMR_Fedora 2015-09-11 13:01:26 EDT
The problem with DNF is libcurl. Curl vs Wget is below at 2minutes vs 0.1sec respectively!!

$ curl -o foobar ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1409k  100 1409k    0     0  11313      0  0:02:07  0:02:07 --:--:--  348k

$ wget ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
--2015-09-11 12:56:58--  ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
           => 'maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm.1'
Resolving mirror.symnds.com (mirror.symnds.com)... 63.245.196.124
Connecting to mirror.symnds.com (mirror.symnds.com)|63.245.196.124|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /distributions/fedora/releases/22/Everything/i386/os/Packages/m ... done.
==> SIZE maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm ... 1442888
==> PASV ... done.    ==> RETR maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm ... done.
Length: 1442888 (1.4M) (unauthoritative)

maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm.1          100%[=======================================================================================================================>]   1.38M  --.-KB/s   in 0.1s

2015-09-11 12:56:59 (10.6 MB/s) - 'maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm.1' saved [1442888]
Comment 13 Giovanni Tirloni 2015-09-11 13:10:20 EDT
It seems curl is using EPSV by default, while wget is not. Could you try curl with `--verbose` and `--disable-epsv` and compare?
Comment 14 EMR_Fedora 2015-09-11 13:13:13 EDT
Yeah, you were right! 

$ curl --disable-epsv -o foobar ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1409k  100 1409k    0     0  2819k      0 --:--:-- --:--:-- --:--:-- 2818k

So how do we make dnf use --disable-epsv??
Comment 15 Giovanni Tirloni 2015-09-11 18:25:56 EDT
Created attachment 1072651 [details]
Add ftp_use_epsv option to dnf
Comment 16 Giovanni Tirloni 2015-09-11 18:26:26 EDT
Created attachment 1072652 [details]
Exposes libcurl EPSV enable/disable option
Comment 17 John C Peterson 2016-01-07 14:04:02 EST
A "me too" post, with suggested temporary fix.

This seems to be more or less the same problem as reported in Bug 1229050 ???

This problem can be very annoying because it seems to have nothing to do with the peak transfer speeds your network connection is capable of. Results with a Telebit Worldblazer modem set for 50 baud and results with 10^100 Google fibre lines are identical.

The comments above appear to be correct; if curl is configured to disable the use of the EPSV command, everything runs smooth as silk. A suggested short term fix (works for me) is to create a .curlrc in the home directory of the root user (/root/.curlrc) containing;

#
# .curlrc - curl configuration file
#
--disable-epsv
Comment 18 Giovanni Tirloni 2016-01-21 14:53:54 EST
I've tried the .curlrc suggestion but it didn't work here. I can see from the packet traces that it's still using EPSV whenever it connects to a FTP server.

Maybe someone with contacts to the developers could point out this bug report contains patches to address this issue? It's very annoying.
Comment 19 John C Peterson 2016-01-26 19:54:58 EST
Giovanni, you are right.

My suggestion was premature as I started to have this problem all over again this weekend. (As you no doubt noticed from the process of putting together your patch, dnf uses libcurl, rather than running the curl application. The curl application reads the .curlrc file, but libcurl does not).

I think we just need to bring this to the attention of the right people. I have cleaned up your patches to librepo (Gratze!) and will submit them to librepo upstream on Github (just some changes for shifts of line numbers due to recently added code). Once that gets fixed, we can ping the dnf folks with the other patch.

Indeed, this problem is very annoying. It has been said; "The definition of insanity is trying something over and over, and expecting a different result". That is exactly what dnf does, so I guess it has gone insane!! 

I did a little detective work to determine why this problem went away for me (for awhile...) and it seemed related to the fact that I also had enabled the dnf fastestmirror option. The following seems to work as a short term fix;

1) Edit /etc/dnf/dnf.conf and add the line "fastestmirror=true"

2) Whenever dnf gets stuck because it insists on using an ftp based mirror, edit the file /var/cache/dnf/fastestmirror.cache. Look for any ftp entries, and change the value assigned to the connectime parameter to -1, so it looks something like this;

[ftp://mirror.cs.princeton.edu]
ts=1453835022
connectime=-1

It may be necessary to edit the file all over again when dnf updates it. If you mess it up, it looks like you can just delete it, and dnf will regenerate it the next time it runs. A big pain for sure, but pretty minor when compared to the agony of watching dnf go on and on (and on... and on... and on... and on... and on...) for hours just to download a few kilobytes.
Comment 20 Fedora Admin XMLRPC Client 2016-07-08 05:28:28 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 21 Fedora End Of Life 2016-07-19 13:12:19 EDT
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.