Bug 1192524

I've found some logs of dnf, here is the relevant part

Feb 17 06:37:47 DDEBUG repo: downloading from remote: fedora, _Handle: metalnk: https://mirrors.fedoraproject.org/metalink?repo=fedora-21&arch=x86_64, mlist: None, urls [].
Feb 17 06:37:56 DDEBUG Cleaning up.
Feb 17 06:37:56 SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 84, in main
    return _main(base, args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 134, in _main
    cli.run()
  File "/usr/lib/python2.7/site-packages/dnf/cli/cli.py", line 1059, in run
    return self.command.run(self.base.extcmds)
  File "/usr/lib/python2.7/site-packages/dnf/cli/commands/makecache.py", line 87, in run
    self.base.fill_sack() # performs the md sync
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 222, in fill_sack
    self._add_repo_to_sack(r.id)
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 98, in _add_repo_to_sack
    repo.load()
  File "/usr/lib/python2.7/site-packages/dnf/repo.py", line 781, in load
    raise dnf.exceptions.RepoError(msg)
RepoError: Failed to synchronize cache for repo 'fedora' from 'https://mirrors.fedoraproject.org/metalink?repo=fedora-21&arch=x86_64': Yum repo downloading error: Downloading error(s): repodata/abb4ea5ccb9ad46253984126c6bdc86868442a4662dbcfa0e0f51b1bb209331e-filelists.xml.gz - Download failed: Curl error (18): Transferred a partial file for ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/21/Everything/x86_64/os/repodata/abb4ea5ccb9ad46253984126c6bdc86868442a4662dbcfa0e0f51b1bb209331e-filelists.xml.gz [transfer closed with 10133739 bytes remaining to read]
Feb 17 06:37:56 CRITICAL Error: Failed to synchronize cache for repo 'fedora' from 'https://mirrors.fedoraproject.org/metalink?repo=fedora-21&arch=x86_64': Yum repo downloading error: Downloading error(s): repodata/abb4ea5ccb9ad46253984126c6bdc86868442a4662dbcfa0e0f51b1bb209331e-filelists.xml.gz - Download failed: Curl error (18): Transferred a partial file for ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/21/Everything/x86_64/os/repodata/abb4ea5ccb9ad46253984126c6bdc86868442a4662dbcfa0e0f51b1bb209331e-filelists.xml.gz [transfer closed with 10133739 bytes remaining to read]

It cannot download this file.
What is odd is that yum can finish the download with a partial download.

Has the logic to download a file changed between yum and dnf?

Thanks for the report. Can you set in /etc/dnf/dnf.conf `skip_if_unavailable=True` install dnf-0.64 and librepo-1.7.14, test it again and share the result, please?

Nope

dnf-0.64 was already there, installed 

librepo-1.7.14-1.fc23.x86_64.rpm
python-librepo-1.7.14-1.fc23.x86_64.rpm

done a dnf clean up and then

xxxxxx@localhost:~$ sudo time dnf update
Fedora 21 - x86_64                                                                                                                                          44 kB/s |  39 MB     15:14    
Error: Failed to synchronize cache for repo 'updates' from 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f21&arch=x86_64': Yum repo downloading error: Downloading error(s): repodata/3efed204dbb15a6c7fe0137c5f66446d86749f32c2b0f645accbcaac638cebbe-filelists.xml.gz - Cannot download, all mirrors were already tried without success
32.75user 14.57system 56:50.40elapsed 1%CPU (0avgtext+0avgdata 246848maxresident)k
16inputs+176096outputs (0major+111244minor)pagefaults 0swaps

content of /etc/dnf/dnf.conf

xxxxxx@localhost:/tmp$ cat /etc/dnf/dnf.conf 
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=true

proxy=http://localhost:3128

skip_if_unavailable=True

attached is the relevant part of dnf.log

as a comparison, yum was successful and took 55 seconds.

Created attachment 999784 [details]
dnf.log for skip_if_unavailable=True

/var/log/dnf.log with librepo-1.7.14 and dnf 0.6.4

with

skip_if_unavailable=True

It seems like the connection of mirrors is closed because of timeout. Can you try to set `timeout=100` (or more) [1] into dnf.conf, please?

[1] http://dnf.readthedocs.org/en/latest/conf_ref.html#options-for-both-main-and-repo

it worked!!!!!

I set 300 and finally I got a little update with dnf.

I will try to find out what is (roughly) the minimum acceptable.

My fear is the following: when install F22, will I be able to change this *during* the install?

(In reply to Andrea from comment #6)
> it worked!!!!!
> 
> I set 300 and finally I got a little update with dnf.
> 
> I will try to find out what is (roughly) the minimum acceptable.
> 
> My fear is the following: when install F22, will I be able to change this
> *during* the install?

Do you mean clean install from Anaconda or FedUp / Fedora upgrade / distro-sync to newer Fedora release?

Either way the conf should be set beforehand.

I mean brand new install of F22 on a new PC / VM, where yum has been removed.

In any case, dnf is much much slower than yum, even now that it succeeds.

Yeah, DNF is very slow... it hangs on almost every repository. It attempts to connect on ports that are either blocked, or unavailable on the remote server. (i.e.: SYN_SENT for servers that aren't responding) Attempted the above timeout=5 and still nothing.

Yeah, this is definitely a bug... curl vs wget:

2 minutes with curl, 0.1seconds with wget:

$ curl -o foobar ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1409k  100 1409k    0     0  11313      0  0:02:07  0:02:07 --:--:--  348k


$ wget ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
--2015-09-11 12:56:58--  ftp://mirror.symnds.com/distributions/fedora/releases/22/Everything/i386/os/Packages/m/maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm
           => 'maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm.1'
Resolving mirror.symnds.com (mirror.symnds.com)... 63.245.196.124
Connecting to mirror.symnds.com (mirror.symnds.com)|63.245.196.124|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /distributions/fedora/releases/22/Everything/i386/os/Packages/m ... done.
==> SIZE maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm ... 1442888
==> PASV ... done.    ==> RETR maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm ... done.
Length: 1442888 (1.4M) (unauthoritative)

maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm.1          100%[=======================================================================================================================>]   1.38M  --.-KB/s   in 0.1s

2015-09-11 12:56:59 (10.6 MB/s) - 'maven-scm-javadoc-1.9.2-2.fc22.noarch.rpm.1' saved [1442888]

I am hitting the same problems on Copr builders.

Copr builders have:
  dnf-1.1.3-1.fc23.noarch

Yum is able to download packages and repositories, but DNF fails with approximately 30% probability with:

/usr/bin/dnf -d 2 -y --enablerepo=updates-testing check-update\'.\nSee \'man dnf\' and \'man yum2dnf\' for more information.\nTo transfer transaction metadata from yum to DNF, run:\n\'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate\'\n\nError: Failed to synchronize cache for repo \'updates-testing\' from \'https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f22&arch=ppc64le\': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Those builders are in PHX datacenter so connectivity should not be such issue. Yet DNF very often die with this error.

I put as workaround 
  timeout=1000
in dnf.conf. But this is not sustainable as I will need to modify all mock configs too. Right now we have just 6 of them which use dnf, but it will grow soon.

Could you please take a look Tomas whether we are able to influence this on the librepo level?

Hi Miroslav, could you please try this scratch build of librepo?
http://koji.fedoraproject.org/koji/taskinfo?taskID=13226518

This version has two main features:
1) It uses HTTP headers that tell to proxy server to always serve fresh metalink.xml, mirrorlist and repomd.xml files - no proxy cache should be causing issues anymore.

2) Has a new option LRO_FTPUSEEPSV [1] by witch an extended passive mode for FTP could be disabled (the extended passive mode was probably the problem which was described by "EMR_Fedora").

[1] http://rpm-software-management.github.io/librepo/lib.html#librepo.LRO_FTPUSEEPSV

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Missing needinfo  for half a year, closing.

If you still hit this issue, feel free to reopen.

Clearing needinfo.