Thanks for you report. The best place to report mirror problems would be https://pagure.io/fedora-infrastructure/ , but this is probably difficult to figure out. Bugzilla is for mirrormanager packaging bug reports.

Unfortunately it is difficult to help you. We can assign countries to other continents, which would help you, but I guess not all Internet traffic in Uruguay is routed over Miami, so helping you would break things for other people.

Can you give us some more details about your IP addresses, netblocks, IP address ranges, ASNs? Maybe we can add your network, which seems to be routed over Miami to some mirrors from Florida, but we would need some details about your network and your routing. Please open a ticket at https://pagure.io/fedora-infrastructure/ with more details. Closing this here for now.

created: https://pagure.io/fedora-infrastructure/issue/7333

and added mtr traceroute there.

I can try with more sites, my IP is dynamic (it changes on every connection and every 12h)

I can try from some enterprise environment too.

Hi. Let me propose another soluction. I set up a mirror for Uruguay. Help me getting it on-line. It is already serving content and up-to-date. The announce was made last week to the mirror list. I am sure everybody is working on release 29 final release. 
Anyway http://espejito.fder.edu.uy/fedora/ might help.

Hi

@asbigue your mirror works really fast but I discovered that I had to use ipv4 specifically to get it...

See my last comment here: https://pagure.io/fedora-infrastructure/issue/7333#comment-550607

Today testing with Fedora live 30, I see these errors with DNF:


[liveuser@localhost-live ~]$ cat /var/log/dnf.
dnf.librepo.log  dnf.log          dnf.rpm.log      
[liveuser@localhost-live ~]$ cat /var/log/dnf.log 
2019-04-12T01:24:55Z INFO --- logging initialized ---
2019-04-12T01:24:55Z DDEBUG timer: config: 7 ms
2019-04-12T01:24:55Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2019-04-12T01:24:55Z DEBUG DNF version: 4.2.1
2019-04-12T01:24:55Z DDEBUG Command: dnf install iotop 
2019-04-12T01:24:55Z DDEBUG Installroot: /
2019-04-12T01:24:55Z DDEBUG Releasever: 30
2019-04-12T01:24:55Z DEBUG cachedir: /var/cache/dnf
2019-04-12T01:24:55Z DDEBUG Base command: install
2019-04-12T01:24:55Z DDEBUG Extra commands: ['install', 'iotop']
2019-04-12T01:24:55Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2019-04-12T01:24:55Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2019-04-12T01:24:55Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2019-04-12T01:24:57Z DEBUG repo: downloading from remote: fedora-modular
2019-04-12T01:25:27Z DEBUG error: Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64 [SSL connection timeout] (https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64).
2019-04-12T01:25:27Z DEBUG Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64 [SSL connection timeout].
2019-04-12T01:25:27Z ERROR Failed to synchronize cache for repo 'fedora-modular'
2019-04-12T01:25:27Z DDEBUG Cleaning up.
2019-04-12T01:25:27Z SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/repo.py", line 563, in load
    ret = self._repo.load()
  File "/usr/lib64/python3.7/site-packages/libdnf/repo.py", line 391, in load
    return _repo.Repo_load(self)
RuntimeError: Failed to synchronize cache for repo 'fedora-modular'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 99, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 115, in cli_run
    cli.run()
  File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 1111, in run
    self._process_demands()
  File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 815, in _process_demands
    load_available_repos=self.demands.available_repos)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 409, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 135, in _add_repo_to_sack
    repo.load()
  File "/usr/lib/python3.7/site-packages/dnf/repo.py", line 569, in load
    raise dnf.exceptions.RepoError(str(e))
dnf.exceptions.RepoError: Failed to synchronize cache for repo 'fedora-modular'
2019-04-12T01:25:27Z CRITICAL Error: Failed to synchronize cache for repo 'fedora-modular'


---

Looks like problem is with IPv6 only, as if I add ip_resolv = 4 in /etc/dnf/dnf.conf it works fine.

I fail to see how this could be a curl problem.  The connection times out, most likely due to underlying network problem.  Please try to download the mirrorlist using `curl -v` and paste its output here.

I'm not usre where the problem is, currently just trying to follow Kevin instructions in Pagure: https://pagure.io/fedora-infrastructure/issue/7333#comment-558412

```
curl -v --metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64
[1] 9351
[pablo@force1 ~]$ Metalink: parsing (https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30) metalink/XML...
*   Trying 2610:28:3090:3001:dead:beef:cafe:fed3...
* TCP_NODELAY set
* Connected to mirrors.fedoraproject.org (2610:28:3090:3001:dead:beef:cafe:fed3) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat Inc.; CN=*.fedoraproject.org
*  start date: Feb  1 00:00:00 2017 GMT
*  expire date: May  1 12:00:00 2020 GMT
*  subjectAltName: host "mirrors.fedoraproject.org" matched cert's "*.fedoraproject.org"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x5618c594c570)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /metalink?repo=fedora-modular-30 HTTP/2
> Host: mirrors.fedoraproject.org
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 200 
< date: Mon, 15 Apr 2019 01:45:35 GMT
< server: Apache/2.4.38 (Fedora) mod_wsgi/4.6.4 Python/2.7
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< referrer-policy: same-origin
< content-length: 304
< vary: Accept-Encoding
< content-type: application/metalink+xml
< apptime: D=2850
< x-fedora-proxyserver: proxy04.fedoraproject.org
< x-fedora-requestid: XLPiP4@yzzzeL4IeMGNdvwAAAAE
< 
* Connection #0 to host mirrors.fedoraproject.org left intact
Metalink: parsing (https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30) WARNING (missing or invalid file name)
Metalink: parsing (https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30) FAILED

[1]+  Hecho                   curl -v --metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30
```

Not even sure if the way to test with curl is the one I did above.

But what I'm sure: if I setup ip_resolv=4 to always use IPv4 with dnf: it always works.

Another thing I'm sure: If users install fedora in Uruguay, they will get a bad user experience today due to this bug with IPv6 in dnf.

From the trace it looks like curl successfully connected to an HTTP/2 server over IPv6.

> But what I'm sure: if I setup ip_resolv=4 to always use IPv4 with dnf: it always works.

This does not indicate a curl bug on its own.  I am afraid that without an in house reproducer there is no way to make any progress on this.

Thanks Kamil

I'm reassigning to dnf and let's see if we can find someone that helps on it, I can provide live workstation connected to Uruguay network.

Could you edit /etc/resolv.conf, use a different dns server and try again?
I recommend google public dns: https://developers.google.com/speed/public-dns/docs/using

The problem might be with invalid dns server.


Could you also try the command you ran in comment#7, but with quotes?
curl -v --metalink 'https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64'

The way you ran the command is incorrect, because the url contains '&' character that spawns a background job for the string before the character and ignores what follows.

Does it happen each time you fetch the same URL?

I can see CURLE_OPERATION_TIMEDOUT in comment #5 but comment #7 captures a different scenario.  Is dnf actually needed to trigger the issue?

(In reply to Daniel Mach from comment #10)
> Could you also try the command you ran in comment#7, but with quotes?
> curl -v --metalink
> 'https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-
> 30&arch=x86_64'
> 
> The way you ran the command is incorrect, because the url contains '&'
> character that spawns a background job for the string before the character
> and ignores what follows.

Good point.  I did not spot it created a background job with a shortened URL.

It works if I don't set timeout in curl, but takes veeery long time.


then with timeout:

curl -v --metalink -m 30 'https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64'
Metalink: parsing (https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64) metalink/XML...
*   Trying 2605:bc80:3010:600:dead:beef:cafe:fed9...
* TCP_NODELAY set
*   Trying 152.19.134.142...
* TCP_NODELAY set
* Connected to mirrors.fedoraproject.org (2605:bc80:3010:600:dead:beef:cafe:fed9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* Operation timed out after 30000 milliseconds with 0 out of 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 30000 milliseconds with 0 out of 0 bytes received

It looks like the TCP connection over IPv6 does not work (connects successfully but fails to transfer data).  Please try to run `tcpdump ip6` in background to see what happens.

Created attachment 1556254 [details]
tcpdump ip6

attached tcpdump ip6 result

I see curl tries both IPv4 and IPv6.  Could you please try the same command additionally with the -6 option?

$ curl -6 -v --metalink -m 30 'https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64'

Are you able to connect the host in question using s_client?

$ openssl s_client -connect '[2605:bc80:3010:600:dead:beef:cafe:fed9]:443'

Here it is the output:


$ curl -6 -v --metalink -m 30 'https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64'
Metalink: parsing (https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-30&arch=x86_64) metalink/XML...
*   Trying 2605:bc80:3010:600:dead:beef:cafe:fed9...
* TCP_NODELAY set
* Immediate connect fail for 2605:bc80:3010:600:dead:beef:cafe:fed9: Network is unreachable
*   Trying 2610:28:3090:3001:dead:beef:cafe:fed3...
* TCP_NODELAY set
* Immediate connect fail for 2610:28:3090:3001:dead:beef:cafe:fed3: Network is unreachable
*   Trying 2604:1580:fe00:0:dead:beef:cafe:fed1...
* TCP_NODELAY set
* Immediate connect fail for 2604:1580:fe00:0:dead:beef:cafe:fed1: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server

$ openssl s_client -connect '[2605:bc80:3010:600:dead:beef:cafe:fed9]:443'
140283041752896:error:02002065:system library:connect:Network is unreachable:crypto/bio/b_sock2.c:110:
140283041752896:error:2008A067:BIO routines:BIO_connect:connect error:crypto/bio/b_sock2.c:111:
connect:errno=101

It looks like comment #17 captures a different situation than comment #13.  It fails to connect in one case whereas connects successfully and then times out in the other case.

Anyway, this really seems to be a network issue.  I just wanted to confirm that it is not some tricky bug in curl's Happy Eyeballs implementation.

Because the previous comment suggests it's a network issue, there's nothing to be fixed in DNF.
Closing NOTABUG.

Is it possible to do something I think, maybe using IPv4 when IPv6 fails? instead of crashing with timeout?

(In reply to Pablo Estigarribia from comment #20)
> Is it possible to do something I think, maybe using IPv4 when IPv6 fails?

But libcurl already does it if the connection over IPv6 fails.  You can see in comment #13 that libcurl tries both IPv4 and IPv6.  It successfully establishes a TCP connection over IPv6 but then it fails to do the TLS handshake on top of that connection and times out.  This is a specific network problem that cannot be fixed (or worked around) in libcurl, except configuring it explicitly not to use IPv6.

But currently what really happens to dnf:

It fails many times, and you can successfully install/update after manually retry many times in command line.

But in gnome-software it is worse, you constantly get errors messages when it tries to refresh metadata or when you try to install something. So for a normal user you get too many errors.

So if nothing is possible to be done from dnf or curl, should I try to contact ISP?

(In reply to Pablo Estigarribia from comment #22)
> So if nothing is possible to be done from dnf or curl, should I try to
> contact ISP?

Exactly.  I do not think this problem is specific to dnf or curl.

Pablo,
since you're probably solving this problem with ISP and there's no change to be done in DNF (which is the assigned component), I'm closing this bug.