Bug 1844087 - Repo fedora-cisco-openh264 mirrorlist empty from some source ip addresses
Summary: Repo fedora-cisco-openh264 mirrorlist empty from some source ip addresses
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: mirrormanager2
Version: 32
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Adrian Reber
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1846747
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-04 15:20 UTC by Edgar Hoch
Modified: 2020-06-28 10:04 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-28 10:04:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Result from step 1 called from subnet 141.58.127.0/24 (838 bytes, application/metalink+xml)
2020-06-04 15:20 UTC, Edgar Hoch
no flags Details
Result from step 1 called from subnet 89.244.184.0/24 (986 bytes, application/metalink+xml)
2020-06-04 15:21 UTC, Edgar Hoch
no flags Details

Description Edgar Hoch 2020-06-04 15:20:40 UTC
Created attachment 1695228 [details]
Result from step 1 called from subnet 141.58.127.0/24

Description of problem:
Fedora 32 has repo fedora-cisco-openh264 enabled by default.
When I call dnf with some parameters, e.g. "dnf upgrade", I get the folloging error:

Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: No URLs in metalink

I have noticed that this error does not occour in another location. It depends on the ip address of the host calling dnf or calling the metalink url directly.

If I call the url (see below) from hosts with an ip address from subnet 141.58.127.0/24, then the list is empty, no url is listed.
If I call the url from hosts with 89.244.184.0/24, then the list is not empty, (only) one url is listed.


Version-Release number of selected component (if applicable):
fedora-repos-32-2.noarch

How reproducible:
The problem occures only on some ip addresses.


Steps to Reproduce:
1. wget "https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-32&arch=x86_64"


Actual results:
The metalink file contains no url.

Expected results:
The metalink file contains at least one url.

Comment 1 Edgar Hoch 2020-06-04 15:21:16 UTC
Created attachment 1695229 [details]
Result from step 1 called from subnet 89.244.184.0/24

Comment 2 Kevin Fenzi 2020-06-04 17:55:42 UTC
Moving to mirrormanager...

Comment 3 Adrian Reber 2020-06-04 18:20:22 UTC
Hmm, I can confirm, this actually does not work. Need to take a closer look at this.

Comment 4 Adrian Reber 2020-06-05 17:29:02 UTC
I think I understand the problem. It is a bug in the mirrorlist-server.

The requests is received by the mirrorlist-server and then the mirrorlist server tries to figure out which mirror is responsible for that network. It first looks if there is a mirror in the same network, ASN, country, continent. Between each of the steps it checks if we have reached the minimum number of mirrors we want for each request. If we are below we go to the next (ASN, country, continent) if we are over the minimum number of networks we continue looking for mirrors. The last step is to add global mirrors to the answer. As the fedora-cisco-openh264 repository is only available from one mirror in the US the number of mirrors needs to be less than the minimum number of mirrors.

For the network in this bug report there are a couple of mirrors which say they are responsible for an IP address in that network so that we never reach the global part and so the only mirror for this is not added.

The actual bug is that we count mirrors independent of the repository. We first do a repository independent check of the number of possible mirrors and in the last step we check if those mirrors actual have the requested repository. The check if enough mirrors have been found needs to happen on the number of actual mirrors carrying that repository.

One possible easy quick fix would be to change all mirrorlist-servers on the proxies to use a much larger number for minimum number of mirrors. The default is 5, if we set it to 15 we probably could work around this until the code is fixed.

Comment 5 coillancer 2020-06-12 20:29:36 UTC
Hi, I`m having the same problem.

> One possible easy quick fix would be to change all mirrorlist-servers on the
> proxies to use a much larger number for minimum number of mirrors. The
> default is 5, if we set it to 15 we probably could work around this until
> the code is fixed.

Sorry for the noob question, but how do I change the minimum number of mirrors to 15? Is there a dnf command for that?

Comment 6 Edgar Hoch 2020-06-12 21:30:03 UTC
(In reply to coillancer from comment #5)
> > One possible easy quick fix would be to change all mirrorlist-servers on the
> > proxies to use a much larger number for minimum number of mirrors. The
> > default is 5, if we set it to 15 we probably could work around this until
> > the code is fixed.
> 
> Sorry for the noob question, but how do I change the minimum number of
> mirrors to 15? Is there a dnf command for that?

I think the easy quick fix is a task for the mirrorlist-server administrators.


A temporary workaround for you may be to replace the metalink declaration by a baseurl declaration, because currently there is only one server for this repo.

Edit file /etc/yum.repos.d/fedora-cisco-openh264.repo:

Place "# " in front of the line to comment out it:
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-$releasever&arch=$basearch

Add the following line near (e.g. above or below) the commented metalink declaration:
baseurl=https://codecs.fedoraproject.org/openh264/$releasever/$basearch/

Comment 7 Edgar Hoch 2020-06-12 21:40:42 UTC
(In reply to Adrian Reber from comment #4)
> I think I understand the problem. It is a bug in the mirrorlist-server.
> [...}
> As the fedora-cisco-openh264 repository is only
> available from one mirror in the US the number of mirrors needs to be less
> than the minimum number of mirrors.

Thanks for taking care of the problem!

Of course, the code should be fixed. But another question is, why is there only one mirror? Every fedora host with Fedora 32 (and possibly successors) will check the mirror (if it works) for updates every six hours.

The disk space for this repo is very small. So I think that sites that already mirror the main fedora repos, they wouldn't have problems (at least no disk space problem) to also mirror this small repo too. I suggest that the admins that maintain the current main mirrors should be asked to also mirror this small repo too. This would distribute the network traffice and load of the server across the world.

Comment 8 Kevin Fenzi 2020-06-15 23:38:06 UTC
(In reply to Edgar Hoch from comment #7)
>
> Thanks for taking care of the problem!
> 
> Of course, the code should be fixed. But another question is, why is there
> only one mirror? Every fedora host with Fedora 32 (and possibly successors)
> will check the mirror (if it works) for updates every six hours.

It's complicated, but there's not one mirror, there's one CDN. 
 
> The disk space for this repo is very small. So I think that sites that
> already mirror the main fedora repos, they wouldn't have problems (at least
> no disk space problem) to also mirror this small repo too. I suggest that
> the admins that maintain the current main mirrors should be asked to also
> mirror this small repo too. This would distribute the network traffice and
> load of the server across the world.

No, we cannot legally distribute the rpms in this repo. 
Cisco can do so, so we build the package, sign it and they distribute it. So the 'one mirror' here is actually the cisco CDN. 

See https://fedoraproject.org/wiki/OpenH264 for more information and links.

Comment 9 Edgar Hoch 2020-06-15 23:49:41 UTC
Oh, I didn't know that. Thanks for the info. I was wondering why there was an extra repo for that. With the CDN, the load is distributed.

Comment 10 Edgar Hoch 2020-06-17 11:46:00 UTC
Why do you close it as NOTABUG?

It _is_ a bug in mirrorlist-server, see comment 4. It is still not solved, neither in the source (at least not noted here) nor the workaround ("easy quick fix") from comment 4. I still get the error messages from dnf.

Comment 11 Peter Robinson 2020-06-17 11:49:35 UTC
Sorry, wrong bug.

Comment 12 Adrian Reber 2020-06-17 11:53:22 UTC
It is fixed upstream. The package just needs to make its way into Fedora: https://github.com/adrianreber/mirrorlist-server/commit/be2c35a9f49f7ac45766787259f52e44023708f0

Comment 13 Adrian Reber 2020-06-18 08:52:05 UTC
Fedora packages have been built and now it needs to be installed on our proxies. Because of the datacenter move we currently do not have a staging environment to test this. Might take a couple more days to get this deployed.

Comment 14 Adrian Reber 2020-06-28 10:04:30 UTC
This should be fixed now.


Note You need to log in before you can comment on or make changes to this bug.