Bug 1638048

Summary: Number of build fail due to repository problems
Product: [Community] Copr Reporter: Martin Kyral <mkyral>
Component: backendAssignee: Miroslav Suchý <msuchy>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: praiskup
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-26 09:14:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Kyral 2018-10-10 14:25:57 UTC
Description of problem:
Today, some 90% of my builds in mkyral/plasma-unstable fail shamefully due to problems when downloading build dependencies (sometimes even in syncing repos) [1]. That is very frustrating and it makes copr completely unusable.

See for yourselves [2]: all of the failed builds of plasma-5.14 failed due to this issue.

[1] https://copr-be.cloud.fedoraproject.org/results/mkyral/plasma-unstable/fedora-28-x86_64/00807908-kgamma/builder-live.log
[2] https://copr.fedorainfracloud.org/coprs/mkyral/plasma-unstable/builds/

FYI: this kind of problems started to appear few months ago. It was hardly bearable when it screwed 10-20% of the builds. Today, it breaks almost all the builds.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 clime 2018-10-10 17:57:05 UTC
Sorry for the problems. We are looking at the problem with Fedora infrastructure. It was an error when downloading packages from public mirrors.

Comment 2 clime 2018-10-11 08:32:03 UTC
Hello, I filed an issue about this here where you can follow details: 

https://pagure.io/fedora-infrastructure/issue/7301

Could you, please, retry the builds if the problem persists?

Comment 3 Martin Kyral 2018-10-17 12:27:38 UTC
New release of plasma shows the problem is still there. It behaves better, when I spawn the builds sequentially rather in parallel, which makes the build painfully slow - but even so around 30% of the builds fail the to the problem.

See how serious it is:
https://copr.fedorainfracloud.org/coprs/mkyral/plasma-unstable/builds/

All the fails on 5.14.1 and 5.14.0 (12 pages of them now) are caused by insfrastructure. Not one of them is actual build problem.

Comment 4 clime 2018-10-17 16:06:37 UTC
That's pretty bad. We will try to come up with some solution to this and keep you informed.

Comment 5 clime 2018-10-25 20:30:40 UTC
Hello Martin,

I've opened Bug 1643281 - dnf does not try other mirrors if the current mirror fails when downloading a package

and I've also pushed https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=dbb822036b9e4da778972b4a40fec2df0b8ca829 to help debug the issue.

Could you do another try of the rebuild so that we can potentially collect more data?

Thank you!

Comment 7 Pavel Raiskup 2019-07-26 09:14:03 UTC
(In reply to clime from comment #5)
> I've opened Bug 1643281 - dnf does not try other mirrors if the current
> mirror fails when downloading a package

These kind of errors are probably fixed already in the librepo package,
dnf team confirmed that the related code got rewritten.  Also I haven't
seen those problems for a long time.  I'm closing this bug.

> and I've also pushed
> https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/
> ?id=dbb822036b9e4da778972b4a40fec2df0b8ca829 to help debug the issue.

I reverted this, it unnecessarily overloaded the dl05 server (which isn't
otherwise used in copr), and I don't really got the point of that change
(except that it was deterministically broken either on all builders, or
nowhere).

There remains one problem with getting mirrorlist/metalink files, as
discussed in:
https://github.com/rpm-software-management/librepo/pull/158
https://github.com/rpm-software-management/librepo/pull/159
https://pagure.io/fedora-infrastructure/issue/7987
Which should be in copr production soon (TM).  But this is orthogonal
thing.