Bug 1756681

Summary: Mock EPEL 8 build fails to install build dependencies - same srpm succeeds as EPEL 8 koji scratch build
Product: [Fedora] Fedora Reporter: Mattias Ellert <mattias.ellert>
Component: mock-core-configsAssignee: Jakub Kadlčík <jkadlcik>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: dave.love, fschwarz, gary.buhrmaster, jmracek, lance, msuchy, nkadel, philip.wyett, praiskup, toracat, uid.marketplace
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-03 08:03:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Example spec file that demonstrates the problem none

Description Mattias Ellert 2019-09-29 05:44:04 UTC
Created attachment 1620697 [details]
Example spec file that demonstrates the problem

Description of problem:

The attached test specfile fails to build in mock's epel-8-candidate target:

$ mock --root epel-8-x86_64 ~/rpmbuild/SRPMS/test-1.0-1.fc30.src.rpm

The resulting error is:

Error: 
 Problem: cannot install the best candidate for the job
  - package perl-Test-Harness-1:3.42-1.module_el8.0.0+50+c3b345cd.noarch is excluded

Building the same srpm as a koji scratchbuild for epel8-candidate succeeds:

$ koji build --scratch epel8-candidate ~/rpmbuild/SRPMS/test-1.0-1.fc30.src.rpm

Here the root.log saya:

DEBUG util.py:595:  Installing:
DEBUG util.py:595:   perl-Test-Harness          noarch     1:3.42-1.el8             build     279 k

Version-Release number of selected component (if applicable):

mock-core-configs-31.5-1.fc30.noarch

How reproducible:

Always

Steps to Reproduce:
1. Create srpm from attached spec:
   rpmbuild -bs test.spec
2. Build for EPEL 8 in mock:
   mock --root epel-8-x86_64 ~/rpmbuild/SRPMS/test-1.0-1.fc30.src.rpm

Actual results:

Mock fails to install build dependencies

Expected results:

Successful build - as in koji

Comment 1 Miroslav Suchý 2019-09-30 08:43:42 UTC
Can you please try again with `--bootstrap-chroot`? Does it succeed?

Comment 2 Dave Love 2019-10-01 12:45:51 UTC
I have a similar problem with java BRs.  --bootstrap-chroot doesn't help (running mock on an el8 system). It fails in copr too, which is where I've just noticed it.

Comment 3 Mattias Ellert 2019-10-01 12:57:21 UTC
I get the same error with the --bootstrap-chroot option.

Comment 4 Miroslav Suchý 2019-10-01 15:33:59 UTC
Hmm, when I run
  mock -r epel-8-x86_64 install perl-Test-Harness
then it pass. The package is in AppStream repository which contains modules.

It looks like:
  https://bugzilla.redhat.com/show_bug.cgi?id=1673851
because it fails to install using buildep, but this should not be happening in EL8 as it has been patched. Weird.

@jmracek can you look into this please?

Comment 5 Jaroslav Mracek 2019-10-02 07:14:28 UTC
Please could you provide versions of followed component?
dnf, libdnf, libsolv

Comment 6 Miroslav Suchý 2019-10-02 09:29:35 UTC
With the `--bootstrap-chroot` the package from bootstrap chroot is used, which is:
  dnf-4.0.9.2-5.el8.noarch.rpm
  libdnf-0.22.5-5.el8_0.x86_64.rpm
  libsolv-0.6.35-6.el8.x86_64.rpm

The reproducer on DNF level is:
  mock -r epel-8-x86_64 --enable-bootstrap evil-1-1.fc31.src.rpm
  # this will fail, but prepare the files
  mock -r epel-8-x86_64 install dnf dnf-plugins-core
  mock -r epel-8-x86_64 shell --enable-network
     dnf builddep ./builddir/build/SRPMS/evil-1-1.el8.src.rpm
     # the src.rpm got there from step one

after that I get:
   Problem: cannot install the best candidate for the job
  - package perl-Test-Harness-1:3.42-1.module_el8.0.0+50+c3b345cd.noarch is excluded
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

#rpm -q dnf libdnf libsolv
dnf-4.0.9.2-5.el8.noarch
libdnf-0.22.5-5.el8_0.x86_64
libsolv-0.6.35-6.el8.x86_64

Comment 7 Jaroslav Mracek 2019-10-02 18:22:48 UTC
The issue will be fixed by "libsolv-0.7.4-2.el8".

Comment 8 Miroslav Suchý 2019-10-03 08:03:17 UTC
This will be fixed in the next minor version of RHEL (and CentOS).

Comment 9 Miroslav Suchý 2019-10-03 09:04:17 UTC
*** Bug 1711216 has been marked as a duplicate of this bug. ***

Comment 10 Nico Kadel-Garcia 2019-10-03 12:17:47 UTC
The longer term solution is to set "best=0" in the RHEL 8 and CentOS 8 configs. This is going to be a recurring problem since the introduction of modules, which are being profoundly mishandled in RHEL 8 with the multiple streams. Setting "best=0" is going to be the only reliable fix in RHEL 8 and CentOS 8, which is suffering the same issue.

Comment 11 Miroslav Suchý 2019-10-07 06:59:28 UTC
*** Bug 1758851 has been marked as a duplicate of this bug. ***

Comment 12 Pavel Raiskup 2019-10-09 10:51:01 UTC
The thing is that no one wants best=0 longterm, I'd guess.  But being very
verbose on devel list with announcement, we decided to set best=0 at least
for now for el8.  Jakub is working on this.
This is only because it doesn't make sense to break majority of builds in
copr, the epel-8* chroots seem to be pretty useless.

FWIW, two _probably_ related RFEs regarding design flaws in modularity
which I'm not terribly happy about:

https://bugzilla.redhat.com/show_bug.cgi?id=1758470
https://bugzilla.redhat.com/show_bug.cgi?id=1758467

Comment 13 Nico Kadel-Garcia 2019-10-14 03:32:14 UTC
Please set it in centos-8.tpl and rhel-8.tpl. They *both* need it, and will for the foreseeable furutre. The underlying problem is not the release, per se, but the activation of "modular" packages in multiple software channels such as BaseOS and AppStream. There has been some philosophical discussion of how to really fix modular dependencies, but I'm afraid it's not a problem that can be solved in concert with multiple overlapping channels and the "best-1" setting for mock or for dnf.