Description of problem: The builds of the Vespa RPM (https://copr.fedorainfracloud.org/coprs/g/vespa/vespa/builds/) has startd to fail the last week. The errors seem to be related to the installation of yum in the build process and the version of libelf. In the failing builds we see a version of libelf beeing installed from a old devtoolset: devtoolset-3-elfutils-libelf x86_64 0.161-1.el7 http_mirror_centos_org_centos_7_sclo_basearch_rh 189 k devtoolset-3-elfutils-libs x86_64 0.161-1.el7 http_mirror_centos_org_centos_7_sclo_basearch_rh 258 k devtoolset-3-runtime x86_64 3.1-12.el7 http_mirror_centos_org_centos_7_sclo_basearch_rh 23 k Successful build: https://download.copr.fedorainfracloud.org/results/@vespa/vespa/epel-7-x86_64/01244399-vespa/builder-live.log.gz Failing builld: https://download.copr.fedorainfracloud.org/results/@vespa/vespa/epel-7-x86_64/01251839-vespa/builder-live.log.gz Version-Release number of selected component (if applicable): N/A How reproducible: Every time Steps to Reproduce: 1. All triggered builds fail the build 2. 3. Actual results: Build fails with: Start: yum install There was a problem importing one of the Python modules required to run yum. The error leading to this problem was: libelf.so.1: cannot open shared object file: No such file or directory Please install a package which provides this module, or verify that the module is installed correctly. It's possible that the above module doesn't match the current version of Python, which is: 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] Expected results: Would expect this to start building the vespa RPM. Additional info: I have tried to rebuild a previously successful version of Vespa (7.176.11-1.fc31), but this failed the same way. This leads me to believe that the build environment has somehow changed. We appreciate any help on this issue.
Hello, thank you for reporting this bug. We will look at it and inform you ASAP. Have a nice day, Tomas
Hi, I have been investigating this a bit more. We have an additional repository added for the Software Collections: http://mirror.centos.org/centos/7/sclo/$basearch/rh/ What seems to have changed is that this repository is enabled when installing yum as part of setting up the chroot. We require this repository to be added because we have dependencies on packages from the Software Collections. Another issue is that even with the Software Collections enabled, I can not understand why an installation of yum (one of the first steps in setting up the chroot) should pull in devtoolset-3-elfutils* packages. Best regards, Arnstein
The repo is broken, see this mock hack: https://github.com/rpm-software-management/mock/commit/fd78b04112ec95164c8118ace4a7925b093ec99e We add the sclo repository ourselves, in mock-core-configs package. But we only allow packages from developer toolset by: `includepkgs=devtoolset*` So you are basically re-adding the problem we hacked-around. This so far only broke `use_bootstrap` option. Don't you want to disable it in your project?
Thanks. I did try to remove the sclo repository we provided and the build broke on installing rh-maven35. But this is explained if you provide the repo and only include 'devtoolset*' packages. This would actually have worked if you allowed 'rh*' packages as well. However, I have now disabled the 'use_bootstrap' option and it seems to be able to progress with the build. Thanks for helping out, Arnstein
> However, I have now disabled the 'use_bootstrap' option and it seems to be able to progress with the build. Thanks for confirmation. I'm not going to close this because it still is an valid use-case, and we should make bootstrap work in such situation: * re-add the sclo repositories with broken 'libelf' provides * build something which depends on libelf * see that wrong libelf is installed This is known problem, but the fix in the CentOS repo is not going to happen unfortunately. So we should be able to work-around such problems in copr project/chroot configuration.
The bugs related to Copr build system are now migrated to the default Copr team tracker: https://github.com/fedora-copr/copr/issues/2615