Description of problem: Unfortunately I haven't been able to build a RHEL-7.x kernel using our internal COPR instance for the past couple of weeks despite the builds succeeding using Brew/Koji. The builds always seem to fail just after the RPMs are created, a snippet from the build logs is below: Checking for unpackaged file(s): /usr/lib/rpm/check-files /builddir/build/BUILDROOT/kernel-3.10.0-897.el7.1.secrhel.x86_64 Wrote: /builddir/build/RPMS/kernel-3.10.0-897.el7.1.secrhel.x86_64.rpm ... ERROR: Exception(intermediate-srpm/kernel-3.10.0-897.el7.1.secrhel.src.rpm) Config(pmoore-kernel-secrhel_rhel-7.dev-x86_64) 151 minutes 59 seconds INFO: Results and/or logs in: /var/lib/copr-builder/results INFO: Cleaning up build root ('cleanup_on_failure=True') Start: clean chroot INFO: unmounting tmpfs. Finish: clean chroot ERROR: Command failed: # bash --login -c /usr/bin/rpmbuild -bb --target x86_64 --nodeps /builddir/build/SPECS/kernel.spec A list of the most recent failed builds are included below: * https://copr.devel.redhat.com/coprs/pmoore/kernel-secrhel/build/19995 * https://copr.devel.redhat.com/coprs/pmoore/kernel-secrhel/build/19859 * https://copr.devel.redhat.com/coprs/pmoore/kernel-secrhel/build/19634 * https://copr.devel.redhat.com/coprs/pmoore/kernel-secrhel/build/19632
Any progress on this?
Hi Paul, thanks for the report and sorry for the delay. The rpmbuild process is getting KILL, probably because of insufficient ram. Internal Copr has 4G RAM + 16G swap, but mock has the tmpfs turned on. Nothing has changed recently from this POV in Copr, but it might be that kernel build system becomes more demanding now. I'm curious whether you can do something about this in your package, or whether I should turn off the tmpfs feature for everyone (there's not yet possible to turn on/off the tmpfs feature per project).
We have this on copr builders, in site-defaults.cfg: config_opts['plugin_conf']['tmpfs_enable'] = False config_opts['plugin_conf']['tmpfs_opts'] = {} config_opts['plugin_conf']['tmpfs_opts']['required_ram_mb'] = 1024 config_opts['plugin_conf']['tmpfs_opts']['max_fs_size'] = '30g' config_opts['plugin_conf']['tmpfs_opts']['mode'] = '0755' config_opts['plugin_conf']['tmpfs_opts']['keep_mounted'] = True which seems to be wrong anyways. The max_fs_size should be probably about ~16G to let something for real operational memory; and to give the tooling a chance to report more clear "no space left on device" error.
I know that some people use '%global debug_package %{nil}' to make the build significantly less demanding; would something like that be suitable for your use-case?
I'm not able to complete a build using the standard RHEL-7.x kernel SRPMs, if we want the internal COPR instance to be useful we should be able to build any of the RHEL-X software packages on it.
There are limited resources for copr builders at this moment though. Paul, if you are anyhow able to help with feeding copr with more hardware, let me know - it would be awesome. E.g. if the RAM or disk space was doubled, we would have half of the builders and that would be overall slowdown for everyone (and long builds like the kernel one would block the service noticeably). There's potential in the bug 1334701, but it requires tagging mechanism for builds to be implemented upstream, and it's not something which will happen promptly.
I tried to build the kernel package with '%define debugbuildsenabled 0' (one-liner). I admit it might be a problem... but it worked for me: https://copr.devel.redhat.com/coprs/praiskup/kernel-test/build/20506/
(In reply to Pavel Raiskup from comment #6) > There are limited resources for copr builders at this moment though. > Paul, if you are anyhow able to help with feeding copr with more hardware, > let me know - it would be awesome. Sorry, I do not have access to additional hardware.
(In reply to Pavel Raiskup from comment #7) > I tried to build the kernel package with '%define debugbuildsenabled 0' > (one-liner). I admit it might be a problem... but it worked for me: > https://copr.devel.redhat.com/coprs/praiskup/kernel-test/build/20506/ Unfortunately building packages without the debug builds is not very useful for my internal COPR use case. I'm using the internal COPR much in the same way I use the Fedora COPR: make early development builds regularly available for people who want to test new/backported functionality and fixes. Considering the nature of the changes (early development work), having the debug builds/info is an important part of this effort.
Paul, would you mind trying to build once more and confirm that the issue is not resolved for you?
I just submitted the COPR kernel build below, unfortunately this is my last week at RH so if this fails I likely won't be able to help you verify this in the future. However, simply attempting to build the RHEL-7.x kernel should be sufficient to reproduce the problem. * https://copr.devel.redhat.com/coprs/build/22592
The build mentioned above in comment #11 appears to still be failing.
The max tmpfs size is now 30G and the swap space is correctly configured, so we should build kernel package just fine (tested today on RHEL7 kernels).