After updating the build toolchain, its highly desirable to be able to check that all arch-specific RPMs still build correctly, without actually needing to do a mass rebuild of all the packages in Koji. The latter involves a lot of bookkeeping to keep track of the resulting build artifacts, which we don't actually care about in this case (this is a *test* for buildchain updates). Beaker is well suited to automating this check, as the same task can be run across all supported architectures and distros, we can capture the logs for the ones that fail, and we don't need to preserve the build artifacts. Some specific suggestions from Dan Horák: * accept details of the koji build tag we're wanting to check * configure mock with a repo pointing to the relevant koji build repo, and use mock's return value to determine whether or not each SRPM in that tag was rebuilt successfully * we'd need a way to ensure some components (like gcc and the RPM config files) were updated in the mock buildroot * all SRPMs that only produce noarch RPMs could be skipped, since those should be independent of the buildchain
Dan answered some additional questions for me regarding how the initial version of this task should work ("I" below = Dan) ================= > * What should we accept as input? A koji tag? A set of them? A > set of arbitrary SRPMs? I would say read the SRPMs from a compose, because it is what we will distribute, and create the list as an output of another tool or write it manually > * What's the best way to actually do the rebuild in the task? - prepare the toolchain repo - prepare /etc/mock/rebuild-$arch.cfg - for s in *.src.rpm; do mock -r rebuild-$arch $s; store_logs; done > * What level of granularity would we want in the reporting? Result per > input SRPM? Per output RPM? per input SRPM > * What build logs should we collect for failed builds? For successful > ones? all *.log files from mock result directory (/var/lib/mock/*/result) > * What knobs and dials might we need as task parameters in order to > check *proposed* updates to the build chain? if this task could be parametrized somehow (architecture, set of the SRPMs, repo for the buildroot, repo for the new toolchain), it could be used as a standard procedure to test other arches and other distros (eg. Fedora), which would be IMO great for things like the recent intent of introducing gcc5 in Fedora 22, and we don't have to reinvent the wheel every time. ================= Dan also pointed out that there's another possible variant of this task which would add each new RPM to the buildroot after it is built, but I think we can leave that capability out of the initial iteration and focus on the "does everything still build in isolation with this new version of the toolchain?" to help identify problems *before* we trigger a mass rebuild in Koji.
See also http://fedoraproject.org/wiki/Koschei
I brought this task up in the Fedora Environments & Stacks context at the Feb 12 meeting (hence the Koschei link from Colin above) and realised I needed to be a bit clearer about the intended purpose of this task. Koschei aims to regularly check that distro packages still build from source in the official build system, rather than expecting other projects to systematically check they haven't broken anything before pushing a change. The purpose of this task is to provide the other side of that coin: allowing the build toolchain developers to run such a systematic check *before* pushing a new version into the official repos. Between them, the two capabilities should help minimise the number of "Fails-to-build-from-source" errors encountered in Fedora and related distros.
I just read about mockchain --recurse, I think we can use that to do 90% of the hard work here.
(In reply to Dan Callaghan from comment #4) Oh never mind, I thought mockchain could fetch SRPMs from the configured repos, but it doesn't do that. So it's not really much help in this case.
http://gerrit.beaker-project.org/4001
... and corresponding docs: http://gerrit.beaker-project.org/4002
One flaw in the current version of the task is that it only captures build.log from mock, not any of the other logs. It appears in the Beaker results as test_log-... for each result. That's because rhts-report-result only allows you to submit a single output file for each result. I never noticed that limitation before. Ideally we would attach all mock logs to each result under their proper names. I think restraint can do better, we may need to backport something to rhts-test-env as well.
The test run showed a potential problem for full distro rebuilds - the local watchdog timed out after 24 hours. I think that's still worth releasing as an initial iteration of the task, but there are likely improvements to be made on that front. (For example, assuming I'm reading it correctly, the current iteration rebuilds everything, rather than scanning the specfile and skipping SRPMs that only generate noarch RPMs)
(In reply to Nick Coghlan from comment #10) Yeah, I will bump up TestTime to 3 days. It's hard to know how long is really sufficient. And yes, skipping noarch RPMs is a good idea. Something like this should work (it's not perfect because the spec can have conditional subpackages etc, but it should be close enough): rpm -q -p *.src.rpm --qf '%{arch}\n' | grep -qv noarch Matt also suggested a blacklist and a whitelist of packages/SRPMs which is a good idea. For example I could blacklist atlas which takes a very long time.
Looks as what I had in mind, thanks. The only thing I would probably change is the name, from "rebuild" to "seq_requild", as it goes sequentially over the srpm and doesn't try to follow the dependency chains. And in the future we might have an "ordered_rebuild" task.
(In reply to Dan Horák from comment #13) That's a good point. Unlike mockchain this task is *not* inserting the built RPMs back into the buildroot. Actually it doesn't keep them at all right now, I should probably add an option for that, so that someone can log in afterwards and copy all the built RPMs out to somewhere.
Having said that, I don't think we need to rename it. We can add an option later for doing the builds in dependency ordering, or make it a separate task if we need to. However I will make it explicit in the docs that the task builds in alphabetical order and doesn't add the results back to the buildroot.
Patches are updated to add the KEEP_RESULTS param.
/distribution/rebuild 1.0-1 is published.
Beaker 20.0 has been released.